From patchwork Fri Jan 24 16:53:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51E8A13A4 for ; Fri, 24 Jan 2020 16:57:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EB12220709 for ; Fri, 24 Jan 2020 16:57:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="mL7hhvik" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB12220709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44934 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2GT-00057Q-QI for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:57:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45541) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Fr-0003cU-RV for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2Fq-0002vm-UX for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:23 -0500 Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]:55740) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2Fq-0002vA-PK for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:22 -0500 Received: by mail-pj1-x102b.google.com with SMTP id d5so61373pjz.5 for ; Fri, 24 Jan 2020 08:56:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=X0NxLy9YfOGRAzvR5nre9SreXUazO7Ppv8M7S281EDw=; b=mL7hhvikCjOfHRj5MDWBzw3Oi1/OGEqV5ZojROPHMBNhk4rd6opa737N0fpeb+ZTae FIZevJI8OJoFcSnRod6z0NDeTtz06fxHJ5k8/SZW4jeiQQOLmKqrS6gYqZ+MSso4ytaX 5h6aPLSImgq2xx3T7TNAavxxlXy3UoQ7IW8dtDc+Fmz+du+tWKqC+7Z1OFm4M1nt6BxZ 1kKqvvYv+WtAhSurdxKZBmj4Gz0Vhqfzt+bxbnXIgSdQGzAn1cufH4KDB3j4KysrTssP lNUTg3NcWlED68lrQaP2r6PRVTnzy+ZRAzZCT0TwkC7YG/J58XkEoxLMc6scQv9+tUuk KjZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=X0NxLy9YfOGRAzvR5nre9SreXUazO7Ppv8M7S281EDw=; b=GAGH4fDoWy54wgVuGpICj93KP5dO2GQ/s9/T5zb4fxfyC8vNM8B0EZkHT46T7mwecc 8syhyD3TOD2phWQHJ/aHYrHwClsSLFTJxYggns2npvYEHAfcgDZP3plxYWn4v/iDQK6K OkEUPnZSYDyLv3gBht4AmLOoluxIQ/xmm3QYbRTqywKsBhR7qt9Gkh8SmoqKYfe4Jn6h Uz8YyGZTnIhEgzm+r1hj/rTF+2i/MNI/uVnT5LaHu4pmJ3euGbwCzn7Gl0TpnFMHdwvC GvCcuO5kdyZ8/K7xwUM6hJHqfrl4PrFzErPs4hnwRgdaYN65GA9RKTOra8q/Ck73CEk1 YNMw== X-Gm-Message-State: APjAAAU+mn+rxa5y3nniyXeesMLYAzIZ+MVVS51mbAEmpp0VU64DfUuL xURrh77/lMKZhPJgih1YF3TI38n+sls= X-Google-Smtp-Source: APXvYqza1F0gg1YP6n0jX3v6NCiga5uzfdZBZtZx9TJ3dw0Rnib+xD4tDoODGQdZYf2q+dRZYa9cIw== X-Received: by 2002:a17:902:708b:: with SMTP id z11mr4552410plk.121.1579884981668; Fri, 24 Jan 2020 08:56:21 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:21 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 1/8] tests/vm: Debug mode shows ssh output. Date: Fri, 24 Jan 2020 11:53:28 -0500 Message-Id: <20200124165335.422-2-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::102b X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add changes to tests/vm/basevm.py so that during debug mode we show ssh output. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov Reviewed-by: Alex Bennée --- tests/vm/basevm.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index ed5dd4f3d0..991115e44b 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -122,11 +122,16 @@ class BaseVM(object): return fname def _ssh_do(self, user, cmd, check): - ssh_cmd = ["ssh", "-q", "-t", + ssh_cmd = ["ssh", + "-t", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=" + os.devnull, "-o", "ConnectTimeout=1", "-p", self.ssh_port, "-i", self._ssh_key_file] + # If not in debug mode, set ssh to quiet mode to + # avoid printing the results of commands. + if not self.debug: + ssh_cmd.append("-q") for var in self.envvars: ssh_cmd += ['-o', "SendEnv=%s" % var ] assert not isinstance(cmd, str) From patchwork Fri Jan 24 16:53:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350725 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AEB213A4 for ; Fri, 24 Jan 2020 16:57:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D5ECB20709 for ; Fri, 24 Jan 2020 16:57:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="vUEEWhQB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D5ECB20709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44942 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2GW-0005FL-SB for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:57:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45561) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Fu-0003d0-16 for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2Ft-0002y6-4i for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:25 -0500 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]:34163) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2Fs-0002ws-Vz for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:25 -0500 Received: by mail-pj1-x1033.google.com with SMTP id f2so740623pjq.1 for ; Fri, 24 Jan 2020 08:56:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RIEQhZ5WEe7hDe6XHzbT9XXua29+aUUP5Ja/7DtAgr0=; b=vUEEWhQBNCKS171eNvRz/83KqU1NbzdbhzdeWwM2r4jkPu7kAWWMonbXAH1J73bZTX DSN2EXhyAEH4CngzqFGd03EH/c1oKTAofkSnyQQqsmwjsitAi+aneegyNvciBBpdJF4x mjdV0Oge04M9UKvUjPS7HjI2d15/XXF09TJzgkmfB/cMfBbn/IVV1+/oMZ5PFA7gAl3N Z/7u31UMrXEL67V/lKRH+2mbbVfJkkQIh0XEno96TmkaBzuQqkhD23lXYKRHzsAnugoi NgWgdFeNFhWiRtHRB+BCaz9cTDLE542dT/dydxxhmxOk6OSe+XoaMYBqT+zEXws0A9iL U+8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RIEQhZ5WEe7hDe6XHzbT9XXua29+aUUP5Ja/7DtAgr0=; b=GwOjooaelHVdwuqVYeBuu6OqEJBspKDApqYwXw4YoVEkvCdF/eFOnAstaUlyUxEqDm xbj6eC2ji6nvILM+kfMAp4C8gD78vngBWE4cqywyNlNwDu0SGZrmLjHo1IVzkKxFoLJR qDNSaIjpwgBWcYawDnsEezj/ARkUlaZ83oydQujYkNwMV+wcrnSzKeA6AYvK6XA/13Dk QLC/0DVPEWBRZmNUKez6ceQNbOcwuOjpR4ZPlm+F0u+aVVa3HrriYivlCpZmHU2dQWos vJ7M4lWhAtswUG7nVyXXTQzJivKhFf9tfJFdqZ4jyBP4+9LxsCTEbzPk+zWOzHoKh2mR /J5A== X-Gm-Message-State: APjAAAU8ZSzerjJwmejVgM28wCA0rFY/vrceLsGaYULxE7Gt8N44Cifp J6HsCaL3C5Ga2bIqbXwl1R/nmMV6yMQ= X-Google-Smtp-Source: APXvYqzaQ8OfGf7XsfO7LUcxwRuVj2VyJIeEiLtL0z5IabDBVRIbXEJfgZBLaRDB0qfq9yMzziv3DQ== X-Received: by 2002:a17:90a:d205:: with SMTP id o5mr204259pju.46.1579884983677; Fri, 24 Jan 2020 08:56:23 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:23 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 2/8] tests/vm: increased max timeout for vm boot. Date: Fri, 24 Jan 2020 11:53:29 -0500 Message-Id: <20200124165335.422-3-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1033 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Add change to increase timeout waiting for VM to boot. Needed for some emulation cases where it can take longer than 5 minutes to boot. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 991115e44b..86908f58ec 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -310,7 +310,7 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) - def wait_ssh(self, seconds=300): + def wait_ssh(self, seconds=600): starttime = datetime.datetime.now() endtime = starttime + datetime.timedelta(seconds=seconds) guest_up = False From patchwork Fri Jan 24 16:53:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350733 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D13316C1 for ; Fri, 24 Jan 2020 16:58:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A6DB720709 for ; Fri, 24 Jan 2020 16:58:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="kD8ZhaZI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A6DB720709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44986 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Hu-0008Td-R8 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:58:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45578) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Fw-0003gR-2N for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2Fv-000309-0R for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:28 -0500 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]:40404) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2Fu-0002z1-RB for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:26 -0500 Received: by mail-pl1-x642.google.com with SMTP id p12so604897plr.7 for ; Fri, 24 Jan 2020 08:56:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NCvSSBqxEUQx9J/P5MFSza4FDjSc3lDbb0fGxUZvm1A=; b=kD8ZhaZIC+fbufp9yJakIiLu3HvZVa9u6F1XPm3nLHu8GYr9kdaL97rUZi3by6ZWLr L2SMzF11dfZKtuoEPQm63dwyRBwYOAZ/3hTUWiM1MMtCnVhYY2KCoU9YqjLJ6KbgtuUq FWY1uSQ0jM5x+kCaAhkccAQ/wPuLGdtaumfctur/l376r5zfoIAJ2OH5UZe6ryTggz2F dwAC+3byPo/rW6BvNUKVRxtqE/0kVMPGfwivE2OS8FRdIhSxOdHFd8fZ7HG9T1+Xbq2l iU+FYL5YHCEamCyNJ4DJQBJmZreCKk9BIylYD4ibA1KsxhWfZriW9+JMwFEz86LK/9s6 VaDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NCvSSBqxEUQx9J/P5MFSza4FDjSc3lDbb0fGxUZvm1A=; b=SVp1D9rMLS6LK68a3j2jYOh0TvlrIHMC1BwGhfJGSI4sAAq5/dxIPLK47pN0hth7aS J1sK5hTdjqo1iMjnkNP4zpHLINEmb5dLzjeV3JYDLxs/XwUVz9p07dBfrdUQ6UBQH0Io JqN8JpHNq1bAntN50tGQCTfAZ2V+ijfLXg/V0KJQv9ZzFYNJF+9QjgbjMG+xuxH5t6rn /AxURQ234sY8VRsDNFcnaqaH8zo0PZADvF6i3n0IdjwxZhGvEpFfxLM7PKDd13lT/dnE fkmkDKG3SRINYQ6bYqL/rz3boPXdT2v0lvSYpzVRKg12spzaMavc+rE62tQlJB5WeWeb IuKg== X-Gm-Message-State: APjAAAUKHFPCoscqfZ+NwFjXDyArB40BlAIvkBAkfMaCJkL7WS5xlbZi dW9S8n7So27Q1XqirwANDmV9S4kYxk0= X-Google-Smtp-Source: APXvYqxTsj5SxM5QCENm+CjFYJx5wQ9hyCaSon3DugO50L5ZgsXsZT/0/gCcmQZQFPBxAy75Y8Z0yw== X-Received: by 2002:a17:902:9003:: with SMTP id a3mr4298090plp.339.1579884985645; Fri, 24 Jan 2020 08:56:25 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:25 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 3/8] tests/vm: change wait_ssh to optionally wait for root. Date: Fri, 24 Jan 2020 11:53:30 -0500 Message-Id: <20200124165335.422-4-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::642 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Allow wait_ssh to wait for root user to be ready. This solves the issue where we perform a wait_ssh() successfully, but the root user is not yet ready to be logged in. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 86908f58ec..3b4403ddcb 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -310,12 +310,17 @@ class BaseVM(object): def print_step(self, text): sys.stderr.write("### %s ...\n" % text) - def wait_ssh(self, seconds=600): + def wait_ssh(self, wait_root=False, seconds=600): starttime = datetime.datetime.now() endtime = starttime + datetime.timedelta(seconds=seconds) guest_up = False while datetime.datetime.now() < endtime: - if self.ssh("exit 0") == 0: + if wait_root: + if self.ssh("exit 0") == 0 and\ + self.ssh_root("exit 0") == 0: + guest_up = True + break + elif self.ssh("exit 0") == 0: guest_up = True break seconds = (endtime - datetime.datetime.now()).total_seconds() From patchwork Fri Jan 24 16:53:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350729 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B4A8138C for ; Fri, 24 Jan 2020 16:58:28 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 21C7C20709 for ; Fri, 24 Jan 2020 16:58:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="OlvSD5HJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 21C7C20709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44978 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Hr-0008PV-37 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:58:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45598) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Fz-0003nw-2c for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2Fx-00032m-CZ for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:31 -0500 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]:44839) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2Fx-00031X-42 for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:29 -0500 Received: by mail-pl1-x630.google.com with SMTP id d9so1016367plo.11 for ; Fri, 24 Jan 2020 08:56:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aVBYNqhMnJi7bxbz3Ybs5e2VT5E8fdOaR1g785IV9mw=; b=OlvSD5HJT5ihWD0gUoygNSLFW9B2C380HIwJ9lFLajAyTWmSQLAc8j4A2HQKqINIPm bn8H8dVJDlE+rSRdT3AxQomRFrsIrFrC2amqd8M8OlWB87R03IKf7mBhswCfuKiRP0bh sJ8VPJAsLwJeCisrnPjFqcvlGcN18cc9ykoKMWU/kyEJEpuc0Ub0fbEowJSbxASR8oym A2orLqf+xFdlr3On1l9ZpWGiRv018NdU4x8qeNk9pVw/UTBqpzv4HOGqfJ++2X14NKRU 7p55HGm7TN80UP4qkDX+le/m9GbhSMOv0vM+nGq6RME5cXe/becASLtTcoQHMdvFFsM/ E9+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aVBYNqhMnJi7bxbz3Ybs5e2VT5E8fdOaR1g785IV9mw=; b=ZZFcC1Zgt/cEQFwsryMc0ytZKGIwxrJ0cXcCUCD92SUAahDjipNLLKOyvja47qOnDE I6BSU9I8dL5JxC+xTjFMwRDLJluUHnqbKDom7Yq87+DiC6lS2zRkLdU0e6VCArIYdKd4 2sb7skB2VSRmV43hN0nTpq2PAXOKg+yP6bt9UG9lsiRvZJyD7UsPm6KQBi/cNdPftGUS qifi3bt/lnpzullgeyBEN9tuwjy45FWf+Jg6sM3u54DCqvyuZhAzaclAXmVN4o377T1v Iv6B+yILbTTgLnm6DdQqJyAkfqlqz2PLUveAP81lizkLmRXKppP9qZChSUJg5kMtuaKH J8og== X-Gm-Message-State: APjAAAWzCQmVzo0b7IfGB5JeSlRZLfK0Cb/w7d5Hy1OnFMQHDAACMGVv ghZbwt8mTIJQIlF97xK/pXaIhBT3Swc= X-Google-Smtp-Source: APXvYqyb+OOKVoTi8EcT5vYwgXRS15l/TWNbWHtbXiq0xkOv1Uv8D/greU4RNmjX7J941iCv+MfXDQ== X-Received: by 2002:a17:90a:b10b:: with SMTP id z11mr182316pjq.132.1579884987768; Fri, 24 Jan 2020 08:56:27 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:26 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 4/8] tests/vm: Add configuration to basevm.py Date: Fri, 24 Jan 2020 11:53:31 -0500 Message-Id: <20200124165335.422-5-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::630 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being created or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 108 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 82 insertions(+), 26 deletions(-) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 3b4403ddcb..ec92c8f105 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -32,15 +32,40 @@ import shutil import multiprocessing import traceback -SSH_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") +SSH_KEY = open(SSH_KEY_FILE).read() +SSH_PUB_KEY = open(SSH_PUB_KEY_FILE).read() + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE = { + 'block' : "-drive file={},if=none,id=drive0,cache=writeback "\ + "-device virtio-blk,drive=drive0,bootindex=0", + 'scsi' : "-device virtio-scsi-device,id=scsi "\ + "-drive file={},format=raw,if=none,id=hd0 "\ + "-device scsi-hd,drive=hd0,bootindex=0", +} class BaseVM(object): - GUEST_USER = "qemu" - GUEST_PASS = "qemupass" - ROOT_PASS = "qemupass" envvars = [ "https_proxy", @@ -59,19 +84,26 @@ class BaseVM(object): poweroff = "poweroff" # enable IPv6 networking ipv6 = True - def __init__(self, debug=False, vcpus=None): + def __init__(self, debug=False, vcpus=None, config=None): self._guest = None + # Allow input config to override defaults. + self._config = DEFAULT_CONFIG.copy() + if config != None: + self._config.update(config) self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-", suffix=".tmp", dir=".")) atexit.register(shutil.rmtree, self._tmpdir) - + self._config['ssh_key'] = \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] = \ + open(self._config['ssh_pub_key_file']).read().rstrip() self._ssh_key_file = os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) + open(self._ssh_key_file, "w").write(self._config['ssh_key']) subprocess.check_call(["chmod", "600", self._ssh_key_file]) self._ssh_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + open(self._ssh_pub_key_file, "w").write(self._config['ssh_pub_key']) self.debug = debug self._stderr = sys.stderr @@ -80,11 +112,14 @@ class BaseVM(object): self._stdout = sys.stdout else: self._stdout = self._devnull + netdev = "user,id=vnet,hostfwd=:127.0.0.1:{}-:22" self._args = [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" + - (",ipv6=no" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=no" if not self.ipv6 else "") + + (",dns=" + self._config['dns'] if self._config['dns'] else ""), "-device", "virtio-net-pci,netdev=vnet", "-vnc", "127.0.0.1:0,to=20"] if vcpus and vcpus > 1: @@ -95,6 +130,25 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args = [] + def wait_boot(self, wait_string=None): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string = self.login_prompt + self.console_init() + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=None, sha512sum=None): def check_sha256sum(fname): if not sha256sum: @@ -126,7 +180,8 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=" + os.devnull, - "-o", "ConnectTimeout=1", + "-o", + "ConnectTimeout={}".format(self._config["ssh_timeout"]), "-p", self.ssh_port, "-i", self._ssh_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. @@ -176,15 +231,15 @@ class BaseVM(object): "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)] def boot(self, img, extra_args=[]): - args = self._args + [ - "-device", "VGA", - "-drive", "file=%s,if=none,id=drive0,cache=writeback" % img, - "-device", "virtio-blk,drive=drive0,bootindex=0"] - args += self._data_args + extra_args + boot_dev = BOOT_DEVICE[self._config['boot_dev_type']] + boot_params = boot_dev.format(img) + args = self._args + boot_params.split(' ') + args += self._data_args + extra_args + self._config['extra_args'] + args += ["-device", "VGA"] logging.debug("QEMU args: %s", " ".join(args)) qemu_bin = os.environ.get("QEMU", "qemu-system-" + self.arch) guest = QEMUMachine(binary=qemu_bin, args=args) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -331,7 +386,6 @@ class BaseVM(object): def shutdown(self): self._guest.shutdown() - def wait(self): self._guest.wait() @@ -379,15 +433,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() -def main(vmcls): +def main(vmcls, config=None): try: + if config == None: + config = {} args, argv = parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) - vm = vmcls(debug=args.debug, vcpus=args.jobs) + vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.image, From patchwork Fri Jan 24 16:53:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350737 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 49FF46C1 for ; Fri, 24 Jan 2020 16:59:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 210E62072C for ; Fri, 24 Jan 2020 16:59:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="MwOm5ktl" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 210E62072C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:45026 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2J7-0002L8-C7 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:59:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45623) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2G1-0003sO-0D for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2Fz-00034o-Ii for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:32 -0500 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:34659) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2Fz-00034B-CP for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:31 -0500 Received: by mail-pg1-x544.google.com with SMTP id r11so1409434pgf.1 for ; Fri, 24 Jan 2020 08:56:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MwX8tXxSDd35fuy08I41Y0xncP6DuvbsklPwCAUTvwA=; b=MwOm5ktlM7+HK+md2zbDxrdFdZfbx2+qbTdvOyBSQA1OBh3wIeUNrRPCaZDSm9C9L6 /Nv1MerRi0T/0gl9mKv3FDe178tLcWOrgP3vLhpd7Qdj1D4Pim+MzjAEnm8aJb1oZsKT b34Ql4pzMWQDi33+s+zVzSe5u2eUJnZoK/DtiH5ozn8XpFjE7l8ovrNlCCRLKZsoVdNQ iwsP+oiAdhJ2eEr5QSfUJx0zyaQZwZVgUUkYEMB0EtQuPZtfPkWcFSw+1biFW9NNZC9P X84hDb9+D+wzMliptZf+lpqEPt3lDM9BTnuSVpslfdHLIcExtGh1t4bhipLG2Tetud8v +Fqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MwX8tXxSDd35fuy08I41Y0xncP6DuvbsklPwCAUTvwA=; b=oq9oXS0voZg9PXUU8R0uKQjBcYOCghbzQxu0mqc83pFLn1LJ987mV9Km5NxeU3U7JW Tnwg5Ow5UHoIkVF/6YFP5FprGN3LBK+rP/bgqr28nCQAG1gpCOH1Gb0xNVW+2mAAEYD/ iZGQ7pL8E2Po3zcoXk00Zk7ypHA7sl7pMHZEFfDVg+H5syOzJqWBSSWHcSJkAPqetz24 fToOq7p0QvkIHk9ui4R5MEicntEa5m5FqhfqLXmV9Ufn/S+55elVXFYwdE6mcpbvqBGu 7MjELj2H4H0UhivacZJlSvIQUcSizKvP8X/Pwc3oxUULwlqdRa5AhDe+OUG5rPtbfgDa B4dg== X-Gm-Message-State: APjAAAVGR6MfT74Z89Rlpp6Px/bWzrhfeCw6u1NNef2xvYkKxcjYrrLz 8jMo4CMT0B/dECj8w6Hyhg9tHZAbmi4= X-Google-Smtp-Source: APXvYqz6XTm6Q7cEUEXsTcSRzl3A9hx23I9QsHtea3slJvyXe3SiKJcRDeHVXLRku/Y1UDxsdZE6mA== X-Received: by 2002:a63:364f:: with SMTP id d76mr5175232pga.215.1579884990017; Fri, 24 Jan 2020 08:56:30 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:29 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 5/8] tests/vm: Added configuration file support Date: Fri, 24 Jan 2020 11:53:32 -0500 Message-Id: <20200124165335.422-6-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Changes to tests/vm/basevm.py to allow accepting a configuration file as a parameter. Allows for specifying VM options such as cpu, machine, memory, and arbitrary qemu arguments for specifying options such as NUMA configuration. Also added an example config_example.yml. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 60 +++++++++++++++++++++++++++++++++++++ tests/vm/config_example.yml | 52 ++++++++++++++++++++++++++++++++ 2 files changed, 112 insertions(+) create mode 100644 tests/vm/config_example.yml diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index ec92c8f105..08a8989ac0 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -31,6 +31,7 @@ import tempfile import shutil import multiprocessing import traceback +import yaml SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), "..", "keys", "id_rsa") @@ -396,6 +397,61 @@ class BaseVM(object): def qmp(self, *args, **kwargs): return self._guest.qmp(*args, **kwargs) + +def parse_config(config, args): + """ Parse yaml config and populate our config structure. + The yaml config allows the user to override the + defaults for VM parameters. In many cases these + defaults can be overridden without rebuilding the VM.""" + if args.config: + config_file = args.config + elif 'QEMU_CONFIG' in os.environ: + config_file = os.environ['QEMU_CONFIG'] + else: + return config + if not os.path.exists(config_file): + raise Exception("config file {} does not exist".format(config_file)) + with open(config_file) as f: + yaml_dict = yaml.safe_load(f) + if 'target-conf' in yaml_dict: + target_dict = yaml_dict['target-conf'] + if 'username' in target_dict and target_dict['username'] != 'root': + config['guest_user'] = target_dict['username'] + if 'password' in target_dict: + config['root_pass'] = target_dict['password'] + config['guest_pass'] = target_dict['password'] + if any (k in target_dict for k in ("ssh_key","ssh_pub_key")) and \ + not all (k in target_dict for k in ("ssh_key","ssh_pub_key")): + missing_key = "ssh_pub_key" \ + if 'ssh_key' in target_dict else "ssh_key" + raise Exception("both ssh_key and ssh_pub_key required. " + "{} key is missing.".format(missing_key)) + if 'ssh_key' in target_dict: + config['ssh_key_file'] = target_dict['ssh_key'] + if not os.path.exists(config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key' in target_dict: + config['ssh_pub_key_file'] = target_dict['ssh_pub_key'] + if not os.path.exists(config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + if 'machine' in target_dict: + config['machine'] = target_dict['machine'] + if 'qemu_args' in target_dict: + qemu_args = target_dict['qemu_args'] + qemu_args = qemu_args.replace('\n', ' ').replace('\r', '') + config['extra_args'] = qemu_args.split(' ') + if 'memory' in target_dict: + config['memory'] = target_dict['memory'] + if 'dns' in target_dict: + config['dns'] = target_dict['dns'] + if 'cpu' in target_dict: + config['cpu'] = target_dict['cpu'] + if 'ssh_port' in target_dict: + config['ssh_port'] = target_dict['ssh_port'] + if 'install_cmds' in target_dict: + config['install_cmds'] = target_dict['install_cmds'] + return config + def parse_args(vmcls): def get_default_jobs(): @@ -430,6 +486,9 @@ def parse_args(vmcls): help="Interactively run command") parser.add_option("--snapshot", "-s", action="store_true", help="run tests with a snapshot") + parser.add_option("--config", "-c", default=None, + help="Provide config yaml for configuration. "\ + "See config_example.yaml for example.") parser.disable_interspersed_args() return parser.parse_args() @@ -441,6 +500,7 @@ def main(vmcls, config=None): if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 + config = parse_config(config, args) logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config) diff --git a/tests/vm/config_example.yml b/tests/vm/config_example.yml new file mode 100644 index 0000000000..0a1fec3824 --- /dev/null +++ b/tests/vm/config_example.yml @@ -0,0 +1,52 @@ +# +# Example yaml for use by any of the scripts in tests/vm. +# Can be provided as an environment variable QEMU_CONFIG +# +target-conf: + + # If any of the below are not provided, we will just use the qemu defaults. + + # Login username (has to be sudo enabled) + #username: qemu + + # Password is used by root and default login user. + #password: "qemupass" + + # If one key is provided, both must be provided. + #ssh_key: /complete/path/of/your/keyfile/id_rsa + #ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub + + cpu: max + machine: virt,gic_version=3 + memory: 16G + + # The below is an example for how to configure NUMA topology with + # 4 NUMA nodes and 2 different NUMA distances. + qemu_args: "-smp cpus=16,sockets=2,cores=8 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node0 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=0,id=ram-node1 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node2 + -object memory-backend-ram,size=4G,policy=bind,host-nodes=1,id=ram-node3 + -numa node,memdev=ram-node0,cpus=0-3,nodeid=0 -numa node,memdev=ram-node1,cpus=4-7,nodeid=1 + -numa node,memdev=ram-node2,cpus=8-11,nodeid=2 -numa node,memdev=ram-node3,cpus=12-15,nodeid=3 + -numa dist,src=0,dst=1,val=15 -numa dist,src=2,dst=3,val=15 + -numa dist,src=0,dst=2,val=20 -numa dist,src=0,dst=3,val=20 + -numa dist,src=1,dst=2,val=20 -numa dist,src=1,dst=3,val=20" + + # By default we do not set the DNS. + # You override the defaults by setting the below. + #dns: 1.234.567.89 + + # By default we will use a "block" device, but + # you can also boot from a "scsi" device. + # Just keep in mind your scripts might need to change + # As you will have /dev/sda instead of /dev/vda (for block device) + #boot_dev_type: "scsi" + + # By default the ssh port is not fixed. + # A fixed ssh port makes it easier for automated tests. + #ssh_port: 5555 + + # To install a different set of packages, provide a command to issue + #install_cmds: "apt-get update ; apt-get build-dep -y qemu" + From patchwork Fri Jan 24 16:53:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0FBC13A4 for ; Fri, 24 Jan 2020 16:57:16 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 878CC20709 for ; Fri, 24 Jan 2020 16:57:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="B8Upu9Im" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 878CC20709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Gh-0005hf-Nk for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:57:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45639) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2G2-0003vZ-9o for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2G1-000362-9z for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:34 -0500 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]:40772) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2G1-00035Z-4u for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:33 -0500 Received: by mail-pf1-x443.google.com with SMTP id q8so1395774pfh.7 for ; Fri, 24 Jan 2020 08:56:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WJEd3YfXjo1+GH81ylCClGb6F9dkMQFDTXt8V62PTH0=; b=B8Upu9ImBNxuthR0WIU+ChO581OZZ9jYelqOfnmIphEBewFzsneWArREzFHYDFnZBr M6ZHQlv03vXxkm9nl3D3BYZ8Dx+aMPm/SppzQCTES3M5ycP2JX+8VxRIQXTvKN3JLHDL e1KPCqwKhLqsm8LvjziG65PcImQ5hIZKNny87NZStwcRvP/kVhYf4CrTjLGoKmRN3wOb nO6zkdIgX+T7m1YYBUdh0unMsbC4MhcyGecy9eEib0wW2AGq0GsLVXT2FZ0NQU7+bLoV fqPRHxzBoESnvcHPExx5f1JYSGOu2Zr12iITNJdErACL4EcY87QcsCShlPhrQJxFn+BM aQtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WJEd3YfXjo1+GH81ylCClGb6F9dkMQFDTXt8V62PTH0=; b=lvy/OXIOUrWg3bdBpIOAiwkz4SFizrjLEkQER7Dw55DCV50jNru9LhP6cOIhUrRwcL 5HRl+2C051Neo/qQ4ujaJXaRcUI8Hz06sH4uoQQYRowxynGcdpjy2H7AksV+hZGJFByF dGbpltREFIpRt5t5ylXfjHnfrg264Y8Q9E3bY81Id+MWCUwNny1pBHBjDX2oDpq7ZfkA zKyaX0lDCsHYPLrQCiXvL8yyktqz/YcbGQc922TmkbzEgOJkq9PCduWJUIj7d2G/rJ6V SoWCBIZO0enZhwBZNkRzECGAQouy+GnYry76dmP6qFJDUfzpWvfoRxnSEH155gZu/Vx/ pOOQ== X-Gm-Message-State: APjAAAU9hbkhCEZCN/+qbz+qYQU8neTqHRF+GCvyuh1noD+2wwwI4FGq OYfgPuqXcVNmuzDR7tInY7SDXVVB30Y= X-Google-Smtp-Source: APXvYqxeTcgQw+4IQ7Dfd5nDgG24UfbC583HwwRSRca7JnW3qNUENYTKgL/0u76dD8zCzbBvKGS4kA== X-Received: by 2002:a65:4109:: with SMTP id w9mr4980904pgp.383.1579884991898; Fri, 24 Jan 2020 08:56:31 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:31 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 6/8] tests/vm: add --boot-console switch Date: Fri, 24 Jan 2020 11:53:33 -0500 Message-Id: <20200124165335.422-7-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::443 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Added ability to view console during boot via --boot-console switch. This helps debug issues that occur during the boot sequence. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/basevm.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 08a8989ac0..aa8b39beb7 100755 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -489,6 +489,8 @@ def parse_args(vmcls): parser.add_option("--config", "-c", default=None, help="Provide config yaml for configuration. "\ "See config_example.yaml for example.") + parser.add_option("--boot-console", action="store_true", + help="Show console during boot. ") parser.disable_interspersed_args() return parser.parse_args() @@ -523,6 +525,10 @@ def main(vmcls, config=None): if args.snapshot: img += ",snapshot=on" vm.boot(img) + wait_boot = getattr(vm, "wait_boot", None) + if args.boot_console and callable(wait_boot): + vm.console_init() + wait_boot() vm.wait_ssh() except Exception as e: if isinstance(e, SystemExit) and e.code == 0: From patchwork Fri Jan 24 16:53:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350731 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A29E8138C for ; Fri, 24 Jan 2020 16:58:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 67ABC20709 for ; Fri, 24 Jan 2020 16:58:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="aW2nE3Xc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67ABC20709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:44982 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2Ht-0008R9-Ew for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:58:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45653) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2G5-00044c-GC for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2G3-000378-Gh for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:37 -0500 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]:44763) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2G3-00036Y-8q for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:35 -0500 Received: by mail-pf1-x42a.google.com with SMTP id 62so1385028pfu.11 for ; Fri, 24 Jan 2020 08:56:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=caK2mLT/TcZir8e4gOeKsOqEnAOnx33niuYDtDOksDI=; b=aW2nE3Xc3Q3q9wkRhAj4PfEuVaxvGFPnMjrl7BdXQTyOVjWGKe6F9j+S5XInmko+GA hAl4t1828P9699ufEK7iGIX/2ki4VJPTQ3bBlL9NutgkhBKwooNHlCsTYt7GN/wD/HpK GshPVroPqyk7K4FN6Z4P3wQUTigPnX5jlYjcpmMHy9G6KAMGRI8V5gf4p1WqK5AG/DjQ EwhJCnNEIj8hUCoeGvKlAY/uREth1GWm6Yse4HMo5KsuzeqnBxhhDhuitDjb2oAOlvHL 5OMDrtM547W4BVxNhUeUH0hzTnhQ4XS2r4XgkaUZihI80hMbU4bkypOP2Lr/zbIrWdxN Q+yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=caK2mLT/TcZir8e4gOeKsOqEnAOnx33niuYDtDOksDI=; b=D7g03+KxNNiHkwPKqhSECTqTIO19YaJFsoeljtIpEMG21RY2Dz4Ell4nxGu8HLg6o5 ya1nAir9Wy2p6p2UW9oAciRJfJ+z2IlqTLNSnVQCiGPwkhq30tpzt1SJvVLyB9c+EPzW Y7tJzhRf4Hh4KY94JnRIJEKx4dEOU32WYpeBpN4i3copTn/aBJ80c9R7UDIW9a1yHbiA MbIdPW9ukk3Y0w5Gvtt36/dCptcxe4a4POuua1zmxhxAYz+zxpTYeXR20Q/016Ag46+g hjUlO536LiIIugj8VQYS+DhGnd2VLRKuMVzCMk9+WvQnidGNt4+fbiTdptO7Gqu1WSgF +5lw== X-Gm-Message-State: APjAAAWgNE/qKIk3RmfIrYAm6m984qpuIuTwQHEH9vkkAgYCbYbrA3D7 HsE26XvM0AigXSUhJEz5XFbs9lQpDao= X-Google-Smtp-Source: APXvYqyz1Pom4Eb2bwkMvAF7qYQelfvNrq8sf5W4IkbW/kgj6kmWrtsZt/SSq9rBMIY+/pNDl0JgRg== X-Received: by 2002:a63:1756:: with SMTP id 22mr5175326pgx.109.1579884993843; Fri, 24 Jan 2020 08:56:33 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:33 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 7/8] tests/vm: Added a new script for ubuntu.aarch64. Date: Fri, 24 Jan 2020 11:53:34 -0500 Message-Id: <20200124165335.422-8-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42a X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" ubuntu.aarch64 provides a script to create an Ubuntu 18.04 VM. Another new file is also added aarch64vm.py, which is a module with common methods used by aarch64 VMs, such as how to create the flash images. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 7 +- tests/vm/aarch64vm.py | 41 +++++++++++ tests/vm/ubuntu.aarch64 | 144 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 190 insertions(+), 2 deletions(-) create mode 100644 tests/vm/aarch64vm.py create mode 100755 tests/vm/ubuntu.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 9e7c46a473..966b417ba7 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -2,7 +2,7 @@ .PHONY: vm-build-all vm-clean-all -IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora +IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64 IMAGES_DIR := $(HOME)/.cache/qemu-vm/images IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES)) @@ -18,6 +18,7 @@ vm-help vm-test: @echo " vm-build-openbsd - Build QEMU in OpenBSD VM" @echo " vm-build-centos - Build QEMU in CentOS VM, with Docker" @echo " vm-build-fedora - Build QEMU in Fedora VM" + @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 VM" @echo "" @echo " vm-build-all - Build QEMU in all VMs" @echo " vm-clean-all - Clean up VM images" @@ -35,6 +36,8 @@ vm-help vm-test: @echo " V=1 - Enable verbose ouput on host and guest commands" @echo " QEMU=/path/to/qemu - Change path to QEMU binary" @echo " QEMU_IMG=/path/to/qemu-img - Change path to qemu-img tool" + @echo " QEMU_CONFIG=/path/conf.yml - Change path to VM configuration .yml file." + @echo " See config_example.yml for file format details." vm-build-all: $(addprefix vm-build-, $(IMAGES)) @@ -80,7 +83,7 @@ vm-boot-serial-%: $(IMAGES_DIR)/%.img vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(call quiet-command, \ - $(SRC_PATH)/tests/vm/$* \ + $(PYTHON) $(SRC_PATH)/tests/vm/$* \ $(if $(J),--jobs $(J)) \ --image "$<" \ --interactive \ diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py new file mode 100644 index 0000000000..43f841571f --- /dev/null +++ b/tests/vm/aarch64vm.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python +# +# VM testing aarch64 library +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# +import os +import sys +import subprocess +import basevm + + +def create_flash_images(): + """Creates the appropriate pflash files + for an aarch64 VM.""" + subprocess.check_call(["dd", "if=/dev/zero", "of=flash0.img", + "bs=1M", "count=64"]) + # A reliable way to get the QEMU EFI image is via an installed package. + efi_img = "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" + if not os.path.exists(efi_img): + sys.stderr.write("*** {} is missing\n".format(efi_img)) + sys.stderr.write("*** please install qemu-efi-aarch64 package\n") + exit(3) + subprocess.check_call(["dd", "if={}".format(efi_img), + "of=flash0.img", "conv=notrunc"]) + subprocess.check_call(["dd", "if=/dev/zero", + "of=flash1.img", "bs=1M", "count=64"]) + +def get_pflash_args(): + """Returns a string that can be used to + boot qemu using the appropriate pflash files + for aarch64.""" + pflash_args = "-drive file=flash0.img,format=raw,if=pflash "\ + "-drive file=flash1.img,format=raw,if=pflash" + return pflash_args.split(" ") diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64 new file mode 100755 index 0000000000..941f7f5166 --- /dev/null +++ b/tests/vm/ubuntu.aarch64 @@ -0,0 +1,144 @@ +#!/usr/bin/env python +# +# Ubuntu aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.i386 Fam Zheng +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +import time +import aarch64vm + +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : "virt,gic-version=max", + 'install_cmds' : "apt-get update,"\ + "apt-get build-dep -y qemu,"\ + "apt-get install -y libfdt-dev flex bison", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM + # especially using softmmu. + 'ssh_timeout' : 60, +} + +class UbuntuAarch64VM(basevm.BaseVM): + name = "ubuntu.aarch64" + arch = "aarch64" + image_name = "ubuntu-18.04-server-cloudimg-arm64.img" + image_link = "https://cloud-images.ubuntu.com/releases/18.04/release/" + image_name + login_prompt = "ubuntu-guest login:" + BUILD_SCRIPT = """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def _gen_cloud_init_iso(self): + cidir = self._tmpdir + mdata = open(os.path.join(cidir, "meta-data"), "w") + mdata.writelines(["instance-id: ubuntu-vm-0\n", + "local-hostname: ubuntu-guest\n"]) + mdata.close() + udata = open(os.path.join(cidir, "user-data"), "w") + print("guest user:pw {}:{}".format(self.GUEST_USER, self.GUEST_PASS)) + udata.writelines(["#cloud-config\n", + "chpasswd:\n", + " list: |\n", + " root:%s\n" % self.ROOT_PASS, + " %s:%s\n" % (self.GUEST_USER, self.GUEST_PASS), + " expire: False\n", + "users:\n", + " - name: %s\n" % self.GUEST_USER, + " sudo: ALL=(ALL) NOPASSWD:ALL\n", + " ssh-authorized-keys:\n", + " - %s\n" % self.ssh_pub_key, + " - name: root\n", + " ssh-authorized-keys:\n", + " - %s\n" % self.ssh_pub_key, + "locale: en_US.UTF-8\n"]) + proxy = os.environ.get("http_proxy") + if not proxy is None: + udata.writelines(["apt:\n", + " proxy: %s" % proxy]) + udata.close() + subprocess.check_call(["genisoimage", "-output", "cloud-init.iso", + "-volid", "cidata", "-joliet", "-rock", + "user-data", "meta-data"], + cwd=cidir, + stdin=self._devnull, stdout=self._stdout, + stderr=self._stdout) + + return os.path.join(cidir, "cloud-init.iso") + + def boot(self, img, extra_args=None): + aarch64vm.create_flash_images() + default_args = aarch64vm.get_pflash_args() + if extra_args: + extra_args.extend(default_args) + else: + extra_args = default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if os.geteuid() != 0: + extra_args.extend(["-accel", "tcg,thread=multi"]) + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to change it. + extra_args.extend(["-smp", "8"]) + + # We have overridden boot() since aarch64 has additional parameters. + # Call down to the base class method. + super(UbuntuAarch64VM, self).boot(img, extra_args=extra_args) + + def build_image(self, img): + os_img = self._download_with_cache(self.image_link) + img_tmp = img + ".tmp" + subprocess.check_call(["cp", "-f", os_img, img_tmp]) + subprocess.check_call(["qemu-img", "resize", img_tmp, "+50G"]) + ci_img = self._gen_cloud_init_iso() + + self.boot(img_tmp, extra_args = ["-cdrom", ci_img]) + self.wait_ssh(wait_root=True) + # Fix for slow ssh login. + self.ssh_root("chmod -x /etc/update-motd.d/*") + self.ssh_root("touch /etc/cloud/cloud-init.disabled") + # Disable auto upgrades. + # We want to keep the VM system state stable. + self.ssh_root('sed -ie \'s/"1"/"0"/g\' /etc/apt/apt.conf.d/20auto-upgrades') + + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] != "": + # Don't check the status in case the guest hang up too quickly + self.ssh_root("sync && reboot") + + self.wait_ssh(wait_root=True) + # The previous update sometimes doesn't survive a reboot, so do it again + self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g /etc/apt/sources.list") + + # Issue the install commands. + # This can be overriden by the user in the config .yml. + install_cmds = self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.graceful_shutdown() + self.wait() + os.rename(img_tmp, img) + return 0 + +if __name__ == "__main__": + sys.exit(basevm.main(UbuntuAarch64VM, DEFAULT_CONFIG)) From patchwork Fri Jan 24 16:53:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Foley X-Patchwork-Id: 11350735 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A05A138C for ; Fri, 24 Jan 2020 16:59:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F389020709 for ; Fri, 24 Jan 2020 16:59:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="MEkBXgM9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F389020709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:45022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2J7-0002Km-73 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 24 Jan 2020 11:59:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45667) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iv2G7-0004C6-Vs for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iv2G5-00038P-Rn for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:39 -0500 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]:45513) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iv2G5-000384-JV for qemu-devel@nongnu.org; Fri, 24 Jan 2020 11:56:37 -0500 Received: by mail-pl1-x62d.google.com with SMTP id b22so1016380pls.12 for ; Fri, 24 Jan 2020 08:56:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tVehXXV9NmGNmPr1T8r1ve+muL/OjkWOWDZ8k9HvDtY=; b=MEkBXgM9gpRMnleMwI2Y0KVajKQeYgsQi+T18x5tu81YqioX0T3gBY3gseE8MBoKNB EzRSfaWkmK4JmnW97JUBkNUsdiXvkL4zkj4TtDmwjgqyJviIPGUQ+j9AeeJCPsMmer9n B7ZsRLCZI14HuIx2COPtkN2jtT6DkEXm7RyfmGYE3QUCeQ4VULPl+58TOSiPkLjFHbiY mZd/gR8L+KhlCc3ZJ+6j2nGiYXRm0nzFsu+Vc8HKiRhL7RUVsfMWULOm8Z+CDCG9cvtr itWFvgM9qNsWf/h/8VD8s8xKfa7RL2S6LtryfUIIYAx6iNi1iXa+yJA3Czyjy9Ur9vuy u1Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tVehXXV9NmGNmPr1T8r1ve+muL/OjkWOWDZ8k9HvDtY=; b=aZK00C/zfPBS2E8v/8X+2QnMv596Kkz8UOHHF3nfNH554mw2cKMcRhN1LmXjz0Q6yt PJOQrKPzmGyXZoANypaVJg639Tcwm9kNDLTnflZIvR84yv8Brgh7r/K366IeCgawCu8o dWnqLGpEuZtscW7QzxM6h9e+AvKFbOGd8dK/wt27T6cU60qXyAq/HY9wbajNsbnUBQjv emf6odTxP5KYvzZYinkL/+zKsYweAdseAQiMWlsJ0AVPu5xvi82KuMliqwsMG7y22MMY eY/DQldIvj6zt0nB2gqXQXK6Bj8TTLRyqmkjJLd2jMdPHgPCDZQOuWxWNslL0lXlo49l mCUQ== X-Gm-Message-State: APjAAAUu0WiPONmgh8Y8Y+quNbmmebjJ2iKLLKMlCvLmX3tn36cU/f/8 uHn0ViWns+vTtdKXcG/s0oQd2fOCiVg= X-Google-Smtp-Source: APXvYqySEyzxM3oUBE/mqwTCTB9/X0ZWxPvVsjUwufEGjcQw1poVeX3HQB1he2xO3bVqV1+m57+/1w== X-Received: by 2002:a17:90a:bf92:: with SMTP id d18mr164304pjs.21.1579884996032; Fri, 24 Jan 2020 08:56:36 -0800 (PST) Received: from Rfoley-MA01.usrd.futurewei.com ([12.111.81.71]) by smtp.gmail.com with ESMTPSA id l10sm6969953pjy.5.2020.01.24.08.56.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jan 2020 08:56:35 -0800 (PST) From: Robert Foley To: qemu-devel@nongnu.org Subject: [PATCH 8/8] tests/vm: Added a new script for centos.aarch64. Date: Fri, 24 Jan 2020 11:53:35 -0500 Message-Id: <20200124165335.422-9-robert.foley@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200124165335.422-1-robert.foley@linaro.org> References: <20200124165335.422-1-robert.foley@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62d X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: fam@euphon.net, philmd@redhat.com, alex.bennee@linaro.org, robert.foley@linaro.org, peter.puhov@linaro.org Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" centos.aarch64 creates a CentOS 8 image. Also added a new kickstart script used to build the centos.aarch64 image. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov --- tests/vm/Makefile.include | 3 +- tests/vm/centos-8-aarch64.ks | 52 +++++++++ tests/vm/centos.aarch64 | 218 +++++++++++++++++++++++++++++++++++ 3 files changed, 272 insertions(+), 1 deletion(-) create mode 100644 tests/vm/centos-8-aarch64.ks create mode 100755 tests/vm/centos.aarch64 diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include index 966b417ba7..febf82fe16 100644 --- a/tests/vm/Makefile.include +++ b/tests/vm/Makefile.include @@ -2,7 +2,7 @@ .PHONY: vm-build-all vm-clean-all -IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64 +IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64 centos.aarch64 IMAGES_DIR := $(HOME)/.cache/qemu-vm/images IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES)) @@ -19,6 +19,7 @@ vm-help vm-test: @echo " vm-build-centos - Build QEMU in CentOS VM, with Docker" @echo " vm-build-fedora - Build QEMU in Fedora VM" @echo " vm-build-ubuntu.aarch64 - Build QEMU in ubuntu aarch64 VM" + @echo " vm-build-centos.aarch64 - Build QEMU in CentOS aarch64 VM" @echo "" @echo " vm-build-all - Build QEMU in all VMs" @echo " vm-clean-all - Clean up VM images" diff --git a/tests/vm/centos-8-aarch64.ks b/tests/vm/centos-8-aarch64.ks new file mode 100644 index 0000000000..5f6093fefa --- /dev/null +++ b/tests/vm/centos-8-aarch64.ks @@ -0,0 +1,52 @@ +# CentOS aarch64 image kickstart file. +# This file is used by the CentOS installer to +# script the generation of the image. +# +# Copyright 2020 Linaro +# +ignoredisk --only-use=vda +# System bootloader configuration +bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda +autopart --type=plain +# Partition clearing information +clearpart --linux --initlabel --drives=vda +# Use text mode install +text +repo --name="AppStream" --baseurl=file:///run/install/repo/AppStream +# Use CDROM installation media +cdrom +# Keyboard layouts +keyboard --vckeymap=us --xlayouts='' +# System language +lang en_US.UTF-8 + +# Network information +network --bootproto=dhcp --device=enp0s1 --onboot=off --ipv6=auto --no-activate +network --hostname=localhost.localdomain +# Run the Setup Agent on first boot +firstboot --enable +# Do not configure the X Window System +skipx +# System services +services --enabled="chronyd" +# System timezone +timezone America/New_York --isUtc + +# Shutdown after installation is complete. +shutdown + +%packages +@^server-product-environment +kexec-tools + +%end + +%addon com_redhat_kdump --enable --reserve-mb='auto' + +%end +%anaconda +pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty +pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok +pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty +%end + diff --git a/tests/vm/centos.aarch64 b/tests/vm/centos.aarch64 new file mode 100755 index 0000000000..d939c2a900 --- /dev/null +++ b/tests/vm/centos.aarch64 @@ -0,0 +1,218 @@ +#!/usr/bin/env python +# +# Centos aarch64 image +# +# Copyright 2020 Linaro +# +# Authors: +# Robert Foley +# Originally based on ubuntu.aarch64 +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +import os +import sys +import subprocess +import basevm +import time +import traceback +import aarch64vm + +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : "virt,gic-version=max", + 'install_cmds' : "yum install -y docker make git python3 gcc, "\ + "yum install -y glib2-devel pixman-devel zlib-devel, "\ + "yum install -y perl-Test-Harness, "\ + "systemctl enable docker", + # We increase beyond the default time since during boot + # it can take some time (many seconds) to log into the VM. + 'ssh_timeout' : 60, +} + +class CentosAarch64VM(basevm.BaseVM): + name = "centos.aarch64" + arch = "aarch64" + login_prompt = "localhost login:" + prompt = '[root@localhost ~]#' + image_name = "CentOS-8-aarch64-1905-dvd1.iso" + image_link = "http://mirrors.usc.edu/pub/linux/distributions/centos/8.0.1905/isos/aarch64/" + image_link += image_name + BUILD_SCRIPT = """ + set -e; + cd $(mktemp -d); + sudo chmod a+r /dev/vdb; + tar --checkpoint=.10 -xf /dev/vdb; + ./configure {configure_opts}; + make --output-sync {target} -j{jobs} {verbose}; + """ + def set_key_perm(self): + """Set permissions properly on certain files to allow + ssh access.""" + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v /root/.ssh\n") + self.console_wait_send(self.prompt, + "/usr/sbin/restorecon -R -v "\ + "/home/{}/.ssh\n".format(self.GUEST_USER)) + + def create_kickstart(self): + """Generate the kickstart file used to generate the centos image.""" + # Start with the template for the kickstart. + ks_file = "../tests/vm/centos-8-aarch64.ks" + subprocess.check_call("cp {} ./ks.cfg".format(ks_file), shell=True) + # Append the ssh keys to the kickstart file + # as the post processing phase of installation. + with open("ks.cfg", "a") as f: + # Add in the root pw and guest user. + f.write("rootpw --plaintext {}\n".format(self.ROOT_PASS)) + add_user = "user --groups=wheel --name={} "\ + "--password={} --plaintext\n" + f.write(add_user.format(self.GUEST_USER, self.GUEST_PASS)) + # Add the ssh keys. + f.write("%post --log=/root/ks-post.log\n") + f.write("mkdir -p /root/.ssh\n") + addkey = 'echo "{}" >> /root/.ssh/authorized_keys\n' + addkey_cmd = addkey.format(self.ssh_pub_key) + f.write(addkey_cmd) + f.write('mkdir -p /home/{}/.ssh\n'.format(self.GUEST_USER)) + addkey = 'echo "{}" >> /home/{}/.ssh/authorized_keys\n' + addkey_cmd = addkey.format(self.ssh_pub_key, self.GUEST_USER) + f.write(addkey_cmd) + f.write("%end\n") + # Take our kickstart file and create an .iso from it. + # The .iso will be provided to qemu as we boot + # from the install dvd. + # Anaconda will recognize the label "OEMDRV" and will + # start the automated installation. + gen_iso_img = 'genisoimage -output ks.iso -volid "OEMDRV" ks.cfg' + subprocess.check_call(gen_iso_img, shell=True) + + def wait_for_shutdown(self): + """We wait for qemu to shutdown the VM and exit. + While this happens we display the console view + for easier debugging.""" + # The image creation is essentially done, + # so whether or not the wait is successful we want to + # wait for qemu to exit (the self.wait()) before we return. + try: + self.console_wait("reboot: Power down") + except Exception as e: + sys.stderr.write("Exception hit\n") + if isinstance(e, SystemExit) and e.code == 0: + return 0 + traceback.print_exc() + finally: + self.wait() + + def build_base_image(self, dest_img): + """Run through the centos installer to create + a base image with name dest_img.""" + # We create the temp image, and only rename + # to destination when we are done. + img = dest_img + ".tmp" + # Create an empty image. + # We will provide this as the install destination. + qemu_img_create = "qemu-img create {} 50G".format(img) + subprocess.check_call(qemu_img_create, shell=True) + + # Create our kickstart file to be fed to the installer. + self.create_kickstart() + # Boot the install dvd with the params as our ks.iso + os_img = self._download_with_cache(self.image_link) + dvd_iso = "centos-8-dvd.iso" + subprocess.check_call(["cp", "-f", os_img, dvd_iso]) + extra_args = "-cdrom ks.iso" + extra_args += " -drive file={},if=none,id=drive1,cache=writeback" + extra_args += " -device virtio-blk,drive=drive1,bootindex=1" + extra_args = extra_args.format(dvd_iso).split(" ") + self.boot(img, extra_args=extra_args) + self.console_wait_send("change the selection", "\n") + # We seem to need to hit esc (chr(27)) twice to abort the + # media check, which takes a long time. + # Waiting a bit seems to be more reliable before hitting esc. + self.console_wait("Checking") + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + time.sleep(5) + self.console_wait_send("Checking", chr(27)) + print("Found Checking") + self.wait_for_shutdown() + os.rename(img, dest_img) + print("Done with base image build: {}".format(dest_img)) + + def check_create_base_img(self, img_base, img_dest): + """Create a base image using the installer. + We will use the base image if it exists. + This helps cut down on install time in case we + need to restart image creation, + since the base image creation can take a long time.""" + if not os.path.exists(img_base): + print("Generate new base image: {}".format(img_base)) + self.build_base_image(img_base); + else: + print("Use existing base image: {}".format(img_base)) + # Save a copy of the base image and copy it to dest. + # which we will use going forward. + subprocess.check_call(["cp", img_base, img_dest]) + + def boot(self, img, extra_args=None): + aarch64vm.create_flash_images() + default_args = aarch64vm.get_pflash_args() + if extra_args: + extra_args.extend(default_args) + else: + extra_args = default_args + # We always add these performance tweaks + # because without them, we boot so slowly that we + # can time out finding the boot efi device. + if os.geteuid() != 0: + extra_args.extend(["-accel", "tcg,thread=multi"]) + if '-smp' not in extra_args and \ + '-smp' not in self._config['extra_args'] and \ + '-smp' not in self._args: + # Only add if not already there to give caller option to change it. + extra_args.extend(["-smp", "8"]) + # We have overridden boot() since aarch64 has additional parameters. + # Call down to the base class method. + super(CentosAarch64VM, self).boot(img, extra_args=extra_args) + + def build_image(self, img): + img_tmp = img + ".tmp" + self.check_create_base_img(img + ".base", img_tmp) + + # Boot the new image for the first time to finish installation. + self.boot(img_tmp) + self.console_init() + self.console_wait_send(self.login_prompt, "root\n") + self.console_wait_send("Password:", "{}\n".format(self.ROOT_PASS)) + + self.set_key_perm() + self.console_wait_send(self.prompt, "rpm -q centos-release\n") + enable_adapter = "sed -i 's/ONBOOT=no/ONBOOT=yes/g'" \ + " /etc/sysconfig/network-scripts/ifcfg-enp0s1\n" + self.console_wait_send(self.prompt, enable_adapter) + self.console_wait_send(self.prompt, "ifup enp0s1\n") + self.console_wait_send(self.prompt, + 'echo "qemu ALL=(ALL) NOPASSWD:ALL" | '\ + 'sudo tee /etc/sudoers.d/qemu\n') + self.console_wait(self.prompt) + + # Rest of the commands we issue through ssh. + self.wait_ssh(wait_root=True) + + # If the user chooses *not* to do the second phase, + # then we will jump right to the graceful shutdown + if self._config['install_cmds'] != "": + install_cmds = self._config['install_cmds'].split(',') + for cmd in install_cmds: + self.ssh_root(cmd) + self.ssh_root("poweroff") + self.wait_for_shutdown() + os.rename(img_tmp, img) + print("image creation complete: {}".format(img)) + return 0 + +if __name__ == "__main__": + sys.exit(basevm.main(CentosAarch64VM, DEFAULT_CONFIG))