From patchwork Wed Mar 12 05:17:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 14012885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 29861C28B28 for ; Wed, 12 Mar 2025 05:18:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tsETc-0002lu-QS; Wed, 12 Mar 2025 01:17:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tsETZ-0002lY-Ql for qemu-devel@nongnu.org; Wed, 12 Mar 2025 01:17:53 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tsETX-00070N-Qn for qemu-devel@nongnu.org; Wed, 12 Mar 2025 01:17:53 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-22423adf751so92748545ad.2 for ; Tue, 11 Mar 2025 22:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741756670; x=1742361470; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=22gBc91mwFFQvV5wLMuqbrX+Fd3pyudRp4NuphL9yhY=; b=HRGWXmVettPypDhSX3h3XTKi7BhlM+Sya1jzot8uMlzrj0s3/vfTZfJWtVcOFHMUfs JwRg0SAMyo9b9lXfjdcGI6dZvYY2mI+cKSxXjunSU3Fj3R8/gKTydJGJ06bjcsd66yp+ AvVZBTfZ+JhQa+L90rv6ZVyrfP24MdquDN/UYirBNQrKM+pPjuGNFZ6shJpWqCddDFGl gfEoabQVTQvzZ+aFJdjACOzfhBSrNSRvNAm/LCTx5+X801XaVf9ATl2ERigTeC/Sh3gS QKt9gfJKM1/BarbZ7GZ32xQjZjbT4qDyVZhe3nB6Ok2wdsDml/kDjfDMyQh4NJc96YLd 5ufA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741756670; x=1742361470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=22gBc91mwFFQvV5wLMuqbrX+Fd3pyudRp4NuphL9yhY=; b=GPYUyzcTA4Bo6PbiU3Twg7PPrXQuCvBRPEPQyt4ESEyr0SvdSP3Na9rCLDGEIRSXhI 8700j1RmtkrcfPT8XSMAwSKrjGJt0GKenDW0iTJT+CIORxQM/kGededtJ4He12VPtwDG 3pB17CaF2TSHWTTOhse2waZSlP3RFNCZoX1N6HPVogiWDT0Yx5pUid3NjEOk9vGZT3mR l2R08ul8cm4Lj0fXJv8UF17dy0fwFJl/4s65i1eAR3sYXkUBB4X62kYUZXCyqViXTHvM JwY48/Mstgmkg+9ak4D7+M5uk3/Bzc40xdGHTjhxf5OENKqM0fnqSIOR4wHb5lFRr9ip u//w== X-Forwarded-Encrypted: i=1; AJvYcCWWaUZuUXeSE7lEmkHWCMGTnyUZ0noB4WLEhRoSuM1V5jd2GK8rrGArD0NExFT1/+rsPZ4SYfElhzI/@nongnu.org X-Gm-Message-State: AOJu0Yzt0S0ZvE02Q2jzHWiObAx0oV8qf2BycHwWNdsa0aWeBc6JoRAr 6c0FMPZQO8y2GRgbmoXpdvEOmJOKv6l5CHWhRMJfWLOEUQBP4QNG X-Gm-Gg: ASbGncspxre34RF8u0kdVUxkLfVIeJ4X0iMsPnV8SSQPia4MhymuhoZ+i5LDsSuUWZ2 3ZvM9mY3+Ll4ia3q8crN3cIu63PiHpoq1IQ0xGklUC0mfTWYD4/Kym9ZcpOI4FJv0c6dvCgeeAI pPCMaPK1WigatRQojIEZ2U1seU6Kr+k8XgeGVfLqca9ydNN4WOClCIrtefoTRGwXcjaWQHPzf5d 3lNu8tMQK44li59pOxXL28tIkS1yqtmoaL7BOB9pES/odlHm8/RYmxBTeyFX5sziz0E3bcbJaeW YzhtYtK0/TFNwvdPAorCRulS9gwCu6dnhm7Mw6w/CctOYKsBNqs= X-Google-Smtp-Source: AGHT+IH2zK7tUKhxahJAD5yTsMG/ZTpr9Ybk4JbbtanHglIvCrP6ishCa0f8RqkTaoElO0u7VqtWnw== X-Received: by 2002:a17:903:2308:b0:223:501c:7576 with SMTP id d9443c01a7336-22592e2d575mr95437295ad.12.1741756670512; Tue, 11 Mar 2025 22:17:50 -0700 (PDT) Received: from wheely.local0.net ([118.208.151.101]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22410a925ddsm107557475ad.203.2025.03.11.22.17.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Mar 2025 22:17:50 -0700 (PDT) From: Nicholas Piggin To: Thomas Huth Cc: Nicholas Piggin , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , qemu-devel@nongnu.org Subject: [PATCH 1/3] tests/functional/asset: Fail assert fetch when retries are exceeded Date: Wed, 12 Mar 2025 15:17:36 +1000 Message-ID: <20250312051739.938441-2-npiggin@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250312051739.938441-1-npiggin@gmail.com> References: <20250312051739.938441-1-npiggin@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=npiggin@gmail.com; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently the fetch code does not fail gracefully when retry limit is exceeded, it just falls through the loop with no file, which ends up hitting other errors. In preparation for adding more cases where a download gets retried, add an explicit check for retry limit exceeded. Signed-off-by: Nicholas Piggin Reviewed-by: Thomas Huth --- tests/functional/qemu_test/asset.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index f0730695f09..6a1c92ffbef 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -116,7 +116,10 @@ def fetch(self): self.log.info("Downloading %s to %s...", self.url, self.cache_file) tmp_cache_file = self.cache_file.with_suffix(".download") - for retries in range(3): + for retries in range(4): + if retries == 3: + raise Exception("Retries exceeded downloading %s", self.url) + try: with tmp_cache_file.open("xb") as dst: with urllib.request.urlopen(self.url) as resp: From patchwork Wed Mar 12 05:17:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 14012884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EED5CC35FF1 for ; Wed, 12 Mar 2025 05:18:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tsETg-0002n3-83; Wed, 12 Mar 2025 01:18:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tsETd-0002mc-ME for qemu-devel@nongnu.org; Wed, 12 Mar 2025 01:17:57 -0400 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tsETb-00070i-OC for qemu-devel@nongnu.org; Wed, 12 Mar 2025 01:17:57 -0400 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-22423adf751so92748995ad.2 for ; Tue, 11 Mar 2025 22:17:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741756673; x=1742361473; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oyrjeqgt+D2Pk5pIxLejPmxWNak0WTTJI8f+RFyfGJk=; b=NyZvg7VYgitZyCtbOl/hBNPD608xyhdfaXBKmO0lwW21o2T0DH7Gfa4/yuBL00eiyZ ao9Bks5rvhFVk4cSVCTY1c1iVHgbQt/NY0aypAkEViuc2hhbOYYikUYp3VZfiGpd2heB GTjGdJ4EfE4rQk2fkiZUfusumoDPv3imRH66dyyxH8jQvTPBQkYb3s3iLlISIbuJrFoZ UBiBxTDPPgrAi29+RMN2Y4ZqH43KPFhEmNoGvEwRG9Z97rLsTEe1Esfzpzm9gn44rrrp WaZ5eLblN6ZHG3RTeS1d5wUUJ4/ixdG+79ezf/8r30NZi/uNwXuZphhQxvgnafYbBLLe 9jlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741756673; x=1742361473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oyrjeqgt+D2Pk5pIxLejPmxWNak0WTTJI8f+RFyfGJk=; b=JG+ycohmzakqEM08heRcT4qEeiCA1M1TgGSfN5WNRrvF1B5KlGffRZ9IUzfmDq6YVl r5/IU33F7201onfzf6da0I3gTXvSAISzL0nchWsEUxr4uNF/f59kzv8/J6RyV25iUdAG TgVuKvn+w8H6JQCUznOgFpBWooe+Yf7ywo5mMkQaHGh3qMW+7m5trziZHj79wIbMTQKp MxLjSO3LBVivrrj7w6j9BQ/n1iTn5PMC3P3m1iukxUnNws15x8mvDGqHbnOIIjpfOycT Iu7PkvULtI+o8TXwiX1S4GIcZi/XzvgxQyHeIl3/NrQomPbqsq+VnM+D6FyCgCRL18t2 zaag== X-Forwarded-Encrypted: i=1; AJvYcCW16GOhy/qqsMvP/FLE2B/xj18OVAycpLCA7q5R2qtKh0KH+hHe6fWAaYsGjWLZCwutU7SYdvglYyIz@nongnu.org X-Gm-Message-State: AOJu0Yx6fEWWCLD4oJ7Kwsm45XSuyZ7D0eCHSt8gY1JDWvt1q7yFei3z Zr5GvX3e/NzupAR32wR/oV1tEugOwK69mYQGumX0MKSDPLtqDIxmmpyEhQ== X-Gm-Gg: ASbGncv7/7uNIjgrpmEUxn/mlUGqTKCCbaF/yoeWBvl9K2gzqpD8oNoNhYhsOCynluL 8R9sjNWVjCEUWNUBuXp8nzN517/jKHTbmezuiFVaM+G09xWTmztx1CDXgPhI2N3+g9A6fTTW6EL 7GQWaj3Ie/prLXr+OZ5r63J4UKFoIlWoK17GKgWUTlPtj9bdfoaXDQ4/IplSWlxkiutHoncmqYT /5X0ijLG57yxltQ18PbEQYbvBUmETivYVf985eVOcqwTr+lOFugfKapue7mtSWVY+pwMc+JPKBD QKLnpRJZwk26CsagR1O68xGljyFED3ccFk3w8kgdSraaC3lJSFlkNc2AFwlYyg== X-Google-Smtp-Source: AGHT+IH8Uc8dFo6t0pzPk/eVuxmlY2/O9fXN4Hk5vPqpJFXtC/Io5mn1NCreOUZuH6dU3JYorVYBbw== X-Received: by 2002:a17:90b:2e42:b0:2f7:7680:51a6 with SMTP id 98e67ed59e1d1-300ff0a4593mr7767104a91.6.1741756673525; Tue, 11 Mar 2025 22:17:53 -0700 (PDT) Received: from wheely.local0.net ([118.208.151.101]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22410a925ddsm107557475ad.203.2025.03.11.22.17.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Mar 2025 22:17:53 -0700 (PDT) From: Nicholas Piggin To: Thomas Huth Cc: Nicholas Piggin , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , qemu-devel@nongnu.org Subject: [PATCH 2/3] tests/functional/asset: Verify downloaded size Date: Wed, 12 Mar 2025 15:17:37 +1000 Message-ID: <20250312051739.938441-3-npiggin@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250312051739.938441-1-npiggin@gmail.com> References: <20250312051739.938441-1-npiggin@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62a; envelope-from=npiggin@gmail.com; helo=mail-pl1-x62a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org If the server provides a Content-Length header, use that to verify the size of the downloaded file. This catches cases where the connection terminates early, and gives the opportunity to retry. Without this, the checksum will likely mismatch and fail without retry. Signed-off-by: Nicholas Piggin --- tests/functional/qemu_test/asset.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index 6a1c92ffbef..d34e8f5e2ad 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -124,6 +124,22 @@ def fetch(self): with tmp_cache_file.open("xb") as dst: with urllib.request.urlopen(self.url) as resp: copyfileobj(resp, dst) + length_hdr = resp.getheader("Content-Length") + + # Verify downloaded file size against length metadata, if + # available. dst must be out of scope before testing st_size + # because # copyfileobj returns before all buffers are + # flushed to filesystem. + if length_hdr: + length = int(length_hdr) + if tmp_cache_file.stat().st_size != length: + print("st_size %ld", tmp_cache_file.stat().st_size) + self.log.error("Unable to download %s: " + "connection closed before " + "transfer complete", + self.url) + tmp_cache_file.unlink() + continue break except FileExistsError: self.log.debug("%s already exists, " From patchwork Wed Mar 12 05:17:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 14012883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F259C28B2E for ; Wed, 12 Mar 2025 05:18:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tsETj-0002nr-QN; Wed, 12 Mar 2025 01:18:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tsETg-0002n7-R2 for qemu-devel@nongnu.org; Wed, 12 Mar 2025 01:18:00 -0400 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tsETe-00070n-8P for qemu-devel@nongnu.org; Wed, 12 Mar 2025 01:18:00 -0400 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-22359001f1aso148463865ad.3 for ; Tue, 11 Mar 2025 22:17:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741756676; x=1742361476; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FKBJLiBMvTXU01UIUdx9mtjp99FG3zwnJ4eVDlAWpCk=; b=OcZ39v2pWXm0kOnE8DmSQtZMALkhyP9o8bKcuwY9p4rGUk+r6U8pJxBj7oNLh9r1oi CJJTnz6yVXy+54mDbnce2D2/67xlWi7LQLSMnTmS/RROzIqLKMZllAKPozaEOSgjekkL kiBgL14b8t7WHNMucYTpfFQA6kPf/5Zo9HRay/3jMMJPe1U7pp+tnb7oRB+v1wMMIrEL 9CpPPc2gzOR2B+NnLVVfyDnFx+Jy7fk0McphSraQsDFWJnAoLUFe/sHWDsSsGeNzpe2S deHpF8KwBDqHoJkuiD+RVKBTOEkHwyEt8D5zQ6PDsE/l0Tuog8MeuYY3FZuULlewa27k R6Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741756676; x=1742361476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FKBJLiBMvTXU01UIUdx9mtjp99FG3zwnJ4eVDlAWpCk=; b=nEthQBmgZN8Wq8ychZNE8Ax6G9IoYtVrfeTmECMzVS2CFotAzMeudc5pNBEXUd4b0M I4QQVZByOjIAnFZksge6DQXi1z6KUmoNwRn53H1w/Re25JnVvyLZAHNYPAlwfTQAutIk jxSdMpLcQdeAr0HgeXyQynLiDV81WlWqQIcs+jRAM/nH0lYpbeI2QXpEx1dYttQNwx+i Svx2XOq3lBiIITqw37OlLIo6VLqjPZeq+ebftC9UIzJmJznpHa9Tsvpz3DaJ7ObJgw43 ETfxtru2pxXf0jFZz7WhhmRNCtnbjwR8+nIgcuPn9SXe74k+YNekRfhnXdZNGX7OhmGj ypFQ== X-Forwarded-Encrypted: i=1; AJvYcCULOafOfdzS8BVSCfQy/ksRNkeX5TqbB91fZ1H2rK7EvnC99PAWYTzj9XgGgcURNMRupob7deAUGjfE@nongnu.org X-Gm-Message-State: AOJu0Yye+T1mHpCltMilK09EJDK7eKi6PjXwp6UT2ItfKETAH10yxa5W SsbdEmkQR+R/rXp6+INT5XZVjunYQO78DkLFS+4JVR+oxvf4V1PB X-Gm-Gg: ASbGncvthig6vY+uRJTOrO/aDYTMZVGqj4JSF1jo/gd48BSu07z33z0wDyrZqZdoJPG GQ+Ko/0JnOs6eWBETXBpt/qRHXQnXb6BHkFXQQADFIrGverWOB+LNKvzJpgIkVEn5g73SfkMeP5 ia8/vwj08x2sHBhFhb7uDCuy/DrUTpTTokGMjbZT6XbXKyshrlulsdldkexLWJ1XX9NGfC9HIl5 noWvMwYPdpmO06Tkeaholr446OMw/KL1/asENqJzws8BKAgph5Z3sAxw2DjTQuGZYeT2PXSp4R5 AKsEMjSThPr7LdLOPoMZXKtCKOJCz9rDuGhkU19ivqTyt2TwH50= X-Google-Smtp-Source: AGHT+IEeo4ug4yqgVLwM9LJ5Z6wU27TfL7B32UK1A0Cf4hQ+QeLvTmk9c3+N3Ggiig9cVgG72EiOtw== X-Received: by 2002:a17:902:e80a:b0:224:1074:63a0 with SMTP id d9443c01a7336-22428ab7313mr309492825ad.34.1741756676472; Tue, 11 Mar 2025 22:17:56 -0700 (PDT) Received: from wheely.local0.net ([118.208.151.101]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22410a925ddsm107557475ad.203.2025.03.11.22.17.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Mar 2025 22:17:56 -0700 (PDT) From: Nicholas Piggin To: Thomas Huth Cc: Nicholas Piggin , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , qemu-devel@nongnu.org Subject: [PATCH 3/3] tests/functional/asset: Add AssetError exception class Date: Wed, 12 Mar 2025 15:17:38 +1000 Message-ID: <20250312051739.938441-4-npiggin@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250312051739.938441-1-npiggin@gmail.com> References: <20250312051739.938441-1-npiggin@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::629; envelope-from=npiggin@gmail.com; helo=mail-pl1-x629.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Assets are uniquely identified by human-readable-ish url, so make an AssetError exception class that prints url with error message. A property 'transient' is used to capture whether the client may retry or try again later, or if it is a serious and likely permanent error. This is used to retain the existing behaviour of treating HTTP errors other than 404 as 'transient' and not causing precache step to fail. Additionally, partial-downloads and stale asset caches that fail to resolve after the retry limit are now treated as transient and do not cause precache step to fail. For background: The NetBSD archive is, at the time of writing, failing with short transfer. Retrying the fetch at that position (as wget does) results in a "503 backend unavailable" error. We would like to get that error code directly, but I have not found a way to do that with urllib, so treating the short-copy as a transient failure covers that case (and seems like a reasonable way to handle it in general). Signed-off-by: Nicholas Piggin Reviewed-by: Thomas Huth --- tests/functional/qemu_test/asset.py | 41 ++++++++++++++++++----------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index d34e8f5e2ad..69b7a5ecb0e 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -17,6 +17,14 @@ from shutil import copyfileobj from urllib.error import HTTPError +class AssetError(Exception): + def __init__(self, asset, msg, transient=False): + self.url = asset.url + self.msg = msg + self.transient = transient + + def __str__(self): + return "%s: %s" % (self.url, self.msg) # Instances of this class must be declared as class level variables # starting with a name "ASSET_". This enables the pre-caching logic @@ -51,7 +59,7 @@ def _check(self, cache_file): elif len(self.hash) == 128: hl = hashlib.sha512() else: - raise Exception("unknown hash type") + raise AssetError(self, "unknown hash type") # Calculate the hash of the file: with open(cache_file, 'rb') as file: @@ -111,14 +119,16 @@ def fetch(self): return str(self.cache_file) if not self.fetchable(): - raise Exception("Asset cache is invalid and downloads disabled") + raise AssetError(self, + "Asset cache is invalid and downloads disabled") self.log.info("Downloading %s to %s...", self.url, self.cache_file) tmp_cache_file = self.cache_file.with_suffix(".download") for retries in range(4): if retries == 3: - raise Exception("Retries exceeded downloading %s", self.url) + raise AssetError(self, "Download retries exceeded", + transient=True) try: with tmp_cache_file.open("xb") as dst: @@ -152,10 +162,17 @@ def fetch(self): tmp_cache_file) tmp_cache_file.unlink() continue + except HTTPError as e: + tmp_cache_file.unlink() + # Treat 404 as fatal, since it is highly likely to + # indicate a broken test rather than a transient + # server or networking problem + raise AssetError(self, "Unable to download: " + "HTTP error %d" % e.code, + transient = e.code != 404) except Exception as e: - self.log.error("Unable to download %s: %s", self.url, e) tmp_cache_file.unlink() - raise + raise AssetError(self, "Unable to download: " % e) try: # Set these just for informational purposes @@ -169,8 +186,7 @@ def fetch(self): if not self._check(tmp_cache_file): tmp_cache_file.unlink() - raise Exception("Hash of %s does not match %s" % - (self.url, self.hash)) + raise AssetError(self, "Hash does not match %s" % self.hash) tmp_cache_file.replace(self.cache_file) # Remove write perms to stop tests accidentally modifying them os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP) @@ -192,15 +208,10 @@ def precache_test(test): log.info("Attempting to cache '%s'" % asset) try: asset.fetch() - except HTTPError as e: - # Treat 404 as fatal, since it is highly likely to - # indicate a broken test rather than a transient - # server or networking problem - if e.code == 404: + except AssetError as e: + if not e.transient: raise - - log.debug(f"HTTP error {e.code} from {asset.url} " + - "skipping asset precache") + log.error("%s: skipping asset precache" % e) log.removeHandler(handler)