From patchwork Wed Mar 12 13:00:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 14013497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7638C28B30 for ; Wed, 12 Mar 2025 13:00:54 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tsLhE-0002hD-7k; Wed, 12 Mar 2025 09:00:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tsLhC-0002gr-0D for qemu-devel@nongnu.org; Wed, 12 Mar 2025 09:00:26 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tsLh8-0006Sp-Bo for qemu-devel@nongnu.org; Wed, 12 Mar 2025 09:00:23 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-22409077c06so38151345ad.1 for ; Wed, 12 Mar 2025 06:00:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741784419; x=1742389219; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0PMBkVd9uVcdqrs2UDs6V3l11sQUJC5pHQobfquDNeU=; b=BEhNVz+U748U+ltJnCZZlTrsBSHM+VmujtfDPZbM/qkyfllKz1Kuggae0QqANPAbIY wQkR2mHL/+C+YOJozV40VtfY4k/oD8Py86XQ1StqHueR9hJgiUGBdj/ONYHI17+I8q1v hCwmRPZURu3yjBS2fzYOqtqjwFe+JTJh8dIHZIlIlS3XxKeg5PDhsAxWi5msqH5PcHBm vA1NsgYIF66X3AzBgyxrUAGtcOcv/fGCsGdr17H0HqZfm/qduFX8ex9Ww9TrplO5FMYm jMWn4ScwO5NCLo16Wt0BlAccRffqiVK7/Qn3JtOMD9L6kX8SPwmNNAH1CTaB0SE1Rwtx B5Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741784419; x=1742389219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0PMBkVd9uVcdqrs2UDs6V3l11sQUJC5pHQobfquDNeU=; b=BIy0WGH3B8/kNYDWe3NCoeyLJLbwpQnIO9xuTVietxt249qyTgoU4R6haymYavWpn7 PPduCF5WNtK112P4BFawD1ff5ng4bHx01LD40fPlQWms/LOd3tSCiYWG5CM5TLk8oB+7 hEl6E7nBjfjwqOyjBtxLFq7TZuaWdgcZ/n/uJouv+ZFw1da4r2ahV5rkFbApDHEGoNdt fkqsxmYfbYDeHVU7OvVzxrsRygptyRm5Wwc7m9R/F+hC4zcLsLSGDRe0EsQYKQPMqaVu 8zROppx7+MfCwolqlNUuVot+Lt0wBQhEYuzmQd8RwPtFQH9hHACm6HTPGliMLZnGPF2h hhNg== X-Forwarded-Encrypted: i=1; AJvYcCWYfbE+7tSUi6IeeFVFXa67pG3sg9FC7JNT8JnrSehEXOyTrntivRCs264pbrhy0rKf48r5QbRnbQD1@nongnu.org X-Gm-Message-State: AOJu0Yx6Pelf7NL72AlC1e1dtTxrgPHgI+zAV8f2xB9UW8H6/AML3Rir z90CmVhL/elBiMSBR0ZGfwwyo4p23CQR/4d2bbYvxA89H5Cmxgw+ X-Gm-Gg: ASbGncvZUQoEV8LAf3nCqNvJ1Sfk+ReNsLjTjLvXeJUZ4GB6nDCGxzXxnk14qV39SrF xwlC9hOWbgUfPAfpsUbmNfMD5Ss1R8y/IawnyX3AXSD3N9VFp8hxRh5iJn3S8Vn7+ZjLS87YLPM DMu+AyfOT3ThYdsMhRc/tJhV+miiKCjF62t6dUgeb4Dv5xsm72HgDtedDuwV5Sieh5/K5C21RO6 IVnhaZK3Y8z68b8c8fjI0JibmPCHbqyfCml1weBq6HPhLg7hFMWdjhQFaqsvH9nAl7H60M2kX1q z3MbS8hm7LvkxD0ALvg0bCFumrlPAUigfWCH6BGlVKiPerDLbR8= X-Google-Smtp-Source: AGHT+IHyRnENf5g3lNCbz2Sn3EgKqhtPMy9uy3PMtFNN7i5zUX+4/omVfPdU85CVQDvIA8u3S6isXw== X-Received: by 2002:a05:6a00:4f90:b0:736:3d6c:aa64 with SMTP id d2e1a72fcca58-736aab13b03mr31985892b3a.21.1741784419342; Wed, 12 Mar 2025 06:00:19 -0700 (PDT) Received: from wheely.local0.net ([118.208.151.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-736bee078fasm8868407b3a.139.2025.03.12.06.00.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Mar 2025 06:00:17 -0700 (PDT) From: Nicholas Piggin To: Thomas Huth Cc: Nicholas Piggin , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , qemu-devel@nongnu.org Subject: [PATCH v3 1/3] tests/functional/asset: Fail assert fetch when retries are exceeded Date: Wed, 12 Mar 2025 23:00:00 +1000 Message-ID: <20250312130002.945508-2-npiggin@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250312130002.945508-1-npiggin@gmail.com> References: <20250312130002.945508-1-npiggin@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=npiggin@gmail.com; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently the fetch code does not fail gracefully when retry limit is exceeded, it just falls through the loop with no file, which ends up hitting other errors. Add a check for non-existing file, which indicates the retry limit was exceeded. Reviewed-by: Daniel P. Berrangé Signed-off-by: Nicholas Piggin --- tests/functional/qemu_test/asset.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index f0730695f09..27dd839e705 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -138,6 +138,9 @@ def fetch(self): tmp_cache_file.unlink() raise + if not os.path.exists(tmp_cache_file): + raise Exception("Retries exceeded downloading %s", self.url) + try: # Set these just for informational purposes os.setxattr(str(tmp_cache_file), "user.qemu-asset-url", From patchwork Wed Mar 12 13:00:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 14013498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C73BEC28B28 for ; Wed, 12 Mar 2025 13:01:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tsLhI-0002ih-Pg; Wed, 12 Mar 2025 09:00:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tsLhD-0002hB-LT for qemu-devel@nongnu.org; Wed, 12 Mar 2025 09:00:27 -0400 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tsLhB-0006T9-L9 for qemu-devel@nongnu.org; Wed, 12 Mar 2025 09:00:27 -0400 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-219f8263ae0so124582395ad.0 for ; Wed, 12 Mar 2025 06:00:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741784423; x=1742389223; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A1CyWLvFz1tx/rdu7DL2wB5x36ElQ2/oE3A0m7I2EJs=; b=mzELpP3n6lbvU/JWfXkbBVjSInJRxNwHIwWzWHVjCGDYdbAJOrMlMZhNsdrBDIZ0KQ VAI8p70y0nft6mNXPW0Y4LnudRYdRoNCewq3axsn1m0v+FH1x9E21W/gzD8LwVWDSaqe MP26R86Y3F1yD/sny7v29wA0tFnYsrhpk2B80vC6+BGJO70DP/JKVYotHidwsfjXK//u oELIBdWY8hb3igc/3mq2rmA4mRyzbAubzbg6ZXIz6JFpFhPhzA2VGo6cwCemhFcql5y2 TdcEHNAS58piKAaeXrfIqeU8XJp1m7DoZtGiTAAACn+kMYqCSHa9BpyMO4awg8CyzwOt axrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741784423; x=1742389223; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A1CyWLvFz1tx/rdu7DL2wB5x36ElQ2/oE3A0m7I2EJs=; b=DYwJNUaRzxfXJtTw60ZspgSN9IwjuxXFa5qP9bK1J3IlPZOfszchE1M6H7Jfl8DqwD yc0lNGI2NHGYG6DclJC0nBUNkXE9lCYDr2V7Phxt+E8V/uNNbDm/2YzS6Lg00PN0zsRq Yf69kBMpTQ2e75PlVOFucQqxdboCWJbzCvBoDIpmmMGqHhE0pzN07Gw6oy66mbW6/5ic sVAZm+EaFKCLUjYuxlWc6idxSXD3eSsqBU9z8ZLUrxroAVj0LmX3faRYOXeAq8NSImTO KI99uDyTknbp60b8wSGa56Uun70uFD8PcG/xnM/uyesDjRoeounFfeG5cLYrmtrBTKdc tFdw== X-Forwarded-Encrypted: i=1; AJvYcCUfnitb66FlITAh+xaJvMfe6/7fTGcoCWcVyhhlgH2d+kwuzaTP31MdlyiQVMJoweaRTbWaufIPoosA@nongnu.org X-Gm-Message-State: AOJu0YzxoYP8nV00JHUvhqJVZxhlrcgHijP8vl6UT2QXV3IZamKt3At7 x/DTNbkY4JaAXczNX4M8YsAVjglA/NiQL12wizREAVW12+j1xYVpUnGX4g== X-Gm-Gg: ASbGncvmAtdiL28+CAnGtmAnGsP4sRihoN5NYpQp0fLwnoJeEGvZtQw4iNfs3N8KYB6 hlbSrqf/upiD3rMK5KQDhZ8Z1GxrtYbv0g4HzmnbromAkNeS+LbEz9lmAoDwUaE+khGYOoOmbgA TJswU1AwostbAxuP+xSlx0xHVBbLRDrbLSVroCqy8zeuLQIFxU9Ywa5a8zrJs1IIPq2gtKpTNk/ t9BDqjq9XyTMXg3Vky/+R/KRkXNjHiDr5692CqsEnRQ+95XD54TIKfibqBiPtPOpLUk3bdce0RI rZ61AgBaIKXrrKF+en4bOix4bjJFG0iSR+ybxZj4paHHNEQyGOw= X-Google-Smtp-Source: AGHT+IGDlsMVYnZxpvy35LFqyNJa8Sgoz1Qv6tnN5NMO82VVImv3BL9HBp1XU69blKC5ODNu43cO4w== X-Received: by 2002:aa7:830c:0:b0:736:5486:781d with SMTP id d2e1a72fcca58-736eb87c1f4mr7766597b3a.19.1741784423419; Wed, 12 Mar 2025 06:00:23 -0700 (PDT) Received: from wheely.local0.net ([118.208.151.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-736bee078fasm8868407b3a.139.2025.03.12.06.00.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Mar 2025 06:00:22 -0700 (PDT) From: Nicholas Piggin To: Thomas Huth Cc: Nicholas Piggin , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , qemu-devel@nongnu.org Subject: [PATCH v3 2/3] tests/functional/asset: Verify downloaded size Date: Wed, 12 Mar 2025 23:00:01 +1000 Message-ID: <20250312130002.945508-3-npiggin@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250312130002.945508-1-npiggin@gmail.com> References: <20250312130002.945508-1-npiggin@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::635; envelope-from=npiggin@gmail.com; helo=mail-pl1-x635.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org If the server provides a Content-Length header, use that to verify the size of the downloaded file. This catches cases where the connection terminates early, and gives the opportunity to retry. Without this, the checksum will likely mismatch and fail without retry. Reviewed-by: Daniel P. Berrangé Signed-off-by: Nicholas Piggin --- tests/functional/qemu_test/asset.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index 27dd839e705..6bbfb9e1cad 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -121,6 +121,20 @@ def fetch(self): with tmp_cache_file.open("xb") as dst: with urllib.request.urlopen(self.url) as resp: copyfileobj(resp, dst) + length_hdr = resp.getheader("Content-Length") + + # Verify downloaded file size against length metadata, if + # available. + if length_hdr is not None: + length = int(length_hdr) + fsize = tmp_cache_file.stat().st_size + if fsize != length: + self.log.error("Unable to download %s: " + "connection closed before " + "transfer complete (%d/%d)", + self.url, fsize, length) + tmp_cache_file.unlink() + continue break except FileExistsError: self.log.debug("%s already exists, " From patchwork Wed Mar 12 13:00:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 14013501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 229C2C28B2F for ; Wed, 12 Mar 2025 13:01:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tsLhN-0002k2-Kt; Wed, 12 Mar 2025 09:00:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tsLhI-0002ig-Il for qemu-devel@nongnu.org; Wed, 12 Mar 2025 09:00:32 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tsLhF-0006Tl-MT for qemu-devel@nongnu.org; Wed, 12 Mar 2025 09:00:31 -0400 Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-225b5448519so5668015ad.0 for ; Wed, 12 Mar 2025 06:00:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741784428; x=1742389228; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=efKsAuHsEC3n3nWPeW8PU/lQXIw7WCfr+JQ3xOUouNk=; b=KAvsn7Gjmj1Cr0bRLBUq4/UCUgHZCpX9EugWZfKXwBw3rTegu6tUwPXzKXhIa/Zuq0 oiok7Ins1QuOmyT79AL+PK+yuxU1g/6sT3GdoxMD+nOTnhBpFlljoyzdu0uHy0zc7BtU TB3hVAf8+Ymel24IsEIvByt43FfrHbohSS3+Q4DE0aIRul7vCj1vyCXefkn7h+YXm+4h nuYDDroC5iSVQIJmzZAIR9NA98o1EaGKY3IGM60KwD1hIQWnz5Ne57wi4cF235LuVWJ4 wqCpnLRKEKDPaZIMofRiFeODgGb8H1GG3pEG2dg695zV0h5uxYw57HZIHMksDBDLGtj3 MJXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741784428; x=1742389228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=efKsAuHsEC3n3nWPeW8PU/lQXIw7WCfr+JQ3xOUouNk=; b=qwb/YW2eINEPnpPOXPEdAxbqWa6ATzaE3PpStHjobOnDBiSRe5X/jlW9V2N3DPneLy qAVUx7e9RD424+6mOAXm6cyYjVrZ0a8XN9/koJQpae8IgkfuJ/XZXfa5HbwpgLCldJxG 0X+3kwNuQSwoE/Ohh+olE5vh25wPksax1GZRBS5LBuXIY94R2Aphejy3aavvfMmhPy1X EAAIHkF1Xnq02NPCNCbre0WsxPviwiUsadEwL0OZd4ERmOn0NW/w5lKEihpVQV0DlHFH IIZXXqGOz5iqOTEFJlVybZPJGZzBHSk0ctPlqWofROOcn7Y9mB/1vMR9BmeNatSnkCJd wzHg== X-Forwarded-Encrypted: i=1; AJvYcCWFKIyFVVqAVg3ApvHDv/qt4k5EM5DMk63Ja2OOOAzRQKDg9ux/UUg8HPULv5NPQ3VRHNtw8VBBJtrm@nongnu.org X-Gm-Message-State: AOJu0YzRFqCljk36Nr4T/40DdysxQl3hGFKYDBPmUyRG+r9vorJI4JRz 3RnYZEomo2P5fXRAj8TW+DjxgqETrmll89bgCXYeWw1EAR5PYTD5 X-Gm-Gg: ASbGncvagA1fFmh5Xu3ugjD3CxTOCgO6T9CClHUV28gyUosFYutz95RqZIxYmQwwnjx oWyZH6U9GwFEPOKvGyy2ID9xTTxqODzsg3DTNfuKz3V78xaHDL1NbYILub4uNX7gguTZUNGJXAD HqUWgC7jkFK1+fR8rv/uLk7HoCiPZpO+gx5XjjRLVPG0JuLqZJgIyKKXvqW1NZQDSYFmlxXj3uH ZQUO2OGmoWrOwteLLp2tbDCl0l3ZyCh4G7mzsY9ZUiqZIOxFTI5fSq0JrGvcyqIT049r6naRfXR XSzlI4IC/LlCOrQi4AyMyml7vfXI9ttZmXspq5SZUBBSUEfGhZA= X-Google-Smtp-Source: AGHT+IHXyOmASSIHQ8tU9bICWHFBVFfRAZZKWnIkOze8wsdxCsLIdq4qml/oFUpZBUCz7+KjUa/P9Q== X-Received: by 2002:a05:6a00:218f:b0:736:3c77:31fd with SMTP id d2e1a72fcca58-736aab0eeabmr29976864b3a.23.1741784427395; Wed, 12 Mar 2025 06:00:27 -0700 (PDT) Received: from wheely.local0.net ([118.208.151.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-736bee078fasm8868407b3a.139.2025.03.12.06.00.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Mar 2025 06:00:26 -0700 (PDT) From: Nicholas Piggin To: Thomas Huth Cc: Nicholas Piggin , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?q?=C3=A9?= , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , qemu-devel@nongnu.org Subject: [PATCH v3 3/3] tests/functional/asset: Add AssetError exception class Date: Wed, 12 Mar 2025 23:00:02 +1000 Message-ID: <20250312130002.945508-4-npiggin@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250312130002.945508-1-npiggin@gmail.com> References: <20250312130002.945508-1-npiggin@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::62f; envelope-from=npiggin@gmail.com; helo=mail-pl1-x62f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Assets are uniquely identified by human-readable-ish url, so make an AssetError exception class that prints url with error message. A property 'transient' is used to capture whether the client may retry or try again later, or if it is a serious and likely permanent error. This is used to retain the existing behaviour of treating HTTP errors other than 404 as 'transient' and not causing precache step to fail. Additionally, partial-downloads and stale asset caches that fail to resolve after the retry limit are now treated as transient and do not cause precache step to fail. For background: The NetBSD archive is, at the time of writing, failing with short transfer. Retrying the fetch at that position (as wget does) results in a "503 backend unavailable" error. We would like to get that error code directly, but I have not found a way to do that with urllib, so treating the short-copy as a transient failure covers that case (and seems like a reasonable way to handle it in general). Reviewed-by: Thomas Huth Reviewed-by: Daniel P. Berrangé Signed-off-by: Nicholas Piggin --- tests/functional/qemu_test/asset.py | 43 +++++++++++++++++++---------- 1 file changed, 28 insertions(+), 15 deletions(-) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py index 6bbfb9e1cad..704b84d0ea6 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -17,6 +17,14 @@ from shutil import copyfileobj from urllib.error import HTTPError +class AssetError(Exception): + def __init__(self, asset, msg, transient=False): + self.url = asset.url + self.msg = msg + self.transient = transient + + def __str__(self): + return "%s: %s" % (self.url, self.msg) # Instances of this class must be declared as class level variables # starting with a name "ASSET_". This enables the pre-caching logic @@ -51,7 +59,7 @@ def _check(self, cache_file): elif len(self.hash) == 128: hl = hashlib.sha512() else: - raise Exception("unknown hash type") + raise AssetError(self, "unknown hash type") # Calculate the hash of the file: with open(cache_file, 'rb') as file: @@ -111,7 +119,8 @@ def fetch(self): return str(self.cache_file) if not self.fetchable(): - raise Exception("Asset cache is invalid and downloads disabled") + raise AssetError(self, + "Asset cache is invalid and downloads disabled") self.log.info("Downloading %s to %s...", self.url, self.cache_file) tmp_cache_file = self.cache_file.with_suffix(".download") @@ -147,13 +156,23 @@ def fetch(self): tmp_cache_file) tmp_cache_file.unlink() continue + except HTTPError as e: + tmp_cache_file.unlink() + self.log.error("Unable to download %s: HTTP error %d", + self.url, e.code) + # Treat 404 as fatal, since it is highly likely to + # indicate a broken test rather than a transient + # server or networking problem + if e.code == 404: + raise AssetError(self, "Unable to download: " + "HTTP error %d" % e.code) + continue except Exception as e: - self.log.error("Unable to download %s: %s", self.url, e) tmp_cache_file.unlink() - raise + raise AssetError(self, "Unable to download: " % e) if not os.path.exists(tmp_cache_file): - raise Exception("Retries exceeded downloading %s", self.url) + raise AssetError(self, "Download retries exceeded", transient=True) try: # Set these just for informational purposes @@ -167,8 +186,7 @@ def fetch(self): if not self._check(tmp_cache_file): tmp_cache_file.unlink() - raise Exception("Hash of %s does not match %s" % - (self.url, self.hash)) + raise AssetError(self, "Hash does not match %s" % self.hash) tmp_cache_file.replace(self.cache_file) # Remove write perms to stop tests accidentally modifying them os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP) @@ -190,15 +208,10 @@ def precache_test(test): log.info("Attempting to cache '%s'" % asset) try: asset.fetch() - except HTTPError as e: - # Treat 404 as fatal, since it is highly likely to - # indicate a broken test rather than a transient - # server or networking problem - if e.code == 404: + except AssetError as e: + if not e.transient: raise - - log.debug(f"HTTP error {e.code} from {asset.url} " + - "skipping asset precache") + log.error("%s: skipping asset precache" % e) log.removeHandler(handler)