From patchwork Thu Apr 10 08:05:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Gomez X-Patchwork-Id: 14046049 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A8E81C5D7B for ; Thu, 10 Apr 2025 08:05:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744272331; cv=none; b=GqIA9dis7y9G4fnkfrqPymbxZAv/JDG6MCRs5FQGy4Uo9XIguOZw2ikLl7CiwfeAav4McJGbsXYSwvwkfFbiSID5HJBLGkfn+CdrqrkhRQ3XVhH02oYjpXeBUAlSoDR3iIXTmgLGijIsE4CNTvw72OzV1PpbRWjOdixZEv22ajI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744272331; c=relaxed/simple; bh=RqiJrPCIViQdIQu0NyV/0tahzUCRLutZSv8hH5PVyH0=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition; b=dp5L7il+AZyKUV3c3COYJuw74GAk44FP+F24A5jJ57AD/oGZl8JP604IZ7MBrs/rq7lqef7cgXvc+RLVLBPoz9nEgwpPKPLEZDX10WEE1Q1oPQpWs62tUkeg/NzcyEF/7GXpdQxegUH7AW85rSEdTocPwgL42k54qDVCJz1wYPw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F4nIf57+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F4nIf57+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4E61BC4CEDD; Thu, 10 Apr 2025 08:05:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744272330; bh=RqiJrPCIViQdIQu0NyV/0tahzUCRLutZSv8hH5PVyH0=; h=Date:From:To:Cc:Subject:From; b=F4nIf57+LMGxmXVubT08VhqQGoaufMXft6wHJ1LBp2Mc3pkPbQAJZctw5pt6cBWKH YSosdYl6sf6FwzVr5vHw1D8fZUdH7e17tI515gOE5Gg2sK7D7emlBMVEWgb1SUZjHA ae1DnvMYuy0dCpnl+nkU3NgbVND+O69lMZ5Vu4yW0mb6a53iBh0DHYLu2rJ3JcnRj6 DSEPaRdLJ0H/F/URIJFh9qyC7GptO9sS1em8qtEfBnC9spzcrsyOlXeoEeInOrq93a ynWV2XXbCndHheuDfWQzJF+nCNgOGewuKGQ9gxO8tRxsnibJ0O9tiPTGsTl9o9HhBM XALIUXY6Xanlw== Date: Thu, 10 Apr 2025 10:05:28 +0200 From: Daniel Gomez To: users@kernel.org Cc: Luis Chamberlain , Chuck Lever , kdevops@lists.linux.dev Subject: Forbidden requests for kernel.org/releases.json Message-ID: Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline We've started encountering "HTTP Error 403: Forbidden" errors in kdevops when querying https://www.kernel.org/releases.json from our CI environments/ deployments. We're using a Python script with the urllib library to fetch the latest kernel release information [1]. As a temporary workaround [2], we are testing a User-Agent header to mimic a browser request. To solve this properly, we have the following questions: * What is the recommended approach for automated tools to access kernel.org/releases.json? * Are there any rate limits we should be aware of? * Would it be possible to serve releases.json from a CDN-backed subdomain to reduce load on the main site? We could mirror the subdomain directory and enable pointers for our datacenter network. [1] https://github.com/linux-kdevops/kdevops/blob/main/scripts/generate_refs.py#L300 [2] add User-Agent headers to query kernel.org/releases.json: diff --git a/scripts/generate_refs.py b/scripts/generate_refs.py index 5171414..41011cf 100755 --- a/scripts/generate_refs.py +++ b/scripts/generate_refs.py @@ -302,7 +302,9 @@ def kreleases(args) -> None: reflist = [] if _check_connection("kernel.org", 80): - with urllib.request.urlopen("https://www.kernel.org/releases.json") as url: + _url = "https://www.kernel.org/releases.json" + req = urllib.request.Request(_url, headers={"User-Agent": "Mozilla/5.0"}) + with urllib.request.urlopen(req) as url: data = json.load(url) for release in data["releases"]: