From patchwork Fri Sep 6 19:46:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794705 Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7625214B959 for ; Fri, 6 Sep 2024 19:46:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651972; cv=none; b=PrY9mXdWQBijGXxjZzWPp5HPu357r3ofBP6w1PCdIV9bjsMrDhiLanAzE7tBS9/zFyWNA2IcQbxn95gBs7Q669vNdu+62E/awaQTRhiMB7tKlw9GXHDN8UaQ/pTVK2uLWD2+RWRS0GXLsa3iV6uh5LUvSuLz5WFJ0hmXZRsfjZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651972; c=relaxed/simple; bh=1OqZqRC1CrZSSpImBHMiCDIKQRPT/8tatXq1Gc7aI9E=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=GKWLyWdjKPV/Xh5lJdRY/Vr9GL7eT1ehYoHSrd3OPsMLfSjG4FzMx3OqiBl8Tp5D2HcB810e1jJ3+kl9mtbG3UHveuwQ1FiWxwMBe2jBHr3D0KX0NsGgylKv91vmwkOqKb08XfoEOPF90dW6pdoybXzGKGzGyVBzpH0vXB6VPL4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=ZjaQVAF2; arc=none smtp.client-ip=209.85.219.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="ZjaQVAF2" Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-e1a7c25e350so2414938276.3 for ; Fri, 06 Sep 2024 12:46:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651968; x=1726256768; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=cuyOPfkT1dv+XTlY81AnoeYXvIHchtUuGIDnOUMROTg=; b=ZjaQVAF27vtR2+gFC2lL9cA9ujwYPvGyeeYWQbJZeEqKW6gyNEPw9aA7ZvYw08tqRk ViympYWvHgox69o6ubZeuS2cCHfL7Bok90VvBERLeVRPpJECyHiyoim+o+jKUKESYPqe YHbGzBF232+5Gw111ClkBMzWWGw4Ilrc3z82U6PJh3g4dHLEoM4xbva7AhsV6Lig/McH dTSRdAfwy5e+v9yLGHccCNDhf88VJS1PiO8Tq+U2M9id7+D9eqPpwJDhdNsHpD+x0ZH1 hJ8opWXPm9WJCiCIIwdvTf0d/Ist8Hd3Duo1uyxb5LAq/L0oLea3ehtilscLJNJHNSBg Qw4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651968; x=1726256768; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=cuyOPfkT1dv+XTlY81AnoeYXvIHchtUuGIDnOUMROTg=; b=BFuwR7mYlgG73HKvHxhGkGFG2TaAJMcx8OkAP1Wt+4/cpvXunOb1WoWY+pm74dbgLE 0TKvaAO+/SlpeA0V46WdA6W5trW/0JhI9CHmaKS39bHGIRkhzZT2oAwVo9vFIDvVCrUP EBycpsX2U1dd3fUaq6FQ5uJ91KTJ2APuA+3BHn5iGHOUO2LCgt7n2yyAI3zbeNe+siuc zZgTuS2vlw3atNYxNV4tnYgmuSt1qM/GDf62fshCoPDRzAgpUrmAYzD27PvE57K7Rat9 1Jks856/S8LU0qfA9vbvJjYNpSbPY4j4+5+KZTuXxwVKgK5gTp78FVQ1PGVVbPhC2S8M yniA== X-Gm-Message-State: AOJu0Yych5D9YBqY+jtZPfLfp38M6snQ8AVDD3EEgIQGHSVuP+vLN0WS R8DoY12RjwOIiZFODzf+k7n6Kk2Acgnzh9ZIkEWe3GWPXng54Y6WAC8RvAmuRjVS1jsBinja+2k Lc/0= X-Google-Smtp-Source: AGHT+IGQRWcSXBbCijYaSByn6EkXqEzbaTPPSyrbrjK8nD7+hictfC9etTFtQhbppQn272pHYANdCg== X-Received: by 2002:a05:690c:386:b0:6c7:a120:e0ec with SMTP id 00721157ae682-6db44f2f717mr51022127b3.22.1725651968112; Fri, 06 Sep 2024 12:46:08 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db565385f0sm1148397b3.62.2024.09.06.12.46.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:07 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:06 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 1/9] finalize_object_file(): check for name collision before renaming Message-ID: <738b1eb17b47c11ae0185b5328bac549dbf05081.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We prefer link()/unlink() to rename() for object files, with the idea that we should prefer the data that is already on disk to what is incoming. But we may fall back to rename() if the user has configured us to do so, or if the filesystem seems not to support cross-directory links. This loses the "prefer what is on disk" property. We can mitigate this somewhat by trying to stat() the destination filename before doing the rename. This is racy, since the object could be created between the stat() and rename() calls. But in practice it is expanding the definition of "what is already on disk" to be the point that the function is called. That is enough to deal with any potential attacks where an attacker is trying to collide hashes with what's already in the repository. Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- object-file.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/object-file.c b/object-file.c index c5994202ba0..683e6b2a0bd 100644 --- a/object-file.c +++ b/object-file.c @@ -1904,6 +1904,7 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo */ int finalize_object_file(const char *tmpfile, const char *filename) { + struct stat st; int ret = 0; if (object_creation_mode == OBJECT_CREATION_USES_RENAMES) @@ -1924,9 +1925,12 @@ int finalize_object_file(const char *tmpfile, const char *filename) */ if (ret && ret != EEXIST) { try_rename: - if (!rename(tmpfile, filename)) + if (!stat(filename, &st)) + ret = EEXIST; + else if (!rename(tmpfile, filename)) goto out; - ret = errno; + else + ret = errno; } unlink_or_warn(tmpfile); if (ret) { From patchwork Fri Sep 6 19:46:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794706 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F5CD1D5CDD for ; Fri, 6 Sep 2024 19:46:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651973; cv=none; b=VHZL+6kDVgfLgY9PX9x8ArXBYLJlViWAs08a1llUTXDaExRrx52gm1dIP6N08YwmzPzekm8DX0i8o+q4Zw9Mte+BW9QPKCB0+zG24DkNDqvr7PFKIrRB7UG3tlHZFnpKTiT1F+U91y6PcllsKYjYAfuG0+BggcJAHso6AKyYLFQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651973; c=relaxed/simple; bh=0PCuwznYuuMmJwyBB8SBMMCsV0vgqVxxIK0Z1qKxcJQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BuuAMvh8ck60qVFbLgux7Cgy2agyuY9IkZzMEHUDdExSHZKMf0c7jUPuR8rzw4CAmRLrPI3pdSm3Q1QwB6KiF747XTSMPE4NARiTtQdiLU/7RH/BBNO00ss2/X7hKWDu53PikqLAaz8L+XE8x8Wb1tcDjzTjO9Ak1KIP98mZmh8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=R7WGgxSI; arc=none smtp.client-ip=209.85.128.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="R7WGgxSI" Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-6b8d96aa4c3so21271177b3.1 for ; Fri, 06 Sep 2024 12:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651971; x=1726256771; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3M/vSL32y+A00DK3aNCbx/JcjT4YoZydZ4tyP6jjVlk=; b=R7WGgxSIX9i1Vnt/ibffKhKuTDcPWrJtIoC0W0BDHM878K8MyG5KcoSRBM7b3mm4HP bPuq0AmGN6z96U4t6WDcLAmXrcmFMrUw7CXXEG8k5tFD2PxNr/J22BW1pj9Uh3lfVuJ1 dUFY4yxQIgAzl+qrjrd9kLDitEed/XVKd9Tdw7GmlGrJJSPOhsiwOBHZcYnE14D4EbHn RLXLXWADkgDTBNW8zhXQy8g08lOD8r1lQYPhbAj9ToJ80LuveQT3Y39OPAmCFfhZMx1O KyqqDCw9nOljp31y4kozH0R3BH6zz5L1rx++k38F9zeX07zQnG+lun4SyjQlpPWPafv9 4NVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651971; x=1726256771; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3M/vSL32y+A00DK3aNCbx/JcjT4YoZydZ4tyP6jjVlk=; b=RzFglZURPLI/JGxka9BiBwKUrgbF1rKVO+UpJX6uQEZfBVtNtVCzAnMnjVRipW8tr4 oul/8ejSyQp3GuUK/573/bGaJFd6mmrghaWCXZiZKlpGqfdW9uuukT/B2ruY8TGC8q/h nJtnuNmzs2p16TaAN81XCsoo644dHkj6xoxCrmzxV7X9se3YnrleDnnyLeYZSVBnaX5T 91hrZI4y/cknXTL8ZePDaGuXdJ2eWyEyNUboE7U49tufomexSlzN2/yc5Y3Tg+vMQzj8 i4pYAPO5v9a1GsToIUP5dZr72zooRdKMHxC2zxcl/zcbm7MsIIQ/Gz0rQe0t8snRe3BW WaRQ== X-Gm-Message-State: AOJu0YynyVajEJpLhMBqYlN0YW4u7qAF4sUicCtOGPPhWLfUApy3rtVN S708/5N8mD2yUajvKOFZGKDNOSuVQRtQ4v2+0mqTOK97e9iB/irFN6+GKre0EHMTKDA+qzEsBB2 J0s8= X-Google-Smtp-Source: AGHT+IF1HbIRXxAVgb92612GcMQ6URQ9NQ0YgYdEruWxwb3N0Foy6h1djiVGknrLNeKmeOwZ3KGVzQ== X-Received: by 2002:a05:690c:f01:b0:6b3:a6ff:76a7 with SMTP id 00721157ae682-6db44a612ebmr51390707b3.0.1725651971050; Fri, 06 Sep 2024 12:46:11 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db56337dbcsm1145247b3.44.2024.09.06.12.46.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:10 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:09 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 2/9] finalize_object_file(): refactor unlink_or_warn() placement Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: As soon as we've tried to link() a temporary object into place, we then unlink() the tempfile immediately, whether we were successful or not. For the success case, this is because we no longer need the old file (it's now linked into place). For the error case, there are two outcomes. Either we got EEXIST, in which case we consider the collision to be a noop. Or we got a system error, in which we case we are just cleaning up after ourselves. Using a single line for all of these cases has some problems: - in the error case, our unlink() may clobber errno, which we use in the error message - for the collision case, there's a FIXME that indicates we should do a collision check. In preparation for implementing that, we'll need to actually hold on to the file. Split these three cases into their own calls to unlink_or_warn(). This is more verbose, but lets us do the right thing in each case. Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- object-file.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/object-file.c b/object-file.c index 683e6b2a0bd..54a82a5f7a0 100644 --- a/object-file.c +++ b/object-file.c @@ -1911,6 +1911,8 @@ int finalize_object_file(const char *tmpfile, const char *filename) goto try_rename; else if (link(tmpfile, filename)) ret = errno; + else + unlink_or_warn(tmpfile); /* * Coda hack - coda doesn't like cross-directory links, @@ -1932,12 +1934,15 @@ int finalize_object_file(const char *tmpfile, const char *filename) else ret = errno; } - unlink_or_warn(tmpfile); if (ret) { if (ret != EEXIST) { + int saved_errno = errno; + unlink_or_warn(tmpfile); + errno = saved_errno; return error_errno(_("unable to write file %s"), filename); } /* FIXME!!! Collision check here ? */ + unlink_or_warn(tmpfile); } out: From patchwork Fri Sep 6 19:46:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794707 Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 540B71D7E2D for ; Fri, 6 Sep 2024 19:46:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651976; cv=none; b=Dx9/73TQxsjt5W+gudPhgWi77cXEYzzjwrqpD9J3DQZaGcuUeMVPrTAmHUeVZ3D/in0smMbJvPCGENuxqqWGXstHrqslUue7w/AqzqS+xIr6kepfrtMGi8bYFptmvKpNM4nW68rCIe7J2yFOkS9f0+cr67KJ2ZYUdaEzmHvV0FQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651976; c=relaxed/simple; bh=4d6MxlBsn3ZVo3gEyXEnWUD5FyCn0c6Yj9AZ4iDsuWg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DWOAuJWSHZsC8LDVacMneH4FUuUtHULtq2WbHq8kp5L87fBX5bYmBYvdMMaJ3xbZfKHTeqNL/CmewlpqojGQoDnBOxgRgNA5XMg0MfHgTQU82Vcxr9KZ8jHg9bNqLnHDvf0vZe+tE3P1M8260qs6rnRfEY0qjoM9K8YJZs5ANQM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=GY3gUCCx; arc=none smtp.client-ip=209.85.219.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="GY3gUCCx" Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-e1d06c728b8so2564469276.0 for ; Fri, 06 Sep 2024 12:46:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651974; x=1726256774; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=J2AdznxtXuiwMeRh1I/dRJwIOrYwjvmsFrfIxW7lzP0=; b=GY3gUCCxj+B6sFYLs3jFK0DoJ5opegGXug3yaenrO/Ysp380r67cc9wUMSPVCTaN3T a4r3dlkS2gJWSLgSvFHf77pLutznhGFMcRORsGibjJIgUfqt/sunwBk2Ub+UL3BnZNhd M4V43+Vcoo4U5lz8b6OKqQ5JUqCkEuTNcw1Vt4v12IbkK7fAYqMQo4SF1+fgigmrgr/5 OzPZYqaV77TsLPmc+u9hh4OWflCvhBuQXXIa/KMNKzpYKxPhheQ1vmYbIU5IRIfpevTo z8txaI2WMhnr2zYCrQJxTevVLHGtGiXtJ8s4galTHQbFX9clpZU6RYM9y97VfFRiMwNm +TFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651974; x=1726256774; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=J2AdznxtXuiwMeRh1I/dRJwIOrYwjvmsFrfIxW7lzP0=; b=eEkvhQeNqdNoJhrnItEalGiS4pOQaLWZui1JULc3LSuxIfYEWjd8Me5g263mFkfu74 SpVbyEsNL3MfBsEnBeDcq/jXnPbs83MWMz3H7q+K2dTBimOGh80Y2IgTZKT/kXtZa4t6 cAwYiqYAcdosjmO0OkyoRUxv8YOz0qeAkfQiaBeZ6gAvM1nDgFjI4vF+3imU9eQW1dmI i5hCvLEdxtWRV7LJ3T8uwc0Ajb6Wj4QinoNUg4p3UVtGOEAY6PLWCJxOLOAnPItKbw2n 76/Smz7z+d8sgZUfHdOx7cUqtk0qi5cgSTC2J1a28yH8hV8ENrvy4TiXLegCXIXG07Ql cQNA== X-Gm-Message-State: AOJu0YwkUeDb1TPx0ebyIlF1lGbG9oxzWnV4RuW445CALd8zAdMHtCtJ 2irXeDLSWEtClq8XPgQUVrKsfybd3d64MGHIQBN7BkHTi5PI+nTlzIj4CTUgHH2/U487mkYUwPE 5j74= X-Google-Smtp-Source: AGHT+IEviLb5og7jvd4KGGqABIG+han46PDUPiIJ73dtqEaLIsSiy6Dpd/53wIAza6LXyJreY0CDRA== X-Received: by 2002:a05:690c:4911:b0:6d6:94b2:f3e4 with SMTP id 00721157ae682-6db44f2f652mr51157427b3.26.1725651974096; Fri, 06 Sep 2024 12:46:14 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db565e6026sm1112407b3.145.2024.09.06.12.46.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:13 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:12 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 3/9] finalize_object_file(): implement collision check Message-ID: <0feee5d1d4fc1e0aa1ca5d98f2f404ecdbb2f6b6.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We've had "FIXME!!! Collision check here ?" in finalize_object_file() since aac1794132 (Improve sha1 object file writing., 2005-05-03). That is, when we try to write a file with the same name, we assume the on-disk contents are the same and blindly throw away the new copy. One of the reasons we never implemented this is because the files it moves are all named after the cryptographic hash of their contents (either loose objects, or packs which have their hash in the name these days). So we are unlikely to see such a collision by accident. And even though there are weaknesses in sha1, we assume they are mitigated by our use of sha1dc. So while it's a theoretical concern now, it hasn't been a priority. However, if we start using weaker hashes for pack checksums and names, this will become a practical concern. So in preparation, let's actually implement a byte-for-byte collision check. The new check will cause the write of new differing content to be a failure, rather than a silent noop, and we'll retain the temporary file on disk. Note that this may cause some extra computation when the files are in fact identical, but this should happen rarely. For example, when writing a loose object, we compute the object id first, then check manually for an existing object (so that we can freshen its timestamp) before actually trying to write and link the new data. Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- object-file.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/object-file.c b/object-file.c index 54a82a5f7a0..85f91516429 100644 --- a/object-file.c +++ b/object-file.c @@ -1899,6 +1899,57 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo hash_object_body(algo, &c, buf, len, oid, hdr, hdrlen); } +static int check_collision(const char *filename_a, const char *filename_b) +{ + char buf_a[4096], buf_b[4096]; + int fd_a = -1, fd_b = -1; + int ret = 0; + + fd_a = open(filename_a, O_RDONLY); + if (fd_a < 0) { + ret = error_errno(_("unable to open %s"), filename_a); + goto out; + } + + fd_b = open(filename_b, O_RDONLY); + if (fd_b < 0) { + ret = error_errno(_("unable to open %s"), filename_b); + goto out; + } + + while (1) { + ssize_t sz_a, sz_b; + + sz_a = read_in_full(fd_a, buf_a, sizeof(buf_a)); + if (sz_a < 0) { + ret = error_errno(_("unable to read %s"), filename_a); + goto out; + } + + sz_b = read_in_full(fd_b, buf_b, sizeof(buf_b)); + if (sz_b < 0) { + ret = error_errno(_("unable to read %s"), filename_b); + goto out; + } + + if (sz_a != sz_b || memcmp(buf_a, buf_b, sz_a)) { + ret = error(_("files '%s' and '%s' differ in contents"), + filename_a, filename_b); + goto out; + } + + if (sz_a < sizeof(buf_a)) + break; + } + +out: + if (fd_a > -1) + close(fd_a); + if (fd_b > -1) + close(fd_b); + return ret; +} + /* * Move the just written object into its final resting place. */ @@ -1941,8 +1992,8 @@ int finalize_object_file(const char *tmpfile, const char *filename) errno = saved_errno; return error_errno(_("unable to write file %s"), filename); } - /* FIXME!!! Collision check here ? */ - unlink_or_warn(tmpfile); + if (check_collision(tmpfile, filename)) + return -1; } out: From patchwork Fri Sep 6 19:46:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794708 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A07FE1D47AF for ; Fri, 6 Sep 2024 19:46:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651980; cv=none; b=WRbXa8b2EsywaSC/5IpAzOFIla9o1cQZjZ9M8RJeTAl0u/nP49rpL+YlxEbZ7bkN0j9XL/mxnpyJ0fZHVQr1gi7gU/veHihQ3EWdDWgh+L7eGnRWawHk+wyGeh5zFhqzz6/mO0R5IcAgSirDDV/z0Ae1t7lkwbM6pRLkjTR+6QI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651980; c=relaxed/simple; bh=th15hE2tcz1Jtp9cEcesboste7+G76vXcUokxo4Rq04=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=XFVN5gsoMwrQhxxGLmKo5ixP74z/RwM/P4ULhXQipUdFmZpACZ8Xfh+/2ph1maZqqei2aQMml9Eh27JhwfniZvHxBldxMuGoG8fXE9gBCjyR5imKkz6BrZnWjQuFYSDlZLwg0YIbkGIxPNNC1cGTutPKHUgPa3Eli3dEZto9JGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=ErpP+6qk; arc=none smtp.client-ip=209.85.219.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="ErpP+6qk" Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-e1a82d1ef62so2521952276.0 for ; Fri, 06 Sep 2024 12:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651977; x=1726256777; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=n5z5LVBw54AQwPy7u5bUYLtuVvsoN7bq2zRUKcEST8M=; b=ErpP+6qkYQIbEF1if9yZQaFguT0WDFRp6o6gHRJ0WUfSoH858d66X92WSLP67+PWKX mKeIfStS5d9xrlCoozsq08cW8SoKVnJVyPtBaaOW6dC5XAVmTWwJTxcTn1kF+AZBBONS mRUbaWM7v3IvtEQcl5z1lnHNsyN0uDjfyDHekIIrwCCzL88x+qjgMpRF5KrJ6kpQ6UUF UysZvIHsnhl/AHg107arKFQrPlrGRznQeM1qToIjEJs00FkriM0PzRvy/uCmW7+km9JZ Tl8kqjGL3mSQyEuj2UTHbZbcghrgjdlLlMja2iTayZrveUGpso19bN1xoPbDXSA8uTsb PADQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651977; x=1726256777; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=n5z5LVBw54AQwPy7u5bUYLtuVvsoN7bq2zRUKcEST8M=; b=tfGYN/3T61PgPFh5E2syeuf56vTyzlqYPqg30UPdHVQKA+rUbTY8IXBVw42PcKJPHw Nw60dPhWRHW4C9zCgV/TSAUbjRgsRPcCyRlzZYxQGQRx1M0qMpeoTdMQxarYaylWuvNb cW1QOUxhjDbiCtnjfd8sE32EEgAR9fZ5sAkfXUTLlJVk7TU3eBNvMLpM5Ti0SvSkqCEx mGls1flADKyOTbhjdaK9PfXaDg6UGh3KPe+GQ63Fcd9TT+T2BTG7GdnwTE7A5UUxZHsO aFWivavSAkJa0mpPPqm53HvEwNsBdp9ANYDLx3Wotd/g85v/VD5eWUkwPkOGd3qgfhLq c4fA== X-Gm-Message-State: AOJu0YwhV4R3InxUOmlHIVlJF05isCyHjmCd5PSYxq7bLybCuDslWgmU wFxvoN3jONqUIeYWFFPla88vIs4+X6vnD/y+lm132q0TnHvoF0TxcoKZIUJfJSYHxg/QnXggiUi aDSo= X-Google-Smtp-Source: AGHT+IEj7iTZXVDShtZQpQnARjQCzt9FdU4m66GNm8zLq+IFKi4GvsZD1AGsZ/O1YdWrJk6EZL9IwQ== X-Received: by 2002:a05:6902:2e10:b0:e1d:1434:98a4 with SMTP id 3f1490d57ef6-e1d3486f977mr4166163276.9.1725651977406; Fri, 06 Sep 2024 12:46:17 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e1d43fae12asm105518276.37.2024.09.06.12.46.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:17 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:15 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 4/9] pack-objects: use finalize_object_file() to rename pack/idx/etc Message-ID: <620dde48a9dece56395a442b0f42a157d06adb29.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In most places that write files to the object database (even packfiles via index-pack or fast-import), we use finalize_object_file(). This prefers link()/unlink() over rename(), because it means we will prefer data that is already in the repository to data that we are newly writing. We should do the same thing in pack-objects. Even though we don't think of it as accepting outside data (and thus not being susceptible to collision attacks), in theory a determined attacker could present just the right set of objects to cause an incremental repack to generate a pack with their desired hash. This has some test and real-world fallout, as seen in the adjustment to t5303 below. That test script assumes that we can "fix" corruption by repacking into a good state, including when the pack generated by that repack operation collides with a (corrupted) pack with the same hash. This violates our assumption from the previous adjustments to finalize_object_file() that if we're moving a new file over an existing one, that since their checksums match, so too must their contents. This makes "fixing" corruption like this a more explicit operation, since the test (and users, who may fix real-life corruption using a similar technique) must first move the broken contents out of the way. Note also that we now call adjust_shared_perm() twice. We already call adjust_shared_perm() in stage_tmp_packfiles(), and now call it again in finalize_object_file(). This is somewhat wasteful, but cleaning up the existing calls to adjust_shared_perm() is tricky (because sometimes we're writing to a tmpfile, and sometimes we're writing directly into the final destination), so let's tolerate some minor waste until we can more carefully clean up the now-redundant calls. Co-authored-by: Jeff King Signed-off-by: Jeff King Signed-off-by: Taylor Blau --- pack-write.c | 7 ++++--- t/t5303-pack-corruption-resilience.sh | 7 ++++++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/pack-write.c b/pack-write.c index d07f03d0ab0..e5beecd3a4f 100644 --- a/pack-write.c +++ b/pack-write.c @@ -8,6 +8,7 @@ #include "csum-file.h" #include "remote.h" #include "chunk-format.h" +#include "object-file.h" #include "pack-mtimes.h" #include "pack-objects.h" #include "pack-revindex.h" @@ -527,9 +528,9 @@ static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source, size_t name_prefix_len = name_prefix->len; strbuf_addstr(name_prefix, ext); - if (rename(source, name_prefix->buf)) - die_errno("unable to rename temporary file to '%s'", - name_prefix->buf); + if (finalize_object_file(source, name_prefix->buf)) + die("unable to rename temporary file to '%s'", + name_prefix->buf); strbuf_setlen(name_prefix, name_prefix_len); } diff --git a/t/t5303-pack-corruption-resilience.sh b/t/t5303-pack-corruption-resilience.sh index 61469ef4a68..e6a43ec9ae3 100755 --- a/t/t5303-pack-corruption-resilience.sh +++ b/t/t5303-pack-corruption-resilience.sh @@ -44,9 +44,14 @@ create_new_pack() { } do_repack() { + for f in $pack.* + do + mv $f "$(echo $f | sed -e 's/pack-/pack-corrupt-/')" || return 1 + done && pack=$(printf "$blob_1\n$blob_2\n$blob_3\n" | git pack-objects $@ .git/objects/pack/pack) && - pack=".git/objects/pack/pack-${pack}" + pack=".git/objects/pack/pack-${pack}" && + rm -f .git/objects/pack/pack-corrupt-* } do_corrupt_object() { From patchwork Fri Sep 6 19:46:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794709 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBF6D1D47AF for ; Fri, 6 Sep 2024 19:46:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651983; cv=none; b=DRpicZZSE6UeAJ6llye/RDxFtHdlIisuEN/s967741c9qN8IsecrDuY6mPR2+0cGSlklLbpPlN/o98lbsgH7mzxdonnSA8VYKTBL9GuJnTmUFjCusgmERf5TqOCWccYILGRhZCoBcesZfT9sxutHaikRqCfuK1ZFXdQSketGtGg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651983; c=relaxed/simple; bh=gaSsO/Xwf6T9g24xg98hpQ3ycGxbruwxEh1Berp9T0A=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=h3zo5KtDLjwxbnBZqUeXhm2xBy35nuVJ+rXUsYmIklb5XqDstC6gOgNEecgHI7Zmg8j7GTymDJa5e37X2YkO7yLtbrAcLxrjLZhECmJL5f8tLwMCtRGhXlPbNFdcZCQrBYMnYUbttriFmTyuwdQUijR2Vtwxm4NFUeCFHcg3Fk0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=whnfdboX; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="whnfdboX" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-68518bc1407so27250497b3.2 for ; Fri, 06 Sep 2024 12:46:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651980; x=1726256780; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=BzZuitm20Jcv335IO2WXHiy01XhKFs8S+X00N+4JxQk=; b=whnfdboX9r3Uzq5J6/6MUuGfyQTui2P9ZTeZcwoQTzkgKUvJI75qhOlpKc70Xa6mia 6BAlLiscGC1wPm2me27uDed90q1ZnIwK4RbfvFkmhV6p2oc8Mp2Qv1P1+hkAJzDk5MpJ SO+JMeFlqWQjgZJ0VqOTQxQEBAaMpC9NURCTi+DgXTnCLJFLl9+1j9ocTGYM28Ul4yaJ PQgVUzFwgRytsILtSfm3dSrx3fbjP+kv/L2fsQxQrqdL26+VHlVSDKfZG39Iu9WU2UDC ouGNzwIJW6OUTliFEY/+tJibD33OHRDG+Z5xYfHGL5eXQyJD7FnjbzmGo/xltbtLJXXz yhgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651980; x=1726256780; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BzZuitm20Jcv335IO2WXHiy01XhKFs8S+X00N+4JxQk=; b=MxUww3cdLGT4H1lRJbyARUaHG/GK9LpL6T60sieB7VliQjuIVfELKOxCpm8x6DTjZ3 mnBeDZTWyFMr/s7YoE9msfzegwC2jY+whfgzUf62LDN60DCx2gpCKgSjDcdF4Nga6C85 HzLlV5rxe4K0sDJmOQxfL/0JHTUe3JVpWu/wuDEppi42nQZhGwL/JP+tddOGOhvV/93f ROKuq1im1xNsASP9BOtuRAPSS+tixw8urE4EPttHr9j2ei8QLh7izyhlMl5X7zrBuQAT lpNYiK+98a6etPG6PpqP4X732DjiS23b4deraRekn3KfbHwbpSl7xrePHbstUFfasMc/ F4Lw== X-Gm-Message-State: AOJu0Yw97jU1HnyIHS/QbcQJ7ayHhUyIgCwPqnOroytFpCJ6ayFRpwTf 2KAYhypZvxhQm6JX9OaNX11FFOHWuPxj64BYQQjIgIZwpsDFzZB0yiUKRcLI4dAQHqoB08Ch792 msCA= X-Google-Smtp-Source: AGHT+IE2P2wc7q/6/1U2GOTwMxI0xYiTi7TpnV0LE7lJyN60Yvj6oYIg2pTJeVN6M+MSsLe63325yw== X-Received: by 2002:a05:690c:67c6:b0:685:3ca1:b9d8 with SMTP id 00721157ae682-6db451544bcmr47329157b3.30.1725651980527; Fri, 06 Sep 2024 12:46:20 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db565e601asm1126437b3.143.2024.09.06.12.46.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:20 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:19 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: For the interop tests exercising basic 'git daemon' functionality, we use version v1.0.0 as the old version of Git (which in this test we happen to designate with using VERSION_B). But that version does not compile with modern versions of OpenSSL, complaining with error messages like: epoch.c:21:16: error: field ‘numerator’ has incomplete type 21 | BIGNUM numerator; | ^~~~~~~~~ epoch.c:22:16: error: field ‘denominator’ has incomplete type 22 | BIGNUM denominator; | ^~~~~~~~~~~ epoch.c: In function ‘new_zero’: Of course, compiling with `NO_OPENSSL=1`, which we have had since dd53c7ab297 ([PATCH] Support for NO_OPENSSL, 2005-07-29) allows us to compile cleanly. This hasn't been such a problem in practice because most builds can use NO_OPENSSL when compiling the older versions of Git used by the interop tests, because often even the current version of Git does not use OpenSSL (e.g., because we use the collision detecting implementation of SHA-1). But subsequent changes will make a build configuration that does use OpenSSL's SHA-1 implementation (at least for non-cryptographic uses) more common, thus breaking this interop build (since only one side will compile with NO_OPENSSL). Let's work around the issue by using a slightly more modern, but still quite old v1.6.6.3, which is used by the i0000-basic.sh test script as well. Signed-off-by: Taylor Blau Signed-off-by: Jeff King Signed-off-by: Jeff King --- t/interop/i5500-git-daemon.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh index 4d22e42f842..c5bf37e4739 100755 --- a/t/interop/i5500-git-daemon.sh +++ b/t/interop/i5500-git-daemon.sh @@ -1,7 +1,7 @@ #!/bin/sh VERSION_A=. -VERSION_B=v1.0.0 +VERSION_B=v1.6.6.3 : ${LIB_GIT_DAEMON_PORT:=5500} LIB_GIT_DAEMON_COMMAND='git.a daemon' From patchwork Fri Sep 6 19:46:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794710 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACCBF1D7E50 for ; Fri, 6 Sep 2024 19:46:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651986; cv=none; b=X0ttkVN9OkRtFVkDmc7pCJ0Lrfgqp+ZaVXXXZUsLr7RDe7Xmo0LsemHe5h4NG/deGEI9szjzdM6BM7ZJ9AxGTyJXWFowj7uYdwxd0NJzGUcK4acBcRT8h/AhFpQkqTbCqJZ2wX7as5piSdvimhXhqExfsTjmB2dcY4zofsQ8lF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651986; c=relaxed/simple; bh=dtoSOlGFMHm0mz7xjUdjLIAgL4WkMFG9XSludvUfv8c=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=E+AkCCwLV07S8/BNvYOZzhcwH+QcUozPT2cP8kOkVlCj3JTNIlm+xTi/B0eRmcMzGGZwSM9h6YZJkWqt9I1zKbOtx36CnR6E5MLuehI0J4qv9jROwbqhQdnYg8fOBoCtgk+IszacKl4LXc6kF8Pah/bKzAGrmko2RQhCFV4K6iw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=cK42F/K+; arc=none smtp.client-ip=209.85.219.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="cK42F/K+" Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-e04196b7603so2670999276.0 for ; Fri, 06 Sep 2024 12:46:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651983; x=1726256783; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=b7hfH8hHEPqjQiI0mja4fwzPLpEdUPZT7sveUu4qSCU=; b=cK42F/K+uo1jI7rwde+vHCgwZLvioNcIaWg/1LE1Ro62Uz398n9/evb9zu5NFQYrwk XPXsi46oECZM66FKCu0uH4BSx+XrU/K8P1AbJp1FIlP0o+7oaVzZ4fqPiU0a9gM8OFd0 NEqc60mb9fbHDHHxWQVsCVKoIqG+mLGE5EjtTQk0E6BWOqX8NJabah9sDY0TfWnIDv+U oCVJkAJdhas0PCij2BZ/iXt6gDrKbHTyEZmsPD1ImkHZ+IZtalyaOupdJ5DrsA+XN7vd qwrQMccCl1mtZIDoB7sQXYmZHD4Ka5Zc46cUx/9FrsRGPwSRbzJajg5oMv9v/iscIhv1 lnmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651983; x=1726256783; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=b7hfH8hHEPqjQiI0mja4fwzPLpEdUPZT7sveUu4qSCU=; b=vM0FnNIzJBUvkjpMqsMK046Vq27Mxzy+H+ro7n1Y5a/a99cp0cG/K/Nd36eqNCoJkV BZ2NSJlDYPmBzE/MFD6MIH+ZMWpoyaOcJiFpdSggG1GgxH4SNdP2Onu5xWzK06z7cmFZ QUPI4wUqIwFHhQJJAKQQaQnrMLUAxazLfsuThAcGTBV6Sep/ID9xIyLoND7UTg51f+gi qZsX72qz08FNcw19YCJnWqFHq+UZCAOC0RxUcA6QLV1sb4tVb2GnweYce4D+3thMyIxo gus+yDzgUHoKdGYZJQVNcSUQ4sTWFL0mETYVqf5/J/xz9a84qY3bbgK++EhxmEapjPPH iSkA== X-Gm-Message-State: AOJu0YxNMM+RWFb5q7Czt3KnR9t9q2b//NK80oIOK76uUCgJ7QUm51iI YTSDgOz83Ccwf6vMtjbVIJXDCTho7V29ovT/Q9zeIhY/dcaftrAKc0u2GkMlqGpJxjpVvbQwgJ1 eoiU= X-Google-Smtp-Source: AGHT+IH88zjvLc8omYy5jFujuz3luUhL7cghp1lczYuyiM6jUmn8uEn71AfSpRQ6vjxMwHvCwOZ8IA== X-Received: by 2002:a05:6902:cc1:b0:e13:83fd:cf19 with SMTP id 3f1490d57ef6-e1d34a1e816mr4854294276.49.1725651983497; Fri, 06 Sep 2024 12:46:23 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e1d43f741d9sm107185276.14.2024.09.06.12.46.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:23 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:22 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 6/9] sha1: do not redefine `platform_SHA_CTX` and friends Message-ID: <22863d9f6dff5fec033f1de0faa93b0ec09d8e24.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related macros to point at the opaque "context" type, init, update, and similar functions for each specific implementation. In hash.h, we use these platform_ variables to set up the function pointers for, e.g., the_hash_algo->init_fn(), etc. But while these header files have a header-specific macro that prevents them declaring their structs / functions multiple times, they unconditionally define the platform variables, making it impossible to load multiple SHA-1 implementations at once. As a prerequisite for loading a separate SHA-1 implementation for non-cryptographic uses, only define the platform_ variables if they have not already been defined. Signed-off-by: Taylor Blau --- block-sha1/sha1.h | 2 ++ sha1/openssl.h | 2 ++ sha1dc_git.h | 3 +++ 3 files changed, 7 insertions(+) diff --git a/block-sha1/sha1.h b/block-sha1/sha1.h index 9fb0441b988..47bb9166368 100644 --- a/block-sha1/sha1.h +++ b/block-sha1/sha1.h @@ -16,7 +16,9 @@ void blk_SHA1_Init(blk_SHA_CTX *ctx); void blk_SHA1_Update(blk_SHA_CTX *ctx, const void *dataIn, size_t len); void blk_SHA1_Final(unsigned char hashout[20], blk_SHA_CTX *ctx); +#ifndef platform_SHA_CTX #define platform_SHA_CTX blk_SHA_CTX #define platform_SHA1_Init blk_SHA1_Init #define platform_SHA1_Update blk_SHA1_Update #define platform_SHA1_Final blk_SHA1_Final +#endif diff --git a/sha1/openssl.h b/sha1/openssl.h index 006c1f4ba54..1038af47daf 100644 --- a/sha1/openssl.h +++ b/sha1/openssl.h @@ -40,10 +40,12 @@ static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst, EVP_MD_CTX_copy_ex(dst->ectx, src->ectx); } +#ifndef platform_SHA_CTX #define platform_SHA_CTX openssl_SHA1_CTX #define platform_SHA1_Init openssl_SHA1_Init #define platform_SHA1_Clone openssl_SHA1_Clone #define platform_SHA1_Update openssl_SHA1_Update #define platform_SHA1_Final openssl_SHA1_Final +#endif #endif /* SHA1_OPENSSL_H */ diff --git a/sha1dc_git.h b/sha1dc_git.h index 60e3ce84395..f6f880cabea 100644 --- a/sha1dc_git.h +++ b/sha1dc_git.h @@ -18,7 +18,10 @@ void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *); void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len); #define platform_SHA_IS_SHA1DC /* used by "test-tool sha1-is-sha1dc" */ + +#ifndef platform_SHA_CTX #define platform_SHA_CTX SHA1_CTX #define platform_SHA1_Init git_SHA1DCInit #define platform_SHA1_Update git_SHA1DCUpdate #define platform_SHA1_Final git_SHA1DCFinal +#endif From patchwork Fri Sep 6 19:46:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794711 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 218B41D7E50 for ; Fri, 6 Sep 2024 19:46:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651989; cv=none; b=RSJbttCFWXwoya/ngLGUr3SJK/2jg3M4PghzcniU//72RrZieGG+lY82QxzALU9uQgMAJGAo92w+1qlTAxzUoWK/iZE4wiSXa/WN4aG3J0OAxI+z9nhJb6/Q/PDe1qOh3Zq0OM2HryvfbcCjZ6SfvonT6jL7esuwQmmSPjBD/xA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651989; c=relaxed/simple; bh=ATDIYSnRo4sAen/XWHemEaumGXJ96Qa7ZNUR52P021I=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=oejWS89Z/8flGNJVPId+q/L23vyG6pqZeStLp6V2DlNUWwuw6caKCEkmyEy1di8kE7QyoWscS6zkSH9sURq8FCC3zopFC4BcSsEHlzd3ON+QkQTVUjevO/vWLcdagD+1Xpt7xqwa8gnDFQ5s/1crKPXi8cpw+t6MNaVXT3d6JDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=qX61LjEE; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="qX61LjEE" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-6db20e22c85so23207977b3.0 for ; Fri, 06 Sep 2024 12:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651987; x=1726256787; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=atLEk9aAusYisCBU6u9KPSlIFmocU+G7V2Azz6cGz1o=; b=qX61LjEEdze/J6DXAETa8parKUdeVPVmp+o01X6jyrIw+gY+ByVldOGam64/+czYaB EPJA2HFX03/BYANWLMOpuxA/c9pqdo/x07a264K19dM7BCH0YcfHn0E4ghsM4tp/ooQR 1t1ZSsb4NqHOymWudrSc27L/4FpUW/7ani/mLT1icqa76eYyT+1zZqVQ6tmL+y2pcTz0 +ElKmNq1Kt/i72NWhdW7dovXa3eQ9Dg6LiuP60nONcKrA60vl0dOCnNxER5xyzNRlsZP QxFhsHbpqH05qJCUyW7t789QUxYL6no9i9xwrDXEs0CgRxOHZ0ayQztIR8zOgh6iZkG9 /bOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651987; x=1726256787; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=atLEk9aAusYisCBU6u9KPSlIFmocU+G7V2Azz6cGz1o=; b=vgvLYQScz2hWrzgKujK0IsEut0lYoxz83bFQjhVDN21qoyXG1lCyeQ6SePqiB8Li9B aeQpOAFLg1sTEWQuT38sR5yc8QGzr3J39N6DjTBBuSSgRGtO+zju7BIJb6n3lSnigc9a wGNkCmJNi+/3sMNDCxiqCgUhG+FhENNw11W+jxYVqdnFYK7pUk8VAoj4y4MYo4Ps2nkY ES6Pirpha1iYA0b2dUXGW3WjFM1QeouX1qAUe5L33ykmwzXg4fFGfnkGBh6DKsngmDRG +sxqpxVTakHBgxX9TBXLalKCmcG2OKi8u9fywF0ytUjVR3CwXSH4HprRc9EcutHh2kL/ moDQ== X-Gm-Message-State: AOJu0YwDT99y267eoXMNgRr58P/sVS0/Pn0cyH3gJ8qAP7DeyLsXToE2 /BGunwA+pX6On6qEz0ABU9GCHh2NuAi15foqsAwA3/3Xv7GWCNiXjglQhKcacoNt1EeQhCbvwtu uYpQ= X-Google-Smtp-Source: AGHT+IFlv9MN4xHV6AqZAbsa6PDQahZ0Gwcy9vR8igQ2VKz0Ts7UE6NPU+VgpGxYv4QqXGv+NTIdXw== X-Received: by 2002:a05:690c:60ca:b0:64a:9832:48a with SMTP id 00721157ae682-6db44a5b400mr55853997b3.0.1725651986803; Fri, 06 Sep 2024 12:46:26 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db565ab8d3sm1130587b3.90.2024.09.06.12.46.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:26 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:25 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 7/9] hash.h: scaffolding for _fast hashing variants Message-ID: <119c318d8123fd9ef404730a923968f29ee184d8.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Git's default SHA-1 implementation is collision-detecting, which hardens us against known SHA-1 attacks against Git objects. This makes Git object writes safer at the expense of some speed when hashing through the collision-detecting implementation, which is slower than non-collision detecting alternatives. Prepare for loading a separate "fast" SHA-1 implementation that can be used for non-cryptographic purposes, like computing the checksum of files that use the hashwrite() API. This commit does not actually introduce any new compile-time knobs to control which implementation is used as the fast SHA-1 variant, but does add scaffolding so that the "git_hash_algo" structure has five new function pointers which are "fast" variants of the five existing hashing-related function pointers: - git_hash_init_fn fast_init_fn - git_hash_clone_fn fast_clone_fn - git_hash_update_fn fast_update_fn - git_hash_final_fn fast_final_fn - git_hash_final_oid_fn fast_final_oid_fn The following commit will introduce compile-time knobs to specify which SHA-1 implementation is used for non-cryptographic uses. Signed-off-by: Taylor Blau --- hash.h | 42 ++++++++++++++++++++++++++++++++++++++++++ object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/hash.h b/hash.h index 72ffbc862e5..5e5b8205b58 100644 --- a/hash.h +++ b/hash.h @@ -44,14 +44,32 @@ #define platform_SHA1_Final SHA1_Final #endif +#ifndef platform_SHA_CTX_fast +# define platform_SHA_CTX_fast platform_SHA_CTX +# define platform_SHA1_Init_fast platform_SHA1_Init +# define platform_SHA1_Update_fast platform_SHA1_Update +# define platform_SHA1_Final_fast platform_SHA1_Final +# ifdef platform_SHA1_Clone +# define platform_SHA1_Clone_fast platform_SHA1_Clone +# endif +#endif + #define git_SHA_CTX platform_SHA_CTX #define git_SHA1_Init platform_SHA1_Init #define git_SHA1_Update platform_SHA1_Update #define git_SHA1_Final platform_SHA1_Final +#define git_SHA_CTX_fast platform_SHA_CTX_fast +#define git_SHA1_Init_fast platform_SHA1_Init_fast +#define git_SHA1_Update_fast platform_SHA1_Update_fast +#define git_SHA1_Final_fast platform_SHA1_Final_fast + #ifdef platform_SHA1_Clone #define git_SHA1_Clone platform_SHA1_Clone #endif +#ifdef platform_SHA1_Clone_fast +# define git_SHA1_Clone_fast platform_SHA1_Clone_fast +#endif #ifndef platform_SHA256_CTX #define platform_SHA256_CTX SHA256_CTX @@ -81,6 +99,13 @@ static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *src) memcpy(dst, src, sizeof(*dst)); } #endif +#ifndef SHA1_NEEDS_CLONE_HELPER_FAST +static inline void git_SHA1_Clone_fast(git_SHA_CTX_fast *dst, + const git_SHA_CTX_fast *src) +{ + memcpy(dst, src, sizeof(*dst)); +} +#endif #ifndef SHA256_NEEDS_CLONE_HELPER static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *src) @@ -178,6 +203,8 @@ enum get_oid_result { /* A suitably aligned type for stack allocations of hash contexts. */ union git_hash_ctx { git_SHA_CTX sha1; + git_SHA_CTX_fast sha1_fast; + git_SHA256_CTX sha256; }; typedef union git_hash_ctx git_hash_ctx; @@ -222,6 +249,21 @@ struct git_hash_algo { /* The hash finalization function for object IDs. */ git_hash_final_oid_fn final_oid_fn; + /* The fast / non-cryptographic hash initialization function. */ + git_hash_init_fn fast_init_fn; + + /* The fast / non-cryptographic hash context cloning function. */ + git_hash_clone_fn fast_clone_fn; + + /* The fast / non-cryptographic hash update function. */ + git_hash_update_fn fast_update_fn; + + /* The fast / non-cryptographic hash finalization function. */ + git_hash_final_fn fast_final_fn; + + /* The fast / non-cryptographic hash finalization function. */ + git_hash_final_oid_fn fast_final_oid_fn; + /* The OID of the empty tree. */ const struct object_id *empty_tree; diff --git a/object-file.c b/object-file.c index 85f91516429..84dd7d0fab6 100644 --- a/object-file.c +++ b/object-file.c @@ -115,6 +115,33 @@ static void git_hash_sha1_final_oid(struct object_id *oid, git_hash_ctx *ctx) oid->algo = GIT_HASH_SHA1; } +static void git_hash_sha1_init_fast(git_hash_ctx *ctx) +{ + git_SHA1_Init_fast(&ctx->sha1_fast); +} + +static void git_hash_sha1_clone_fast(git_hash_ctx *dst, const git_hash_ctx *src) +{ + git_SHA1_Clone_fast(&dst->sha1_fast, &src->sha1_fast); +} + +static void git_hash_sha1_update_fast(git_hash_ctx *ctx, const void *data, + size_t len) +{ + git_SHA1_Update_fast(&ctx->sha1_fast, data, len); +} + +static void git_hash_sha1_final_fast(unsigned char *hash, git_hash_ctx *ctx) +{ + git_SHA1_Final_fast(hash, &ctx->sha1_fast); +} + +static void git_hash_sha1_final_oid_fast(struct object_id *oid, git_hash_ctx *ctx) +{ + git_SHA1_Final_fast(oid->hash, &ctx->sha1_fast); + memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ); + oid->algo = GIT_HASH_SHA1; +} static void git_hash_sha256_init(git_hash_ctx *ctx) { @@ -189,6 +216,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = { .update_fn = git_hash_unknown_update, .final_fn = git_hash_unknown_final, .final_oid_fn = git_hash_unknown_final_oid, + .fast_init_fn = git_hash_unknown_init, + .fast_clone_fn = git_hash_unknown_clone, + .fast_update_fn = git_hash_unknown_update, + .fast_final_fn = git_hash_unknown_final, + .fast_final_oid_fn = git_hash_unknown_final_oid, .empty_tree = NULL, .empty_blob = NULL, .null_oid = NULL, @@ -204,6 +236,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = { .update_fn = git_hash_sha1_update, .final_fn = git_hash_sha1_final, .final_oid_fn = git_hash_sha1_final_oid, + .fast_init_fn = git_hash_sha1_init_fast, + .fast_clone_fn = git_hash_sha1_clone_fast, + .fast_update_fn = git_hash_sha1_update_fast, + .fast_final_fn = git_hash_sha1_final_fast, + .fast_final_oid_fn = git_hash_sha1_final_oid_fast, .empty_tree = &empty_tree_oid, .empty_blob = &empty_blob_oid, .null_oid = &null_oid_sha1, @@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = { .update_fn = git_hash_sha256_update, .final_fn = git_hash_sha256_final, .final_oid_fn = git_hash_sha256_final_oid, + .fast_init_fn = git_hash_sha256_init, + .fast_clone_fn = git_hash_sha256_clone, + .fast_update_fn = git_hash_sha256_update, + .fast_final_fn = git_hash_sha256_final, + .fast_final_oid_fn = git_hash_sha256_final_oid, .empty_tree = &empty_tree_oid_sha256, .empty_blob = &empty_blob_oid_sha256, .null_oid = &null_oid_sha256, From patchwork Fri Sep 6 19:46:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794712 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2186E1D79A4 for ; Fri, 6 Sep 2024 19:46:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651992; cv=none; b=SrPsCSOajP+3Bt1rc3X2MPULYmbu1Ay5EnnkxVOjD5UGiyGt4pCez0x2V3WET0LWzZIwgQKk0XpwMlemYaLKMabmdtCWQO962ivIVg2H4yVgJcSzSYxq0kBLfhbX5z4L9h1LNIQ0D0QZNIQIysDaI2qsk2Emk1XMTfi8SO3Jxbo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651992; c=relaxed/simple; bh=yU2XLiil/Dgd1SR2dDh984LhHX+sGVOf/Y/qgC+kWAk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=evGEv3MWtCX6rhBFminYfRB1AQHuVJcw3hr078IGwoXGk80vo555akc38K8Co7O3xkXjy9F1r5s50b8t4rm1/ys5dDTLrOjTZThv07G2hrIHHBRAZLWrsBSuKNBVuhMqxg6cQ3TcOhPgz0OtCymI5gliGB8fPLrXIR67GkME2cw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=NG1wlJnh; arc=none smtp.client-ip=209.85.128.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="NG1wlJnh" Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-699ac6dbf24so21783037b3.3 for ; Fri, 06 Sep 2024 12:46:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651990; x=1726256790; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=EkgFMJYvEuhrH5mGmGFeZ4T8Q78nXa3QWTMrxfpu9RI=; b=NG1wlJnh6H5R8hV7E2zUDrKx0NS1EXxzCcAGkjjZJIvVIxoyFn4YwcPClXZHD7jQ7e uJ+UFqyX385UwmySqGdo1wyaec+FjgZRM1q5goVF3BZ6vrHs57mFoT9h1Wm1yvGO7tBf qQfBeubZlb5sNhEBJFEkwoYn4nEnEwK285TJDqTl5+IMdIbPMDPtd2tw/oTpei9ZXIJ/ UZVLKF8blVszLT2LeUNvQQcyYvLaIoiTk0qJ4KN29EX7AFs4rRnQ4cQ3GYmuver8QQ15 c2vajPF8KHAaaR0LZEyCTydXqmvcOjrulQDTVqNB7o1jmRFum20eOME96XwVBkrDgZ+J QApA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651990; x=1726256790; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=EkgFMJYvEuhrH5mGmGFeZ4T8Q78nXa3QWTMrxfpu9RI=; b=NdZRvWWln69miZzjg298SjnwRqgN57dI4mbUlLkP8N7uyV+tTu/cCXIQHhTgdujHke /lyKJ9SWG3tfIH+ZAsXshkYp0/ocRfjMdtP09yi9vOC0HP/gZCVIvOinMgvMu2sxSfqO rrVU3cWsFWEWJQsEzeCXshWEuCorYytucmKiMHFlTPaDkh5pFKqh9KHGbu2kiU5Aq+i5 MPAvl8ANSeldRzIU/e3bETCEiG6a0+KyCa2Qj/0qfsl6PAB25HS81P1sWD0C/L325f0y und0K2F6fiQTbnCkrLK9/CbMqVHImvwkvHhHlWisiLwpj7KHRveJEvEVG5djD/WsEZVB HorQ== X-Gm-Message-State: AOJu0Yyd5pf2uIetr3Xb8h1fzdTktq0eSqlryzmUILj7oNiaO94dS8Ux RY8I3XLa8P7xXcfunSZtlgHj6wVix0L1L9tZIc666v34z96XxNJc+IW/XF4HA+J3o3Fmb7xTdlT ci6g= X-Google-Smtp-Source: AGHT+IHMZOyWccxzVOMDrURsiHTtWSsovzhAjcli5sJBwzOtxuOW4xlrzcfhIREiixVoF9+QzXwZCg== X-Received: by 2002:a05:690c:2f0a:b0:6d3:f51b:38a3 with SMTP id 00721157ae682-6db4515475amr29423517b3.33.1725651989793; Fri, 06 Sep 2024 12:46:29 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db5631a7a8sm1142077b3.23.2024.09.06.12.46.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:29 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:28 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 8/9] Makefile: allow specifying a SHA-1 for non-cryptographic uses Message-ID: <137ec30d68a7cf60e738951a7bf283ef78e44345.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Introduce _FAST variants of the OPENSSL_SHA1, BLK_SHA1, and APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1 implementation is to be used for non-cryptographic uses. There are a couple of small implementation notes worth mentioning: - There is no way to select the collision detecting SHA-1 as the "fast" fallback, since the fast fallback is only for non-cryptographic uses, and is meant to be faster than our collision-detecting implementation. - There are no similar knobs for SHA-256, since no collision attacks are presently known and thus no collision-detecting implementations actually exist. Signed-off-by: Taylor Blau --- Makefile | 25 +++++++++++++++++++++++++ hash.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/Makefile b/Makefile index e298c8b55ec..d24f9088802 100644 --- a/Makefile +++ b/Makefile @@ -517,6 +517,10 @@ include shared.mak # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for # SHA-1. # +# Define the same Makefile knobs as above, but suffixed with _FAST to +# use the corresponding implementations for "fast" SHA-1 hashing for +# non-cryptographic purposes. +# # If don't enable any of the *_SHA1 settings in this section, Git will # default to its built-in sha1collisiondetection library, which is a # collision-detecting sha1 This is slower, but may detect attempted @@ -1982,6 +1986,27 @@ endif endif endif +ifdef OPENSSL_SHA1_FAST +ifndef OPENSSL_SHA1 + EXTLIBS += $(LIB_4_CRYPTO) + BASIC_CFLAGS += -DSHA1_OPENSSL_FAST +endif +else +ifdef BLK_SHA1_FAST +ifndef BLK_SHA1 + LIB_OBJS += block-sha1/sha1.o + BASIC_CFLAGS += -DSHA1_BLK_FAST +endif +else +ifdef APPLE_COMMON_CRYPTO_SHA1_FAST +ifndef APPLE_COMMON_CRYPTO_SHA1 + COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL + BASIC_CFLAGS += -DSHA1_APPLE_FAST +endif +endif +endif +endif + ifdef OPENSSL_SHA256 EXTLIBS += $(LIB_4_CRYPTO) BASIC_CFLAGS += -DSHA256_OPENSSL diff --git a/hash.h b/hash.h index 5e5b8205b58..6ea0d968017 100644 --- a/hash.h +++ b/hash.h @@ -15,6 +15,36 @@ #include "block-sha1/sha1.h" #endif +#if defined(SHA1_APPLE_FAST) +# include +# define platform_SHA_CTX_fast CC_SHA1_CTX +# define platform_SHA1_Init_fast CC_SHA1_Init +# define platform_SHA1_Update_fast CC_SHA1_Update +# define platform_SHA1_Final_fast CC_SHA1_Final +#elif defined(SHA1_OPENSSL_FAST) +# include +# if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3 +# define SHA1_NEEDS_CLONE_HELPER_FAST +# include "sha1/openssl.h" +# define platform_SHA_CTX_fast openssl_SHA1_CTX +# define platform_SHA1_Init_fast openssl_SHA1_Init +# define platform_SHA1_Clone_fast openssl_SHA1_Clone +# define platform_SHA1_Update_fast openssl_SHA1_Update +# define platform_SHA1_Final_fast openssl_SHA1_Final +# else +# define platform_SHA_CTX_fast SHA_CTX +# define platform_SHA1_Init_fast SHA1_Init +# define platform_SHA1_Update_fast SHA1_Update +# define platform_SHA1_Final_fast SHA1_Final +# endif +#elif defined(SHA1_BLK_FAST) +# include "block-sha1/sha1.h" +# define platform_SHA_CTX_fast blk_SHA_CTX +# define platform_SHA1_Init_fast blk_SHA1_Init +# define platform_SHA1_Update_fast blk_SHA1_Update +# define platform_SHA1_Final_fast blk_SHA1_Final +#endif + #if defined(SHA256_NETTLE) #include "sha256/nettle.h" #elif defined(SHA256_GCRYPT) From patchwork Fri Sep 6 19:46:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13794713 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF6E21D554 for ; Fri, 6 Sep 2024 19:46:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651995; cv=none; b=GKdpgKrBfG5XxRHZPDrX2sWvxveExiE/iE9NWBU+V5S6gUl9zPjZa+JddVHPhO6hd/G9EYb/QA3BE53lHacmTev3186Ug717kFWmlD56ancpz/5wDU/y/vR5nzj7MNl5Hs9W0B0MFuGPXV127rPv9GhcPJa35IpyYA+tV1agM+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725651995; c=relaxed/simple; bh=52xnQ0mERdPyf52nWx8znCGGXcH1VWXywgo0DOYbK68=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Js1Lx82dt530Yj2FrPTkZfM3dTnrQWwlKzGVaY8xveepfEYG1xsaYMKgCILAGzwlxG1SXAaCzkYwfBdcwqhvsb3fliLtjEBai2kyt2YocCs/M8x10DdU77mpmXH/rXEP03VMrkR2fFVJF3/QZPsraMgv2z7D9gq0TA8I7SjgZY4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=QsKHeTGk; arc=none smtp.client-ip=209.85.219.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="QsKHeTGk" Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-e116d2f5f7fso2848908276.1 for ; Fri, 06 Sep 2024 12:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1725651993; x=1726256793; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=1t1A0Fv5jfIbdu5SYL/DVcAH9CI/7kXlCBnYJscFfnY=; b=QsKHeTGkHseksV0noiQqeq6lKcka/eJuRbsKd7j5ezTxG4yGO4K+wfxald2nlpJR2n h0Yp1AcrNL1C2yrcFGgC19dVB0sM9hMlVkbQHjSOva5KqyQZRvE2A9c+Losw34UxDQ61 RdpT52/pn7wdW8oE5da9IX5V2gFDh/0X7YCzXNdKAXH4TDqrtai9bAGCRnLD/kClZ5Ws 7fBL/g987HJV6KhryJxmHQl3WeZoFXR5s9X9MuwLVTxJe9FdJ16VMBZa3+bWjmyCeRoh TrejG7qxTfPmGQ0AVLVxgrCaOMPa3H4eJPaX85WbH74+zMn37sihSgUAl+3mzr7Ogkt0 Q5sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725651993; x=1726256793; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=1t1A0Fv5jfIbdu5SYL/DVcAH9CI/7kXlCBnYJscFfnY=; b=h5MWTcLQk5SN1kp4732fcLwf6533i/Q5rXO9VstpZBqWr4ack2tIxzgyMWTJVwcQBs 4gpT4tnLqBtLsx6XKhKYrak7WI3Lt9VgsG/QqeLUaYM1JYaSj3ERgDTONQRM+eb9eKtK YJLn+TK2vNpeTfjjleaHTOHtrHqaupDlKi0ldoaxzmTFFDKlZZDPYfmjTLWnM8fZMpjl UOZFTZRN2h4zBywc7F0+fmw2lip5EvYEHeeH+nHmUwmFsiKKa2tIxS8om2UZAx3QJqjo wIQxS0yQ25/Rr1/mCI09J8Q1+AKivj29VKEYMpCUx1XpamLTUqjf0xm9Di/PQ6bVyAf5 +LYA== X-Gm-Message-State: AOJu0YyQpV6bmwgVx1p2PAu+BnkZyMu/YIXzhpqzd+jFWqL8QWYW+vjk XpjC8UN6XiitEuZB6oQ29OEXhIMYpZA0A55i17dljtmr8eNvrqqad81PUAvI6iWHOAwdDG8n7yP s2jE= X-Google-Smtp-Source: AGHT+IH1HxdMko5kRgIbHRnLJ4XxDLy09W2exZKnOAeufhVxb4iOi0TRAWkuOkhcgL+mcp6mFmrfhw== X-Received: by 2002:a05:690c:760a:b0:6b7:f467:e0f5 with SMTP id 00721157ae682-6db442bb283mr29950557b3.9.1725651992796; Fri, 06 Sep 2024 12:46:32 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6db5631a093sm1154197b3.22.2024.09.06.12.46.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 12:46:32 -0700 (PDT) Date: Fri, 6 Sep 2024 15:46:31 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , "brian m. carlson" , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v3 9/9] csum-file.c: use fast SHA-1 implementation when available Message-ID: <4018261366fe2de2def2ec6093ccfb51ef768a7b.1725651952.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Update hashwrite() and friends to use the fast_-variants of hashing functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead of "the_hash_algo->update_fn()". These callers only use the_hash_algo to produce a checksum, which we depend on for data integrity, but not for cryptographic purposes, so these callers are safe to use the fast (and potentially non-collision detecting) SHA-1 implementation. To time this, I took a freshly packed copy of linux.git, and ran the following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both versions were compiled with -O3: $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in $ valgrind --tool=callgrind ~/src/git/git-pack-objects \ --revs --stdout --all-progress --use-bitmap-index /dev/null Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting SHA-1 implementation for both cryptographic and non-cryptographic purposes), we spend a significant amount of our instruction count in hashwrite(): $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1 159,998,868,413 (79.42%) /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects] , and the resulting "clone" takes 19.219 seconds of wall clock time, 18.94 seconds of user time and 0.28 seconds of system time. Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in hashwrite(): $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1 59,164,001,176 (58.79%) /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects] , and generate the resulting "clone" much faster, in only 11.597 seconds of wall time, 11.37 seconds of user time, and 0.23 seconds of system time, for a ~40% speed-up. Signed-off-by: Taylor Blau --- csum-file.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/csum-file.c b/csum-file.c index bf82ad8f9f5..cb8c39ecf3a 100644 --- a/csum-file.c +++ b/csum-file.c @@ -50,7 +50,7 @@ void hashflush(struct hashfile *f) if (offset) { if (!f->skip_hash) - the_hash_algo->update_fn(&f->ctx, f->buffer, offset); + the_hash_algo->fast_update_fn(&f->ctx, f->buffer, offset); flush(f, f->buffer, offset); f->offset = 0; } @@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, if (f->skip_hash) hashclr(f->buffer, the_repository->hash_algo); else - the_hash_algo->final_fn(f->buffer, &f->ctx); + the_hash_algo->fast_final_fn(f->buffer, &f->ctx); if (result) hashcpy(result, f->buffer, the_repository->hash_algo); @@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count) * f->offset is necessarily zero. */ if (!f->skip_hash) - the_hash_algo->update_fn(&f->ctx, buf, nr); + the_hash_algo->fast_update_fn(&f->ctx, buf, nr); flush(f, buf, nr); } else { /* @@ -174,7 +174,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name, f->name = name; f->do_crc = 0; f->skip_hash = 0; - the_hash_algo->init_fn(&f->ctx); + the_hash_algo->fast_init_fn(&f->ctx); f->buffer_len = buffer_len; f->buffer = xmalloc(buffer_len); @@ -208,7 +208,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo { hashflush(f); checkpoint->offset = f->total; - the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx); + the_hash_algo->fast_clone_fn(&checkpoint->ctx, &f->ctx); } int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint) @@ -219,7 +219,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint lseek(f->fd, offset, SEEK_SET) != offset) return -1; f->total = offset; - the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx); + the_hash_algo->fast_clone_fn(&f->ctx, &checkpoint->ctx); f->offset = 0; /* hashflush() was called in checkpoint */ return 0; } @@ -245,9 +245,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len) if (total_len < the_hash_algo->rawsz) return 0; /* say "too short"? */ - the_hash_algo->init_fn(&ctx); - the_hash_algo->update_fn(&ctx, data, data_len); - the_hash_algo->final_fn(got, &ctx); + the_hash_algo->fast_init_fn(&ctx); + the_hash_algo->fast_update_fn(&ctx, data, data_len); + the_hash_algo->fast_final_fn(got, &ctx); return hasheq(got, data + data_len, the_repository->hash_algo); }