From patchwork Mon May 4 19:03:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527431 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D197F81 for ; Mon, 4 May 2020 19:03:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B966E206C0 for ; Mon, 4 May 2020 19:03:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BVDn2qp5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726531AbgEDTDx (ORCPT ); Mon, 4 May 2020 15:03:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725981AbgEDTDx (ORCPT ); Mon, 4 May 2020 15:03:53 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE700C061A0E for ; Mon, 4 May 2020 12:03:52 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id x18so419632wrq.2 for ; Mon, 04 May 2020 12:03:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=q9NuaAvYc4TMXl8Vc00eXBn/gp7cRyZftwBh5NUsgbc=; b=BVDn2qp5JFnkpNnsQOnZQvX9vw6rzHbXP30fOcOkKPkaCufy2zZrbT9QLyV24q93iq TX5fGOAm1fUfuEjYbc8wFo/4a658+KOp60pi/z0la3PKZtm4Ybv2Ms+KqcQE6tQa+Ury oaFQ/08kveaI1DGV0V9QUePC9un5VfTszuyqhPhwqvtC2zZxu9SpNGe2RQHNcONCiz8M 4m76gVt901axUiMxzjWLniRA+asSHLLPhyi9CpRpNMDhSYIc+QXLALDXGDqxsGqxDB8t 0zlmzAvfnt0xKwe7KbjqrhInrzFLhXRKQYAIlDCCrOrI9rnC+UWiiTtdjMCQ+nXnHpuV scJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=q9NuaAvYc4TMXl8Vc00eXBn/gp7cRyZftwBh5NUsgbc=; b=CGg+DFB2FNqpUwQOSWhCShFixbX3dl2Z7En8GhEnMNYiYHmUVSDG6ooQePdnXm8znu Cd1HjufajcP27hUsAJdbc8GtrMU2BuNFltOQDEcYlmNN13tfOi9YR8G6WTIeP0L2DEpD EYgo8HxyRr5wjdsB6IIgUKNaFqXORiWIrUzbGxvJ/to+W+RZyXF9XXc8jQ+NPLVjqR3f 3nbCjz07E2JHhUIr8qlRlNCKnNHfn5F0TvL6dw1V88iIbEweGI0Q1bVgWakDhAxWUZAu siyKcKfDvUXjn5+OMTy3V41TMyD7YfIFT11emsW23cbJrGP2cXsiDdZz+H+Qlv9gXi/w +Zcw== X-Gm-Message-State: AGi0PubGBG7eNovqv6eHo0dvatQ1BSsYlTdsShQnM94JtqfsdJTom+Ej ZYphzVYEmytq0W0yrcMoW/tgkblf X-Google-Smtp-Source: APiQypKH6hDGOcT98Naxsoi96dgoPBilZRwZzvC2wJIbQmKrhLAJ+JymuDggArXWplJyCEzya/dPbg== X-Received: by 2002:a5d:68cb:: with SMTP id p11mr520076wrw.349.1588619031447; Mon, 04 May 2020 12:03:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h2sm542553wmf.34.2020.05.04.12.03.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:51 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:37 +0000 Subject: [PATCH v11 01/12] refs.h: clarify reflog iteration order Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys Signed-off-by: Han-Wen Nienhuys --- refs.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/refs.h b/refs.h index a92d2c74c83..9421c5b8465 100644 --- a/refs.h +++ b/refs.h @@ -432,18 +432,21 @@ int delete_refs(const char *msg, struct string_list *refnames, int refs_delete_reflog(struct ref_store *refs, const char *refname); int delete_reflog(const char *refname); -/* iterate over reflog entries */ +/* Iterate over reflog entries. */ typedef int each_reflog_ent_fn( struct object_id *old_oid, struct object_id *new_oid, const char *committer, timestamp_t timestamp, int tz, const char *msg, void *cb_data); +/* Iterate in over reflog entries, oldest entry first. */ int refs_for_each_reflog_ent(struct ref_store *refs, const char *refname, each_reflog_ent_fn fn, void *cb_data); int refs_for_each_reflog_ent_reverse(struct ref_store *refs, const char *refname, each_reflog_ent_fn fn, void *cb_data); + +/* Call a function for each reflog entry, oldest entry first. */ int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data); int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data); From patchwork Mon May 4 19:03:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527433 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8E1781 for ; Mon, 4 May 2020 19:03:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B038020721 for ; Mon, 4 May 2020 19:03:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="B/xsHBVB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726404AbgEDTDy (ORCPT ); Mon, 4 May 2020 15:03:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725981AbgEDTDy (ORCPT ); Mon, 4 May 2020 15:03:54 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B8B8C061A0E for ; Mon, 4 May 2020 12:03:53 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id d15so415554wrx.3 for ; Mon, 04 May 2020 12:03:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XTW+S1tjt/UjHTmSLUWWzSNcFWS/aoQftFrdnmqTgX8=; b=B/xsHBVBGXUSnFUxKw7EzqjTV+Aq8JVhHw/v0HelxxGO8meonG0Bw5OOdqkWnJqc6p ptRtWpKFVIdCzPyIT9HPpVhsTWcCZvEeM7F+J4gjwVhBGPFBhUR0+toQZv1Ec5BcRrDq ocIa8nBfYLMelbUy7tZBZU38AAfYOtwHKBgrSlnIt1WkkChxfCgtU4XhDfyjbRiA+e6z MaLA1EpWVbhb+tXVI1XGM+hFrMs2thIAdB1sInyEwqscrxCguWILKhsLsBRgd6/EqDuU ajHwJaAcyQoXufFTy11Ms/LzSNhQmfib0hBUcrwS+Qwi8dqsMREXTcBZae2aO/SrQxDe /IbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XTW+S1tjt/UjHTmSLUWWzSNcFWS/aoQftFrdnmqTgX8=; b=bgGJH3sc6e62wgR6Za+SAVwG0gV1nRL9nDNeB5TQptk1CHInYxOOx+wYFf+vQBpOIE UJxocmulvT5kqCdlYlbox/KLIhNHfRvvpQuDxN8k8fcYnHBM9y623Lp+79n5deRZQ2sD naBq0gJ8RJHkn2GEtXpl6PVZybTuRmjalGD5FdUYnw4ZEA5zqACxE1b3Z93Fbo/tVIt6 vm8L2+OZbckLlaPOu81OOIG5aNeYMWK/aqhAIsPe4LgXYk/wrbDis0VgmrnXZw1EhQWf C1vIgDCt6vh5J2pwId9XI3a+tZzb1EHyGwNQXALRr6Bl375vZA8DhUmg45nDyPqC2byc SIKQ== X-Gm-Message-State: AGi0PuYgFrsVSPc6HA28aJsnpmKO6oJ6DkRT/jN2C8NTNC8OhRfI6OWt WxO/UYGrvUSQeww9euokSJzL5awR X-Google-Smtp-Source: APiQypI1uEHZP/AqH57+5HQEa1aZ8UXJivrsB+i08G32gymz6aGtbnp0x9DuI9QZ/nagd2gHRQJCkA== X-Received: by 2002:a5d:4704:: with SMTP id y4mr871464wrq.96.1588619032259; Mon, 04 May 2020 12:03:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a125sm588409wme.3.2020.05.04.12.03.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:51 -0700 (PDT) Message-Id: <340c5c415e17a957b8a38932215b1179fbe68dbd.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:38 +0000 Subject: [PATCH v11 02/12] Iterate over the "refs/" namespace in for_each_[raw]ref Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys This happens implicitly in the files/packed ref backend; making it explicit simplifies adding alternate ref storage backends, such as reftable. Signed-off-by: Han-Wen Nienhuys --- refs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/refs.c b/refs.c index 224ff66c7bb..4db27379661 100644 --- a/refs.c +++ b/refs.c @@ -1525,7 +1525,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix, int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data) { - return do_for_each_ref(refs, "", fn, 0, 0, cb_data); + return do_for_each_ref(refs, "refs/", fn, 0, 0, cb_data); } int for_each_ref(each_ref_fn fn, void *cb_data) @@ -1585,8 +1585,8 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data) int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data) { - return do_for_each_ref(refs, "", fn, 0, - DO_FOR_EACH_INCLUDE_BROKEN, cb_data); + return do_for_each_ref(refs, "refs/", fn, 0, DO_FOR_EACH_INCLUDE_BROKEN, + cb_data); } int for_each_rawref(each_ref_fn fn, void *cb_data) From patchwork Mon May 4 19:03:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527439 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7CEE92A for ; Mon, 4 May 2020 19:04:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A0EDE20721 for ; Mon, 4 May 2020 19:04:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="H9s8XGj2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727815AbgEDTD6 (ORCPT ); Mon, 4 May 2020 15:03:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727802AbgEDTDz (ORCPT ); Mon, 4 May 2020 15:03:55 -0400 Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98E75C061A0E for ; Mon, 4 May 2020 12:03:54 -0700 (PDT) Received: by mail-wm1-x341.google.com with SMTP id e26so669356wmk.5 for ; Mon, 04 May 2020 12:03:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4D690/NLx/9daAF2hWG5tCfDYzmNZsPcuZnuRxKSdQk=; b=H9s8XGj2Ja3Z2zhPQSmElY2AkNxqG9lnC4OS/prY3Fhz2tkZWZn3Ic0sXHf/aAx9ek NXamJOJ2kpPTidDF75yMA2fhE9+P/2YVxaBHyxWa0zuLMwV9rirhWkdBesEKHS3+q41F LkeCM7h15l3DudrgZQz5RIKf/Ze/jaxqSHcT75lstqvAWHd3kFZn+bN85UGldhJFHos9 vYiMW0V41L6dHsAFik8480ro2bPk3S/x4qvx8PlZi6mqKfEJEXKB/pZ8+Su9jpboXI/2 8h5mpQ2g89L0w72f9+dy757WpsfmsZp77yu55PVhgq7PkEdJvVKYkMHI/v+bxxS2x1qy pwgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4D690/NLx/9daAF2hWG5tCfDYzmNZsPcuZnuRxKSdQk=; b=fidsGWIEaR1HXhJYU/bsADlHjAVNEB1LTY/BZoKRXUEyBv9chFIKUo7f98B7LXh5V4 1AJvfPJujMeVVQiuMw6STCTN4+ozXNhT07qpdTPGo3tY1K2RYX4sX+cMOSu/TJ/UDgv3 gTlVvSQFV8gxpWbSS1bTm4416s6H55tSoA8X6GVzudKkorGHXl2TPtfRKRymzPn8b/e1 pBsjZ677rZavsofgmSv189NcS8G8KRG15/1rfAQz/qIwhEvvUVlP6uAxS48kbNsj7i+Z ZJJqzRmNEWgk6Y9n3x63cN/Wz1WiVB0QegKNHYmVHo3aj1hL6Xi5kI9vQnQMd1CiHBZy o9Tg== X-Gm-Message-State: AGi0PuaXCV+wGdMhoUd/In/HLiWK4/ly5qfVdkMwX/qTftoKHdrcy4rY Xq2rQ3t1e6i6H4VHn2twN1oamK5m X-Google-Smtp-Source: APiQypLtnMwRan7X4ssJdiXeOCCq8IRDzrS/ORu/5Xp++ObDcUUhO2dZOXQaLnGi5Xb/L5C8FsFUdg== X-Received: by 2002:a1c:b757:: with SMTP id h84mr16140971wmf.188.1588619033264; Mon, 04 May 2020 12:03:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w15sm3362999wrl.73.2020.05.04.12.03.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:52 -0700 (PDT) Message-Id: <6553285043b2f004cca646aefd59c509324d4da3.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:39 +0000 Subject: [PATCH v11 03/12] refs: document how ref_iterator_advance_fn should handle symrefs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys Signed-off-by: Han-Wen Nienhuys --- refs/refs-internal.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/refs/refs-internal.h b/refs/refs-internal.h index ff2436c0fb7..3490aac3a40 100644 --- a/refs/refs-internal.h +++ b/refs/refs-internal.h @@ -438,6 +438,11 @@ void base_ref_iterator_free(struct ref_iterator *iter); /* Virtual function declarations for ref_iterators: */ +/* + * backend-specific implementation of ref_iterator_advance. + * For symrefs, the function should set REF_ISSYMREF, and it should also + * dereference the symref to provide the OID referent. + */ typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator); typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator, From patchwork Mon May 4 19:03:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527435 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 821AA81 for ; Mon, 4 May 2020 19:03:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 613F0206C0 for ; Mon, 4 May 2020 19:03:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nMVOlHLa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727823AbgEDTD6 (ORCPT ); Mon, 4 May 2020 15:03:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725981AbgEDTDz (ORCPT ); Mon, 4 May 2020 15:03:55 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6213EC061A10 for ; Mon, 4 May 2020 12:03:55 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id e16so384423wra.7 for ; Mon, 04 May 2020 12:03:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=1g7xJNEHljtESnR5/L4oArBHPSBJu5a8+IEAIVDCwIk=; b=nMVOlHLaPU9h8vvvFevmjhknuB+DxkZK5S/BDUdrjD6TlVIpko0wE+Ti0R92nYUac7 HeLQnhaalKJ6YOWnousVB2W/hdlfLE+glLyPW4ci6tHWOO+5fEOFHaQ+cAufF/zClDoU fvM8qUwGw/qgxrtm/nlhDJTsZprzlf57vQIJwR1UOBwcfLQNWSp9jF3KjfswAI6BXTpg IL6uVvfO32SoUqnuRy9UbQY2Z5W1XEdgbuFd9oojHH9UiEcgqGhlwbNYnKTsHTL2L52e SsVzn7nR+6BKLDeijSG7ZvojpL/vrf59Q3AyxEvnPSul50rgKR7FqD/g6KNFRkDiw6mh hMqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=1g7xJNEHljtESnR5/L4oArBHPSBJu5a8+IEAIVDCwIk=; b=ORDXCBjJ/WU+ogji2R2x75gh+vE2hNppoFUlQAakTYHy8sFWh7djr2D9DzsStuJVoy +/IM1pWpy1/KXsr0WI7IrhCq9Yyxwy2wI1PpmhGV9RV21i5h9GFX0AxPqGuIog5yUW8l h8KNsDMM07vS4JHXax7Qit7l3MLXtYX9f0Xtr1AF6esyzmlG9RSB1pPRTZxXWWcq3kCO MtuC1FYnT82GkSpGWP82TMttCS+cKV3hcJ1Yw+5ZQgUouVLbp+oAHfH9+NmdzkTlXwhB QZYtPWTCJTGpV5U2/GFARRn1sasxEXJ5Zis0mLsT/DbiDojHKAEvJwBVTJAnhNX1b7G9 6XEg== X-Gm-Message-State: AGi0PuYRPuV+lnbRBjLEi9kbGJKeIFG9BcLxlH8+Er7L8qC/r7taQgAQ 5ba8ypHk5tqQRICjX87P0NWYfhfx X-Google-Smtp-Source: APiQypJe9qz4yfasMb7/2RVn+EK/NhU8Hu26pXPopVMMBBXUQlsn8+lvHbV8JPmK9jJxTwSvwxtV/Q== X-Received: by 2002:adf:a297:: with SMTP id s23mr860847wra.54.1588619034053; Mon, 04 May 2020 12:03:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k5sm3987430wrx.16.2020.05.04.12.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:53 -0700 (PDT) Message-Id: <7dc47c7756f67522d6279d1936e9d43d9438f215.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:40 +0000 Subject: [PATCH v11 04/12] Add .gitattributes for the reftable/ directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys Signed-off-by: Han-Wen Nienhuys --- reftable/.gitattributes | 1 + 1 file changed, 1 insertion(+) create mode 100644 reftable/.gitattributes diff --git a/reftable/.gitattributes b/reftable/.gitattributes new file mode 100644 index 00000000000..f44451a3795 --- /dev/null +++ b/reftable/.gitattributes @@ -0,0 +1 @@ +/zlib-compat.c whitespace=-indent-with-non-tab,-trailing-space From patchwork Mon May 4 19:03:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527445 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D83492A for ; Mon, 4 May 2020 19:04:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 70B3320721 for ; Mon, 4 May 2020 19:04:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uHPN7HTp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727834AbgEDTEB (ORCPT ); Mon, 4 May 2020 15:04:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727824AbgEDTD7 (ORCPT ); Mon, 4 May 2020 15:03:59 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01992C061A0F for ; Mon, 4 May 2020 12:03:58 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id e16so384552wra.7 for ; Mon, 04 May 2020 12:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=I9f5x2FewgXkAlm4SqEHJD1HcpjENre8rgzTF5FWU0E=; b=uHPN7HTpUw1WfpdfdfyqaVAsC4EdsegbuFKp9fKrNMYq+HXZ2SSj1akDdlL5GKNlMd fI1LDB7QOeUcA3W62qtsZTvuvy8rvdnLTYKf01A0DGideROHUG+VoFao8vMN2ydfJBA0 Ru/07NG3zasTEDivjn5Pzy2zjgV/MoxofCB+qlZxxAPFIZDA0OSlwZo2k5YxHz3VCQ6N ObneOpIvQGRLnIwuvvXSyX2QMGVzwFKqgVPMNH+YDiPkd0KAXPH0Bf4tP3ldVTOD1N/x vBr3v7X5UCUkoUiTAPEDFluTPnRsrO+KBGvyUyJKb9JYC9224WdaR2AvbC5enUbcWy3A Alkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=I9f5x2FewgXkAlm4SqEHJD1HcpjENre8rgzTF5FWU0E=; b=b+hT1sDWyyZ7urCmR5CniCwJxNXj2qrBG4hdTVWmKwcf6A9vllHvPUpzKc0E4VU+rD 486q4T+rr1nd4DFILYXfVY9bqIP16LaBWySRQdVIBSdh/GaB7JbtFF873+RUN4gX4mEg 9/xU450J5AUj7HZG1BsUAjmjsyV+JZ35254YFzlfTQ4Ao9A/UwmGMm+eefW493gLfzcf 2qGI5KjLPdSPT9q381dYdcyIpRcbaeiDrFACtTYg/vm5K70AuOehRegDHWcI5g6VODEK d++23X8bQ1mJFY3R5Jq22mdVMNvqjPkhYl+uRPBFTCuaNUniICrPnh0d5GFVcuPjwjlH gLYg== X-Gm-Message-State: AGi0Pua+LbQzsSisWJaEdFe1ZQaVb6Aw5GzGTz3AXWq3Rs2vpnIAaDAH ZdzomdHSIQAZU0JbhZ+sY/GUtN7q X-Google-Smtp-Source: APiQypI9fXHr6+dBFdSwSkJhu3YERedjh7T93oUciJ4wusSoZjzLAXr5ytF9O0TPqNog4G1riPWN+A== X-Received: by 2002:a5d:6504:: with SMTP id x4mr887012wru.340.1588619034970; Mon, 04 May 2020 12:03:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w18sm19667056wrn.55.2020.05.04.12.03.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:54 -0700 (PDT) Message-Id: <06fcb49e90326414148822a7548f3f3898db60bf.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Jonathan Nieder via GitGitGadget" Date: Mon, 04 May 2020 19:03:41 +0000 Subject: [PATCH v11 05/12] reftable: file format documentation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Jonathan Nieder Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jonathan Nieder Shawn Pearce explains: Some repositories contain a lot of references (e.g. android at 866k, rails at 31k). The reftable format provides: - Near constant time lookup for any single reference, even when the repository is cold and not in process or kernel cache. - Near constant time verification a SHA-1 is referred to by at least one reference (for allow-tip-sha1-in-want). - Efficient lookup of an entire namespace, such as `refs/tags/`. - Support atomic push `O(size_of_update)` operations. - Combine reflog storage with ref storage. This file format spec was originally written in July, 2017 by Shawn Pearce. Some refinements since then were made by Shawn and by Han-Wen Nienhuys based on experiences implementing and experimenting with the format. (All of this was in the context of our work at Google and Google is happy to contribute the result to the Git project.) Imported from JGit[1]'s current version (c217d33ff, "Documentation/technical/reftable: improve repo layout", 2020-02-04) of Documentation/technical/reftable.md and converted to asciidoc by running pandoc -t asciidoc -f markdown reftable.md >reftable.txt using pandoc 2.2.1. The result required the following additional minor changes: - removed the [TOC] directive to add a table of contents, since asciidoc does not support it - replaced git-scm.com/docs links with linkgit: directives that link to other pages within Git's documentation [1] https://eclipse.googlesource.com/jgit/jgit Signed-off-by: Jonathan Nieder --- Documentation/Makefile | 1 + Documentation/technical/reftable.txt | 1067 ++++++++++++++++++++++++++ 2 files changed, 1068 insertions(+) create mode 100644 Documentation/technical/reftable.txt diff --git a/Documentation/Makefile b/Documentation/Makefile index 15d9d04f316..ecd0b340b1c 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -93,6 +93,7 @@ TECH_DOCS += technical/protocol-capabilities TECH_DOCS += technical/protocol-common TECH_DOCS += technical/protocol-v2 TECH_DOCS += technical/racy-git +TECH_DOCS += technical/reftable TECH_DOCS += technical/send-pack-pipeline TECH_DOCS += technical/shallow TECH_DOCS += technical/signature-format diff --git a/Documentation/technical/reftable.txt b/Documentation/technical/reftable.txt new file mode 100644 index 00000000000..9fa4657d9ff --- /dev/null +++ b/Documentation/technical/reftable.txt @@ -0,0 +1,1067 @@ +reftable +-------- + +Overview +~~~~~~~~ + +Problem statement +^^^^^^^^^^^^^^^^^ + +Some repositories contain a lot of references (e.g. android at 866k, +rails at 31k). The existing packed-refs format takes up a lot of space +(e.g. 62M), and does not scale with additional references. Lookup of a +single reference requires linearly scanning the file. + +Atomic pushes modifying multiple references require copying the entire +packed-refs file, which can be a considerable amount of data moved +(e.g. 62M in, 62M out) for even small transactions (2 refs modified). + +Repositories with many loose references occupy a large number of disk +blocks from the local file system, as each reference is its own file +storing 41 bytes (and another file for the corresponding reflog). This +negatively affects the number of inodes available when a large number of +repositories are stored on the same filesystem. Readers can be penalized +due to the larger number of syscalls required to traverse and read the +`$GIT_DIR/refs` directory. + +Objectives +^^^^^^^^^^ + +* Near constant time lookup for any single reference, even when the +repository is cold and not in process or kernel cache. +* Near constant time verification if a SHA-1 is referred to by at least +one reference (for allow-tip-sha1-in-want). +* Efficient lookup of an entire namespace, such as `refs/tags/`. +* Support atomic push with `O(size_of_update)` operations. +* Combine reflog storage with ref storage for small transactions. +* Separate reflog storage for base refs and historical logs. + +Description +^^^^^^^^^^^ + +A reftable file is a portable binary file format customized for +reference storage. References are sorted, enabling linear scans, binary +search lookup, and range scans. + +Storage in the file is organized into variable sized blocks. Prefix +compression is used within a single block to reduce disk space. Block +size and alignment is tunable by the writer. + +Performance +^^^^^^^^^^^ + +Space used, packed-refs vs. reftable: + +[cols=",>,>,>,>,>",options="header",] +|=============================================================== +|repository |packed-refs |reftable |% original |avg ref |avg obj +|android |62.2 M |36.1 M |58.0% |33 bytes |5 bytes +|rails |1.8 M |1.1 M |57.7% |29 bytes |4 bytes +|git |78.7 K |48.1 K |61.0% |50 bytes |4 bytes +|git (heads) |332 b |269 b |81.0% |33 bytes |0 bytes +|=============================================================== + +Scan (read 866k refs), by reference name lookup (single ref from 866k +refs), and by SHA-1 lookup (refs with that SHA-1, from 866k refs): + +[cols=",>,>,>,>",options="header",] +|========================================================= +|format |cache |scan |by name |by SHA-1 +|packed-refs |cold |402 ms |409,660.1 usec |412,535.8 usec +|packed-refs |hot | |6,844.6 usec |20,110.1 usec +|reftable |cold |112 ms |33.9 usec |323.2 usec +|reftable |hot | |20.2 usec |320.8 usec +|========================================================= + +Space used for 149,932 log entries for 43,061 refs, reflog vs. reftable: + +[cols=",>,>",options="header",] +|================================ +|format |size |avg entry +|$GIT_DIR/logs |173 M |1209 bytes +|reftable |5 M |37 bytes +|================================ + +Details +~~~~~~~ + +Peeling +^^^^^^^ + +References stored in a reftable are peeled, a record for an annotated +(or signed) tag records both the tag object, and the object it refers +to. + +Reference name encoding +^^^^^^^^^^^^^^^^^^^^^^^ + +Reference names are an uninterpreted sequence of bytes that must pass +linkgit:git-check-ref-format[1] as a valid reference name. + +Key unicity +^^^^^^^^^^^ + +Each entry must have a unique key; repeated keys are disallowed. + +Network byte order +^^^^^^^^^^^^^^^^^^ + +All multi-byte, fixed width fields are in network byte order. + +Ordering +^^^^^^^^ + +Blocks are lexicographically ordered by their first reference. + +Directory/file conflicts +^^^^^^^^^^^^^^^^^^^^^^^^ + +The reftable format accepts both `refs/heads/foo` and +`refs/heads/foo/bar` as distinct references. + +This property is useful for retaining log records in reftable, but may +confuse versions of Git using `$GIT_DIR/refs` directory tree to maintain +references. Users of reftable may choose to continue to reject `foo` and +`foo/bar` type conflicts to prevent problems for peers. + +File format +~~~~~~~~~~~ + +Structure +^^^^^^^^^ + +A reftable file has the following high-level structure: + +.... +first_block { + header + first_ref_block +} +ref_block* +ref_index* +obj_block* +obj_index* +log_block* +log_index* +footer +.... + +A log-only file omits the `ref_block`, `ref_index`, `obj_block` and +`obj_index` sections, containing only the file header and log block: + +.... +first_block { + header +} +log_block* +log_index* +footer +.... + +in a log-only file the first log block immediately follows the file +header, without padding to block alignment. + +Block size +^^^^^^^^^^ + +The file’s block size is arbitrarily determined by the writer, and does +not have to be a power of 2. The block size must be larger than the +longest reference name or log entry used in the repository, as +references cannot span blocks. + +Powers of two that are friendly to the virtual memory system or +filesystem (such as 4k or 8k) are recommended. Larger sizes (64k) can +yield better compression, with a possible increased cost incurred by +readers during access. + +The largest block size is `16777215` bytes (15.99 MiB). + +Block alignment +^^^^^^^^^^^^^^^ + +Writers may choose to align blocks at multiples of the block size by +including `padding` filled with NUL bytes at the end of a block to round +out to the chosen alignment. When alignment is used, writers must +specify the alignment with the file header’s `block_size` field. + +Block alignment is not required by the file format. Unaligned files must +set `block_size = 0` in the file header, and omit `padding`. Unaligned +files with more than one ref block must include the link:#Ref-index[ref +index] to support fast lookup. Readers must be able to read both aligned +and non-aligned files. + +Very small files (e.g. 1 only ref block) may omit `padding` and the ref +index to reduce total file size. + +Header +^^^^^^ + +A 24-byte header appears at the beginning of the file: + +.... +'REFT' +uint8( version_number = 1 ) +uint24( block_size ) +uint64( min_update_index ) +uint64( max_update_index ) +.... + +Aligned files must specify `block_size` to configure readers with the +expected block alignment. Unaligned files must set `block_size = 0`. + +The `min_update_index` and `max_update_index` describe bounds for the +`update_index` field of all log records in this file. When reftables are +used in a stack for link:#Update-transactions[transactions], these +fields can order the files such that the prior file’s +`max_update_index + 1` is the next file’s `min_update_index`. + +First ref block +^^^^^^^^^^^^^^^ + +The first ref block shares the same block as the file header, and is 24 +bytes smaller than all other blocks in the file. The first block +immediately begins after the file header, at position 24. + +If the first block is a log block (a log-only file), its block header +begins immediately at position 24. + +Ref block format +^^^^^^^^^^^^^^^^ + +A ref block is written as: + +.... +'r' +uint24( block_len ) +ref_record+ +uint24( restart_offset )+ +uint16( restart_count ) + +padding? +.... + +Blocks begin with `block_type = 'r'` and a 3-byte `block_len` which +encodes the number of bytes in the block up to, but not including the +optional `padding`. This is always less than or equal to the file’s +block size. In the first ref block, `block_len` includes 24 bytes for +the file header. + +The 2-byte `restart_count` stores the number of entries in the +`restart_offset` list, which must not be empty. Readers can use +`restart_count` to binary search between restarts before starting a +linear scan. + +Exactly `restart_count` 3-byte `restart_offset` values precedes the +`restart_count`. Offsets are relative to the start of the block and +refer to the first byte of any `ref_record` whose name has not been +prefix compressed. Entries in the `restart_offset` list must be sorted, +ascending. Readers can start linear scans from any of these records. + +A variable number of `ref_record` fill the middle of the block, +describing reference names and values. The format is described below. + +As the first ref block shares the first file block with the file header, +all `restart_offset` in the first block are relative to the start of the +file (position 0), and include the file header. This forces the first +`restart_offset` to be `28`. + +ref record +++++++++++ + +A `ref_record` describes a single reference, storing both the name and +its value(s). Records are formatted as: + +.... +varint( prefix_length ) +varint( (suffix_length << 3) | value_type ) +suffix +varint( update_index_delta ) +value? +.... + +The `prefix_length` field specifies how many leading bytes of the prior +reference record’s name should be copied to obtain this reference’s +name. This must be 0 for the first reference in any block, and also must +be 0 for any `ref_record` whose offset is listed in the `restart_offset` +table at the end of the block. + +Recovering a reference name from any `ref_record` is a simple concat: + +.... +this_name = prior_name[0..prefix_length] + suffix +.... + +The `suffix_length` value provides the number of bytes available in +`suffix` to copy from `suffix` to complete the reference name. + +The `update_index` that last modified the reference can be obtained by +adding `update_index_delta` to the `min_update_index` from the file +header: `min_update_index + update_index_delta`. + +The `value` follows. Its format is determined by `value_type`, one of +the following: + +* `0x0`: deletion; no value data (see transactions, below) +* `0x1`: one 20-byte object id; value of the ref +* `0x2`: two 20-byte object ids; value of the ref, peeled target +* `0x3`: symbolic reference: `varint( target_len ) target` + +Symbolic references use `0x3`, followed by the complete name of the +reference target. No compression is applied to the target name. + +Types `0x4..0x7` are reserved for future use. + +Ref index +^^^^^^^^^ + +The ref index stores the name of the last reference from every ref block +in the file, enabling reduced disk seeks for lookups. Any reference can +be found by searching the index, identifying the containing block, and +searching within that block. + +The index may be organized into a multi-level index, where the 1st level +index block points to additional ref index blocks (2nd level), which may +in turn point to either additional index blocks (e.g. 3rd level) or ref +blocks (leaf level). Disk reads required to access a ref go up with +higher index levels. Multi-level indexes may be required to ensure no +single index block exceeds the file format’s max block size of +`16777215` bytes (15.99 MiB). To acheive constant O(1) disk seeks for +lookups the index must be a single level, which is permitted to exceed +the file’s configured block size, but not the format’s max block size of +15.99 MiB. + +If present, the ref index block(s) appears after the last ref block. + +If there are at least 4 ref blocks, a ref index block should be written +to improve lookup times. Cold reads using the index require 2 disk reads +(read index, read block), and binary searching < 4 blocks also requires +<= 2 reads. Omitting the index block from smaller files saves space. + +If the file is unaligned and contains more than one ref block, the ref +index must be written. + +Index block format: + +.... +'i' +uint24( block_len ) +index_record+ +uint24( restart_offset )+ +uint16( restart_count ) + +padding? +.... + +The index blocks begin with `block_type = 'i'` and a 3-byte `block_len` +which encodes the number of bytes in the block, up to but not including +the optional `padding`. + +The `restart_offset` and `restart_count` fields are identical in format, +meaning and usage as in ref blocks. + +To reduce the number of reads required for random access in very large +files the index block may be larger than other blocks. However, readers +must hold the entire index in memory to benefit from this, so it’s a +time-space tradeoff in both file size and reader memory. + +Increasing the file’s block size decreases the index size. Alternatively +a multi-level index may be used, keeping index blocks within the file’s +block size, but increasing the number of blocks that need to be +accessed. + +index record +++++++++++++ + +An index record describes the last entry in another block. Index records +are written as: + +.... +varint( prefix_length ) +varint( (suffix_length << 3) | 0 ) +suffix +varint( block_position ) +.... + +Index records use prefix compression exactly like `ref_record`. + +Index records store `block_position` after the suffix, specifying the +absolute position in bytes (from the start of the file) of the block +that ends with this reference. Readers can seek to `block_position` to +begin reading the block header. + +Readers must examine the block header at `block_position` to determine +if the next block is another level index block, or the leaf-level ref +block. + +Reading the index ++++++++++++++++++ + +Readers loading the ref index must first read the footer (below) to +obtain `ref_index_position`. If not present, the position will be 0. The +`ref_index_position` is for the 1st level root of the ref index. + +Obj block format +^^^^^^^^^^^^^^^^ + +Object blocks are optional. Writers may choose to omit object blocks, +especially if readers will not use the SHA-1 to ref mapping. + +Object blocks use unique, abbreviated 2-20 byte SHA-1 keys, mapping to +ref blocks containing references pointing to that object directly, or as +the peeled value of an annotated tag. Like ref blocks, object blocks use +the file’s standard block size. The abbrevation length is available in +the footer as `obj_id_len`. + +To save space in small files, object blocks may be omitted if the ref +index is not present, as brute force search will only need to read a few +ref blocks. When missing, readers should brute force a linear search of +all references to lookup by SHA-1. + +An object block is written as: + +.... +'o' +uint24( block_len ) +obj_record+ +uint24( restart_offset )+ +uint16( restart_count ) + +padding? +.... + +Fields are identical to ref block. Binary search using the restart table +works the same as in reference blocks. + +Because object identifiers are abbreviated by writers to the shortest +unique abbreviation within the reftable, obj key lengths are variable +between 2 and 20 bytes. Readers must compare only for common prefix +match within an obj block or obj index. + +obj record +++++++++++ + +An `obj_record` describes a single object abbreviation, and the blocks +containing references using that unique abbreviation: + +.... +varint( prefix_length ) +varint( (suffix_length << 3) | cnt_3 ) +suffix +varint( cnt_large )? +varint( position_delta )* +.... + +Like in reference blocks, abbreviations are prefix compressed within an +obj block. On large reftables with many unique objects, higher block +sizes (64k), and higher restart interval (128), a `prefix_length` of 2 +or 3 and `suffix_length` of 3 may be common in obj records (unique +abbreviation of 5-6 raw bytes, 10-12 hex digits). + +Each record contains `position_count` number of positions for matching +ref blocks. For 1-7 positions the count is stored in `cnt_3`. When +`cnt_3 = 0` the actual count follows in a varint, `cnt_large`. + +The use of `cnt_3` bets most objects are pointed to by only a single +reference, some may be pointed to by a couple of references, and very +few (if any) are pointed to by more than 7 references. + +A special case exists when `cnt_3 = 0` and `cnt_large = 0`: there are no +`position_delta`, but at least one reference starts with this +abbreviation. A reader that needs exact reference names must scan all +references to find which specific references have the desired object. +Writers should use this format when the `position_delta` list would have +overflowed the file’s block size due to a high number of references +pointing to the same object. + +The first `position_delta` is the position from the start of the file. +Additional `position_delta` entries are sorted ascending and relative to +the prior entry, e.g. a reader would perform: + +.... +pos = position_delta[0] +prior = pos +for (j = 1; j < position_count; j++) { + pos = prior + position_delta[j] + prior = pos +} +.... + +With a position in hand, a reader must linearly scan the ref block, +starting from the first `ref_record`, testing each reference’s SHA-1s +(for `value_type = 0x1` or `0x2`) for full equality. Faster searching by +SHA-1 within a single ref block is not supported by the reftable format. +Smaller block sizes reduce the number of candidates this step must +consider. + +Obj index +^^^^^^^^^ + +The obj index stores the abbreviation from the last entry for every obj +block in the file, enabling reduced disk seeks for all lookups. It is +formatted exactly the same as the ref index, but refers to obj blocks. + +The obj index should be present if obj blocks are present, as obj blocks +should only be written in larger files. + +Readers loading the obj index must first read the footer (below) to +obtain `obj_index_position`. If not present, the position will be 0. + +Log block format +^^^^^^^^^^^^^^^^ + +Unlike ref and obj blocks, log blocks are always unaligned. + +Log blocks are variable in size, and do not match the `block_size` +specified in the file header or footer. Writers should choose an +appropriate buffer size to prepare a log block for deflation, such as +`2 * block_size`. + +A log block is written as: + +.... +'g' +uint24( block_len ) +zlib_deflate { + log_record+ + uint24( restart_offset )+ + uint16( restart_count ) +} +.... + +Log blocks look similar to ref blocks, except `block_type = 'g'`. + +The 4-byte block header is followed by the deflated block contents using +zlib deflate. The `block_len` in the header is the inflated size +(including 4-byte block header), and should be used by readers to +preallocate the inflation output buffer. A log block’s `block_len` may +exceed the file’s block size. + +Offsets within the log block (e.g. `restart_offset`) still include the +4-byte header. Readers may prefer prefixing the inflation output buffer +with the 4-byte header. + +Within the deflate container, a variable number of `log_record` describe +reference changes. The log record format is described below. See ref +block format (above) for a description of `restart_offset` and +`restart_count`. + +Because log blocks have no alignment or padding between blocks, readers +must keep track of the bytes consumed by the inflater to know where the +next log block begins. + +log record +++++++++++ + +Log record keys are structured as: + +.... +ref_name '\0' reverse_int64( update_index ) +.... + +where `update_index` is the unique transaction identifier. The +`update_index` field must be unique within the scope of a `ref_name`. +See the update transactions section below for further details. + +The `reverse_int64` function inverses the value so lexographical +ordering the network byte order encoding sorts the more recent records +with higher `update_index` values first: + +.... +reverse_int64(int64 t) { + return 0xffffffffffffffff - t; +} +.... + +Log records have a similar starting structure to ref and index records, +utilizing the same prefix compression scheme applied to the log record +key described above. + +.... + varint( prefix_length ) + varint( (suffix_length << 3) | log_type ) + suffix + log_data { + old_id + new_id + varint( name_length ) name + varint( email_length ) email + varint( time_seconds ) + sint16( tz_offset ) + varint( message_length ) message + }? +.... + +Log record entries use `log_type` to indicate what follows: + +* `0x0`: deletion; no log data. +* `0x1`: standard git reflog data using `log_data` above. + +The `log_type = 0x0` is mostly useful for `git stash drop`, removing an +entry from the reflog of `refs/stash` in a transaction file (below), +without needing to rewrite larger files. Readers reading a stack of +reflogs must treat this as a deletion. + +For `log_type = 0x1`, the `log_data` section follows +linkgit:git-update-ref[1] logging and includes: + +* two 20-byte SHA-1s (old id, new id) +* varint string of committer’s name +* varint string of committer’s email +* varint time in seconds since epoch (Jan 1, 1970) +* 2-byte timezone offset in minutes (signed) +* varint string of message + +`tz_offset` is the absolute number of minutes from GMT the committer was +at the time of the update. For example `GMT-0800` is encoded in reftable +as `sint16(-480)` and `GMT+0230` is `sint16(150)`. + +The committer email does not contain `<` or `>`, it’s the value normally +found between the `<>` in a git commit object header. + +The `message_length` may be 0, in which case there was no message +supplied for the update. + +Contrary to traditional reflog (which is a file), renames are encoded as +a combination of ref deletion and ref creation. + +Reading the log ++++++++++++++++ + +Readers accessing the log must first read the footer (below) to +determine the `log_position`. The first block of the log begins at +`log_position` bytes since the start of the file. The `log_position` is +not block aligned. + +Importing logs +++++++++++++++ + +When importing from `$GIT_DIR/logs` writers should globally order all +log records roughly by timestamp while preserving file order, and assign +unique, increasing `update_index` values for each log line. Newer log +records get higher `update_index` values. + +Although an import may write only a single reftable file, the reftable +file must span many unique `update_index`, as each log line requires its +own `update_index` to preserve semantics. + +Log index +^^^^^^^^^ + +The log index stores the log key +(`refname \0 reverse_int64(update_index)`) for the last log record of +every log block in the file, supporting bounded-time lookup. + +A log index block must be written if 2 or more log blocks are written to +the file. If present, the log index appears after the last log block. +There is no padding used to align the log index to block alignment. + +Log index format is identical to ref index, except the keys are 9 bytes +longer to include `'\0'` and the 8-byte `reverse_int64(update_index)`. +Records use `block_position` to refer to the start of a log block. + +Reading the index ++++++++++++++++++ + +Readers loading the log index must first read the footer (below) to +obtain `log_index_position`. If not present, the position will be 0. + +Footer +^^^^^^ + +After the last block of the file, a file footer is written. It begins +like the file header, but is extended with additional data. + +A 68-byte footer appears at the end: + +.... + 'REFT' + uint8( version_number = 1 ) + uint24( block_size ) + uint64( min_update_index ) + uint64( max_update_index ) + + uint64( ref_index_position ) + uint64( (obj_position << 5) | obj_id_len ) + uint64( obj_index_position ) + + uint64( log_position ) + uint64( log_index_position ) + + uint32( CRC-32 of above ) +.... + +If a section is missing (e.g. ref index) the corresponding position +field (e.g. `ref_index_position`) will be 0. + +* `obj_position`: byte position for the first obj block. +* `obj_id_len`: number of bytes used to abbreviate object identifiers in +obj blocks. +* `log_position`: byte position for the first log block. +* `ref_index_position`: byte position for the start of the ref index. +* `obj_index_position`: byte position for the start of the obj index. +* `log_index_position`: byte position for the start of the log index. + +Reading the footer +++++++++++++++++++ + +Readers must seek to `file_length - 68` to access the footer. A trusted +external source (such as `stat(2)`) is necessary to obtain +`file_length`. When reading the footer, readers must verify: + +* 4-byte magic is correct +* 1-byte version number is recognized +* 4-byte CRC-32 matches the other 64 bytes (including magic, and +version) + +Once verified, the other fields of the footer can be accessed. + +Varint encoding +^^^^^^^^^^^^^^^ + +Varint encoding is identical to the ofs-delta encoding method used +within pack files. + +Decoder works such as: + +.... +val = buf[ptr] & 0x7f +while (buf[ptr] & 0x80) { + ptr++ + val = ((val + 1) << 7) | (buf[ptr] & 0x7f) +} +.... + +Binary search +^^^^^^^^^^^^^ + +Binary search within a block is supported by the `restart_offset` fields +at the end of the block. Readers can binary search through the restart +table to locate between which two restart points the sought reference or +key should appear. + +Each record identified by a `restart_offset` stores the complete key in +the `suffix` field of the record, making the compare operation during +binary search straightforward. + +Once a restart point lexicographically before the sought reference has +been identified, readers can linearly scan through the following record +entries to locate the sought record, terminating if the current record +sorts after (and therefore the sought key is not present). + +Restart point selection ++++++++++++++++++++++++ + +Writers determine the restart points at file creation. The process is +arbitrary, but every 16 or 64 records is recommended. Every 16 may be +more suitable for smaller block sizes (4k or 8k), every 64 for larger +block sizes (64k). + +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. + +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalities for readers +walking through more records after the binary search step. + +A maximum of `65535` restart points per block is supported. + +Considerations +~~~~~~~~~~~~~~ + +Lightweight refs dominate +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The reftable format assumes the vast majority of references are single +SHA-1 valued with common prefixes, such as Gerrit Code Review’s +`refs/changes/` namespace, GitHub’s `refs/pulls/` namespace, or many +lightweight tags in the `refs/tags/` namespace. + +Annotated tags storing the peeled object cost an additional 20 bytes per +reference. + +Low overhead +^^^^^^^^^^^^ + +A reftable with very few references (e.g. git.git with 5 heads) is 269 +bytes for reftable, vs. 332 bytes for packed-refs. This supports +reftable scaling down for transaction logs (below). + +Block size +^^^^^^^^^^ + +For a Gerrit Code Review type repository with many change refs, larger +block sizes (64 KiB) and less frequent restart points (every 64) yield +better compression due to more references within the block compressing +against the prior reference. + +Larger block sizes reduce the index size, as the reftable will require +fewer blocks to store the same number of references. + +Minimal disk seeks +^^^^^^^^^^^^^^^^^^ + +Assuming the index block has been loaded into memory, binary searching +for any single reference requires exactly 1 disk seek to load the +containing block. + +Scans and lookups dominate +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Scanning all references and lookup by name (or namespace such as +`refs/heads/`) are the most common activities performed on repositories. +SHA-1s are stored directly with references to optimize this use case. + +Logs are infrequently read +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Logs are infrequently accessed, but can be large. Deflating log blocks +saves disk space, with some increased penalty at read time. + +Logs are stored in an isolated section from refs, reducing the burden on +reference readers that want to ignore logs. Further, historical logs can +be isolated into log-only files. + +Logs are read backwards +^^^^^^^^^^^^^^^^^^^^^^^ + +Logs are frequently accessed backwards (most recent N records for master +to answer `master@{4}`), so log records are grouped by reference, and +sorted descending by update index. + +Repository format +~~~~~~~~~~~~~~~~~ + +Version 1 +^^^^^^^^^ + +A repository must set its `$GIT_DIR/config` to configure reftable: + +.... +[core] + repositoryformatversion = 1 +[extensions] + refStorage = reftable +.... + +Layout +^^^^^^ + +A collection of reftable files are stored in the `$GIT_DIR/reftable/` +directory: + +.... +00000001-00000001.log +00000002-00000002.ref +00000003-00000003.ref +.... + +where reftable files are named by a unique name such as produced by the +function `${min_update_index}-${max_update_index}.ref`. + +Log-only files use the `.log` extension, while ref-only and mixed ref +and log files use `.ref`. extension. + +The stack ordering file is `$GIT_DIR/reftable/tables.list` and lists the +current files, one per line, in order, from oldest (base) to newest +(most recent): + +.... +$ cat .git/reftable/tables.list +00000001-00000001.log +00000002-00000002.ref +00000003-00000003.ref +.... + +Readers must read `$GIT_DIR/reftable/tables.list` to determine which +files are relevant right now, and search through the stack in reverse +order (last reftable is examined first). + +Reftable files not listed in `tables.list` may be new (and about to be +added to the stack by the active writer), or ancient and ready to be +pruned. + +Backward compatibility +^^^^^^^^^^^^^^^^^^^^^^ + +Older clients should continue to recognize the directory as a git +repository so they don’t look for an enclosing repository in parent +directories. To this end, a reftable-enabled repository must contain the +following dummy files + +* `.git/HEAD`, a regular file containing `ref: refs/heads/.invalid`. +* `.git/refs/`, a directory +* `.git/refs/heads`, a regular file + +Readers +^^^^^^^ + +Readers can obtain a consistent snapshot of the reference space by +following: + +1. Open and read the `tables.list` file. +2. Open each of the reftable files that it mentions. +3. If any of the files is missing, goto 1. +4. Read from the now-open files as long as necessary. + +Update transactions +^^^^^^^^^^^^^^^^^^^ + +Although reftables are immutable, mutations are supported by writing a +new reftable and atomically appending it to the stack: + +1. Acquire `tables.list.lock`. +2. Read `tables.list` to determine current reftables. +3. Select `update_index` to be most recent file’s +`max_update_index + 1`. +4. Prepare temp reftable `tmp_XXXXXX`, including log entries. +5. Rename `tmp_XXXXXX` to `${update_index}-${update_index}.ref`. +6. Copy `tables.list` to `tables.list.lock`, appending file from (5). +7. Rename `tables.list.lock` to `tables.list`. + +During step 4 the new file’s `min_update_index` and `max_update_index` +are both set to the `update_index` selected by step 3. All log records +for the transaction use the same `update_index` in their keys. This +enables later correlation of which references were updated by the same +transaction. + +Because a single `tables.list.lock` file is used to manage locking, the +repository is single-threaded for writers. Writers may have to busy-spin +(with backoff) around creating `tables.list.lock`, for up to an +acceptable wait period, aborting if the repository is too busy to +mutate. Application servers wrapped around repositories (e.g. Gerrit +Code Review) can layer their own lock/wait queue to improve fairness to +writers. + +Reference deletions +^^^^^^^^^^^^^^^^^^^ + +Deletion of any reference can be explicitly stored by setting the `type` +to `0x0` and omitting the `value` field of the `ref_record`. This serves +as a tombstone, overriding any assertions about the existence of the +reference from earlier files in the stack. + +Compaction +^^^^^^^^^^ + +A partial stack of reftables can be compacted by merging references +using a straightforward merge join across reftables, selecting the most +recent value for output, and omitting deleted references that do not +appear in remaining, lower reftables. + +A compacted reftable should set its `min_update_index` to the smallest +of the input files’ `min_update_index`, and its `max_update_index` +likewise to the largest input `max_update_index`. + +For sake of illustration, assume the stack currently consists of +reftable files (from oldest to newest): A, B, C, and D. The compactor is +going to compact B and C, leaving A and D alone. + +1. Obtain lock `tables.list.lock` and read the `tables.list` file. +2. Obtain locks `B.lock` and `C.lock`. Ownership of these locks +prevents other processes from trying to compact these files. +3. Release `tables.list.lock`. +4. Compact `B` and `C` into a temp file +`${min_update_index}-${max_update_index}_XXXXXX`. +5. Reacquire lock `tables.list.lock`. +6. Verify that `B` and `C` are still in the stack, in that order. This +should always be the case, assuming that other processes are adhering to +the locking protocol. +7. Rename `${min_update_index}-${max_update_index}_XXXXXX` to +`${min_update_index}-${max_update_index}.ref`. +8. Write the new stack to `tables.list.lock`, replacing `B` and `C` +with the file from (4). +9. Rename `tables.list.lock` to `tables.list`. +10. Delete `B` and `C`, perhaps after a short sleep to avoid forcing +readers to backtrack. + +This strategy permits compactions to proceed independently of updates. + +Each reftable (compacted or not) is uniquely identified by its name, so +open reftables can be cached by their name. + +Alternatives considered +~~~~~~~~~~~~~~~~~~~~~~~ + +bzip packed-refs +^^^^^^^^^^^^^^^^ + +`bzip2` can significantly shrink a large packed-refs file (e.g. 62 MiB +compresses to 23 MiB, 37%). However the bzip format does not support +random access to a single reference. Readers must inflate and discard +while performing a linear scan. + +Breaking packed-refs into chunks (individually compressing each chunk) +would reduce the amount of data a reader must inflate, but still leaves +the problem of indexing chunks to support readers efficiently locating +the correct chunk. + +Given the compression achieved by reftable’s encoding, it does not seem +necessary to add the complexity of bzip/gzip/zlib. + +Michael Haggerty’s alternate format +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Michael Haggerty proposed +https://public-inbox.org/git/CAMy9T_HCnyc1g8XWOOWhe7nN0aEFyyBskV2aOMb_fe+wGvEJ7A@mail.gmail.com/[an +alternate] format to reftable on the Git mailing list. This format uses +smaller chunks, without the restart table, and avoids block alignment +with padding. Reflog entries immediately follow each ref, and are thus +interleaved between refs. + +Performance testing indicates reftable is faster for lookups (51% +faster, 11.2 usec vs. 5.4 usec), although reftable produces a slightly +larger file (+ ~3.2%, 28.3M vs 29.2M): + +[cols=">,>,>,>",options="header",] +|===================================== +|format |size |seek cold |seek hot +|mh-alt |28.3 M |23.4 usec |11.2 usec +|reftable |29.2 M |19.9 usec |5.4 usec +|===================================== + +JGit Ketch RefTree +^^^^^^^^^^^^^^^^^^ + +https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03073.html[JGit Ketch] +proposed +https://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/[RefTree], +an encoding of references inside Git tree objects stored as part of the +repository’s object database. + +The RefTree format adds additional load on the object database storage +layer (more loose objects, more objects in packs), and relies heavily on +the packer’s delta compression to save space. Namespaces which are flat +(e.g. thousands of tags in refs/tags) initially create very large loose +objects, and so RefTree does not address the problem of copying many +references to modify a handful. + +Flat namespaces are not efficiently searchable in RefTree, as tree +objects in canonical formatting cannot be binary searched. This fails +the need to handle a large number of references in a single namespace, +such as GitHub’s `refs/pulls`, or a project with many tags. + +LMDB +^^^^ + +David Turner proposed +https://public-inbox.org/git/1455772670-21142-26-git-send-email-dturner@twopensource.com/[using +LMDB], as LMDB is lightweight (64k of runtime code) and GPL-compatible +license. + +A downside of LMDB is its reliance on a single C implementation. This +makes embedding inside JGit (a popular reimplemenation of Git) +difficult, and hoisting onto virtual storage (for JGit DFS) virtually +impossible. + +A common format that can be supported by all major Git implementations +(git-core, JGit, libgit2) is strongly preferred. + +Future +~~~~~~ + +Longer hashes +^^^^^^^^^^^^^ + +Version will bump (e.g. 2) to indicate `value` uses a different object +id length other than 20. The length could be stored in an expanded file +header, or hardcoded as part of the version. From patchwork Mon May 4 19:03:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527437 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C090B92A for ; Mon, 4 May 2020 19:04:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A855020721 for ; Mon, 4 May 2020 19:04:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qftBskmp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727827AbgEDTD7 (ORCPT ); Mon, 4 May 2020 15:03:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726796AbgEDTD5 (ORCPT ); Mon, 4 May 2020 15:03:57 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15A98C061A0E for ; Mon, 4 May 2020 12:03:57 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id k1so351342wro.12 for ; Mon, 04 May 2020 12:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bMqfVV0Jthx0FbzkxthTP1FYgECiPP01IK56yDoGvRw=; b=qftBskmpQdJ2dtpmo+99Shib0FBG0ArLS2v6EwD+7YaBYEeLT6BczicpQl6cee2883 dEK5A3kQu0w9Zkm97tbr+tGtyGiEudpQ2v7AWtMIDYjNsUdGuCEOLVPU0ils9+zZdC/J Ysc3mShw1kjt0KclrDsmmxBPeO2KZXPJlYJIixvTR0aBF686rIftCXksMoQ8XPXxwH76 cmgd84mNflZgGXgoDUo+VYfLFK13R562aYxCf1e6zhILsb4TBCODr6Alt0ZouazFTwtW H92Lsjbx41lHqalHXmcRDcsVefesBV9MS23K6htPVD67hp9AaGS0n0ql7udm2Jx7VnwM 0fTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bMqfVV0Jthx0FbzkxthTP1FYgECiPP01IK56yDoGvRw=; b=WGwnlqr4wUabRc3xozCioGwXVRS+ZVTpdb6kf2nuCF7JA7RSlNUgnPyWpCcE6fbyt5 OAO32ZQynGRsdjj7U0yPIgtm1zQ7t8pALrGLik2ZmMBw2b57EPfl0Jz62HSJz3Jwb760 hhfpR+XvRfSLOU8mVyyok1Syg5p3ldDiZ1irkTluS3dlPGJUcZVDtPCXDtaiRM1ZYWFp 6JF6x1CH4IAbvxGC0rcDOVTFUXaKfV2TwWt7RrvrY/nzbvn9q6J0teHdupYref9EzCxu cE6iDRk9oqE4jmCP+wK01gbWfRdY3J2voXMEMlARlBNVehjLpSw9j7Qw/Mfx/eIlEmeu txgw== X-Gm-Message-State: AGi0PuYOhavEzDzbsNAjodIpYefZrw0/JGJQyFzk1Vx4AyE6DlZAz45p YFtPeNVfGiK9uDNg9ykP5Fny9PCv X-Google-Smtp-Source: APiQypI/L49P/15zcBQNfoyRtY+symao+5Fn4+NyqkbZbL1+5jbn5i0XQnIjtpw867y7q9Pru4WnCA== X-Received: by 2002:adf:f8c1:: with SMTP id f1mr543940wrq.171.1588619035702; Mon, 04 May 2020 12:03:55 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l19sm606604wmj.14.2020.05.04.12.03.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:55 -0700 (PDT) Message-Id: <093fa74a3d0e7721093cceb338e8efc9c0c95b1c.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:42 +0000 Subject: [PATCH v11 06/12] reftable: define version 2 of the spec to accomodate SHA256 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys Signed-off-by: Han-Wen Nienhuys --- Documentation/technical/reftable.txt | 50 ++++++++++++++++------------ 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/Documentation/technical/reftable.txt b/Documentation/technical/reftable.txt index 9fa4657d9ff..ee3f36ea851 100644 --- a/Documentation/technical/reftable.txt +++ b/Documentation/technical/reftable.txt @@ -193,8 +193,8 @@ and non-aligned files. Very small files (e.g. 1 only ref block) may omit `padding` and the ref index to reduce total file size. -Header -^^^^^^ +Header (version 1) +^^^^^^^^^^^^^^^^^^ A 24-byte header appears at the beginning of the file: @@ -215,6 +215,24 @@ used in a stack for link:#Update-transactions[transactions], these fields can order the files such that the prior file’s `max_update_index + 1` is the next file’s `min_update_index`. +Header (version 2) +^^^^^^^^^^^^^^^^^^ + +A 28-byte header appears at the beginning of the file: + +.... +'REFT' +uint8( version_number = 1 ) +uint24( block_size ) +uint64( min_update_index ) +uint64( max_update_index ) +uint32( hash_id ) +.... + +The header is identical to `version_number=1`, with the 4-byte hash ID +("sha1" for SHA1 and "s256" for SHA-256) append to the header. + + First ref block ^^^^^^^^^^^^^^^ @@ -671,14 +689,8 @@ Footer After the last block of the file, a file footer is written. It begins like the file header, but is extended with additional data. -A 68-byte footer appears at the end: - .... - 'REFT' - uint8( version_number = 1 ) - uint24( block_size ) - uint64( min_update_index ) - uint64( max_update_index ) + HEADER uint64( ref_index_position ) uint64( (obj_position << 5) | obj_id_len ) @@ -701,12 +713,16 @@ obj blocks. * `obj_index_position`: byte position for the start of the obj index. * `log_index_position`: byte position for the start of the log index. +The size of the footer is 68 bytes for version 1, and 72 bytes for +version 2. + Reading the footer ++++++++++++++++++ -Readers must seek to `file_length - 68` to access the footer. A trusted -external source (such as `stat(2)`) is necessary to obtain -`file_length`. When reading the footer, readers must verify: +Readers must first read the file start to determine the version +number. Then they seek to `file_length - FOOTER_LENGTH` to access the +footer. A trusted external source (such as `stat(2)`) is necessary to +obtain `file_length`. When reading the footer, readers must verify: * 4-byte magic is correct * 1-byte version number is recognized @@ -1055,13 +1071,3 @@ impossible. A common format that can be supported by all major Git implementations (git-core, JGit, libgit2) is strongly preferred. - -Future -~~~~~~ - -Longer hashes -^^^^^^^^^^^^^ - -Version will bump (e.g. 2) to indicate `value` uses a different object -id length other than 20. The length could be stored in an expanded file -header, or hardcoded as part of the version. From patchwork Mon May 4 19:03:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527441 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 788CC81 for ; Mon, 4 May 2020 19:04:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 608F32073B for ; Mon, 4 May 2020 19:04:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f4zpa9Q2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726796AbgEDTEC (ORCPT ); Mon, 4 May 2020 15:04:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727821AbgEDTD6 (ORCPT ); Mon, 4 May 2020 15:03:58 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9F84C061A0E for ; Mon, 4 May 2020 12:03:57 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id v4so704993wme.1 for ; Mon, 04 May 2020 12:03:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=gP2mKurovl5RI1/nfB8JPBDqHhvSxbSjyPMBXIBnAvA=; b=f4zpa9Q25hVtNIT/4CunjPmScLCjsVgrdNFNFktyT4HsaSKP1WHj//q5FqOevnfqUH FmEjaA5UGWqLj+UdJLLyV0BQrIQswIMkNLPoTWuTummoTuxbSQBTwbHvDuhw2sohfYPM xhaHsePzvdBunz7WMPic/3dhlVvEOeZ57T0t2g31nZWXxJYWEGsFU0oddo0n54gwDXtO ewIJzn1Cv8zshTU2nj5GaVFv5ID4u29xgatZBK1P83bgAMzn6iNbK9yFyX6Y3U5EsyXh d18IdLggUUxScI5BD2KSLHoJ0+f6Lv4B5d3ly6YNo+VBa+M0rKFJIO/iyLwLpfz4C97R 62Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=gP2mKurovl5RI1/nfB8JPBDqHhvSxbSjyPMBXIBnAvA=; b=iDhPnkGwH5hyG+1EERWNnPob9e8pw5BJrwitEzKckaouGWSdxwL516HuMtirVCXlED pRtXVHvQunE7i/8HSkHHNutU56U1NYL2fQJqtS6eHQuUKI4WVElK7SDKnqZNTV+CRqMc sPO0MZgVDctQCHQ73B1+x6wLcCjwP5t6omIMKPtNLEkkLecPhUBaxwWqCsFYNFp3M/NF vs+BqSledD2YU12w0vj4BlP0xbdbOuhOF2ZZ46lfZG8LBPyRnFHtXLSRmKYLvkDwfl8x srCNkmMkgescZkRSWo5qcK8QwppSqFFv2GHgjsDOeecfEw5+W2qw5Bgukozri/SKKqo6 Fhew== X-Gm-Message-State: AGi0PubqQjMsV7e2mW1gHGyLjxzDp8jkjZkcAw+URKlGR8Zg1DmrQqcD bdWskqH4yrX2FDEEgxzNxjZyLC9P X-Google-Smtp-Source: APiQypLtQRuMxVyI31bSxDg69fxRvEaYm5U86Xw+5H16agkqjpy0mwTUiOxcEK1g/Xto8GK/wbE7XQ== X-Received: by 2002:a05:600c:2a52:: with SMTP id x18mr15655170wme.37.1588619036559; Mon, 04 May 2020 12:03:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 88sm2507980wrq.77.2020.05.04.12.03.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:56 -0700 (PDT) Message-Id: <6d9031372ce23a3ab25e504e4a0294b68844e27a.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:43 +0000 Subject: [PATCH v11 07/12] reftable: clarify how empty tables should be written Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys The format allows for some ambiguity, as a lone footer also starts with a valid file header. However, the current JGit code will barf on this. This commit codifies this behavior into the standard. Signed-off-by: Han-Wen Nienhuys --- Documentation/technical/reftable.txt | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Documentation/technical/reftable.txt b/Documentation/technical/reftable.txt index ee3f36ea851..df5cb5f11b8 100644 --- a/Documentation/technical/reftable.txt +++ b/Documentation/technical/reftable.txt @@ -731,6 +731,13 @@ version) Once verified, the other fields of the footer can be accessed. +Empty tables +++++++++++++ + +A reftable may be empty. In this case, the file starts with a header +and is immediately followed by a footer. + + Varint encoding ^^^^^^^^^^^^^^^ From patchwork Mon May 4 19:03:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527455 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C24E81 for ; Mon, 4 May 2020 19:04:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC5F7206B8 for ; Mon, 4 May 2020 19:04:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ce7TfNY1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727866AbgEDTEX (ORCPT ); Mon, 4 May 2020 15:04:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727853AbgEDTEE (ORCPT ); Mon, 4 May 2020 15:04:04 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 875C2C0610D5 for ; Mon, 4 May 2020 12:04:02 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id y24so709646wma.4 for ; Mon, 04 May 2020 12:04:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=R0j9LRb8DbGCbw3BS/QOwrdrZM+e3Q0nTvWSLTJHVBc=; b=Ce7TfNY1AlmO8p6AS9X+RI/cM7buiMtzeK4/0mr/8A3GgEo/hRDWuMTC/sNttOhuk1 iHkr/Tlb8dOSPBuc6pHfzcGBRgTMjeuYgEmU0oOD7sFYJUN2k4B0NJ03v4rcDK4X+u29 +JRYTcsLz+6oERApj7BdZkXcjPuL+MItrU/Y+aTD6rk532YFSdTayiujja0OB1eIjEK+ TiWBdQxH6YspivQIqcPCw5yf+po0vV8oRcNVWdyDaSINrON9o9+qppkNKslsfOkKOaJH j6Yei/GQOVpxShSGl3w0rrcJx01OrsVe6rh+1Wd4CRGpTN8bTfWNz9BVB5YK1wwTPniq XnyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=R0j9LRb8DbGCbw3BS/QOwrdrZM+e3Q0nTvWSLTJHVBc=; b=i7AbLcybo2Vz4e08gcOpfnGBtX2ksKTF6/xxwE8m2HPpD8YjH/qrYpnQx7CdPBc68t E1xvV3eCjH0I7vgwqlGiAy2dX6/6U3rlMdggL+Cr8p9JvMm/4KkRe3xqYnn+Q0zd04gR Dem7ubYkA8Kn/7uVFWwsC9WlD568xiOd85bFYYBnJ95oiLpaav7leD4+MtuRxIxh51dW qtpEW17jaC/xj2YDAYxxP8tRTgVv84p7BQKn15thqOeN/Qe1bMq3gRK22dFNAaXFh5pa TEiH0+XSHUWH95RJnyw+LZHXsZUT+809JH5pgSbuYW3LajrWC46W1i1HaKEhd3qXJltq IC6g== X-Gm-Message-State: AGi0PuY9nY5QGK1enIWPL474v97JLckfPVtFGmWlzBJ2tNWAsjp6SWmQ fsS90a2OWSHKafL/wHNtSC+riiUY X-Google-Smtp-Source: APiQypKMHdPFyL0X7mTvc91qDvm6+djOXxrE56oxqI371FYFNWIjlLDthQrhPS8cClc+U2FHqW4jBQ== X-Received: by 2002:a7b:cf25:: with SMTP id m5mr17459926wmg.65.1588619037738; Mon, 04 May 2020 12:03:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b82sm672971wmh.1.2020.05.04.12.03.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:57 -0700 (PDT) Message-Id: <6ee6c44752c66c3a469f80270bbbbf0de33bf24a.1588619028.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:44 +0000 Subject: [PATCH v11 08/12] Add reftable library Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys Reftable is a format for storing the ref database. Its rationale and specification is in the preceding commit. This imports the upstream library as one big commit. For understanding the code, it is suggested to read the code in the following order: * The specification under Documentation/technical/reftable.txt * reftable.h - the public API * record.{c,h} - reading and writing records * block.{c,h} - reading and writing blocks. * writer.{c,h} - writing a complete reftable file. * merged.{c,h} and pq.{c,h} - reading a stack of reftables * stack.{c,h} - writing and compacting stacks of reftable on the filesystem. Signed-off-by: Han-Wen Nienhuys --- reftable/LICENSE | 31 + reftable/README.md | 11 + reftable/VERSION | 5 + reftable/basics.c | 215 +++++++ reftable/basics.h | 53 ++ reftable/block.c | 436 ++++++++++++++ reftable/block.h | 126 ++++ reftable/blocksource.h | 22 + reftable/constants.h | 21 + reftable/file.c | 99 ++++ reftable/iter.c | 240 ++++++++ reftable/iter.h | 63 ++ reftable/merged.c | 327 +++++++++++ reftable/merged.h | 38 ++ reftable/pq.c | 115 ++++ reftable/pq.h | 34 ++ reftable/reader.c | 758 +++++++++++++++++++++++++ reftable/reader.h | 68 +++ reftable/record.c | 1131 ++++++++++++++++++++++++++++++++++++ reftable/record.h | 121 ++++ reftable/refname.c | 215 +++++++ reftable/refname.h | 39 ++ reftable/reftable.c | 91 +++ reftable/reftable.h | 564 ++++++++++++++++++ reftable/slice.c | 225 ++++++++ reftable/slice.h | 76 +++ reftable/stack.c | 1229 ++++++++++++++++++++++++++++++++++++++++ reftable/stack.h | 45 ++ reftable/system.h | 54 ++ reftable/tree.c | 67 +++ reftable/tree.h | 34 ++ reftable/update.sh | 24 + reftable/writer.c | 661 +++++++++++++++++++++ reftable/writer.h | 60 ++ reftable/zlib-compat.c | 92 +++ 35 files changed, 7390 insertions(+) create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/constants.h create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/refname.c create mode 100644 reftable/refname.h create mode 100644 reftable/reftable.c create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100755 reftable/update.sh create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c diff --git a/reftable/LICENSE b/reftable/LICENSE new file mode 100644 index 00000000000..402e0f9356b --- /dev/null +++ b/reftable/LICENSE @@ -0,0 +1,31 @@ +BSD License + +Copyright (c) 2020, Google LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of Google LLC nor the names of its contributors may +be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/reftable/README.md b/reftable/README.md new file mode 100644 index 00000000000..21500563fd4 --- /dev/null +++ b/reftable/README.md @@ -0,0 +1,11 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + sh reftable/update.sh + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable. diff --git a/reftable/VERSION b/reftable/VERSION new file mode 100644 index 00000000000..b947a05c089 --- /dev/null +++ b/reftable/VERSION @@ -0,0 +1,5 @@ +commit 3a486f79abcd17e88e3bca62f43188a7c8b80ff3 +Author: Han-Wen Nienhuys +Date: Mon May 4 19:20:21 2020 +0200 + + C: include "cache.h" in git-core sleep_millisec() diff --git a/reftable/basics.c b/reftable/basics.c new file mode 100644 index 00000000000..14c4dfaf5a6 --- /dev/null +++ b/reftable/basics.c @@ -0,0 +1,215 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "basics.h" + +#include "system.h" + +void put_be24(byte *out, uint32_t i) +{ + out[0] = (byte)((i >> 16) & 0xff); + out[1] = (byte)((i >> 8) & 0xff); + out[2] = (byte)(i & 0xff); +} + +uint32_t get_be24(byte *in) +{ + return (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 | + (uint32_t)(in[2]); +} + +void put_be16(uint8_t *out, uint16_t i) +{ + out[0] = (uint8_t)((i >> 8) & 0xff); + out[1] = (uint8_t)(i & 0xff); +} + +int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args) +{ + size_t lo = 0; + size_t hi = sz; + + /* invariant: (hi == sz) || f(hi) == true + (lo == 0 && f(0) == true) || fi(lo) == false + */ + while (hi - lo > 1) { + size_t mid = lo + (hi - lo) / 2; + + int val = f(mid, args); + if (val) { + hi = mid; + } else { + lo = mid; + } + } + + if (lo == 0) { + if (f(0, args)) { + return 0; + } else { + return 1; + } + } + + return hi; +} + +void free_names(char **a) +{ + char **p = a; + if (p == NULL) { + return; + } + while (*p) { + reftable_free(*p); + p++; + } + reftable_free(a); +} + +int names_length(char **names) +{ + int len = 0; + char **p = names; + while (*p) { + p++; + len++; + } + return len; +} + +void parse_names(char *buf, int size, char ***namesp) +{ + char **names = NULL; + int names_cap = 0; + int names_len = 0; + + char *p = buf; + char *end = buf + size; + while (p < end) { + char *next = strchr(p, '\n'); + if (next != NULL) { + *next = 0; + } else { + next = end; + } + if (p < next) { + if (names_len == names_cap) { + names_cap = 2 * names_cap + 1; + names = reftable_realloc( + names, names_cap * sizeof(char *)); + } + names[names_len++] = xstrdup(p); + } + p = next + 1; + } + + if (names_len == names_cap) { + names_cap = 2 * names_cap + 1; + names = reftable_realloc(names, names_cap * sizeof(char *)); + } + + names[names_len] = NULL; + *namesp = names; +} + +int names_equal(char **a, char **b) +{ + while (*a && *b) { + if (strcmp(*a, *b)) { + return 0; + } + + a++; + b++; + } + + return *a == *b; +} + +const char *reftable_error_str(int err) +{ + static char buf[250]; + switch (err) { + case REFTABLE_IO_ERROR: + return "I/O error"; + case REFTABLE_FORMAT_ERROR: + return "corrupt reftable file"; + case REFTABLE_NOT_EXIST_ERROR: + return "file does not exist"; + case REFTABLE_LOCK_ERROR: + return "data is outdated"; + case REFTABLE_API_ERROR: + return "misuse of the reftable API"; + case REFTABLE_ZLIB_ERROR: + return "zlib failure"; + case REFTABLE_NAME_CONFLICT: + return "file/directory conflict"; + case REFTABLE_REFNAME_ERROR: + return "invalid refname"; + case -1: + return "general error"; + default: + snprintf(buf, sizeof(buf), "unknown error code %d", err); + return buf; + } +} + +int reftable_error_to_errno(int err) +{ + switch (err) { + case REFTABLE_IO_ERROR: + return EIO; + case REFTABLE_FORMAT_ERROR: + return EFAULT; + case REFTABLE_NOT_EXIST_ERROR: + return ENOENT; + case REFTABLE_LOCK_ERROR: + return EBUSY; + case REFTABLE_API_ERROR: + return EINVAL; + case REFTABLE_ZLIB_ERROR: + return EDOM; + default: + return ERANGE; + } +} + +void *(*reftable_malloc_ptr)(size_t sz) = &malloc; +void *(*reftable_realloc_ptr)(void *, size_t) = &realloc; +void (*reftable_free_ptr)(void *) = &free; + +void *reftable_malloc(size_t sz) +{ + return (*reftable_malloc_ptr)(sz); +} + +void *reftable_realloc(void *p, size_t sz) +{ + return (*reftable_realloc_ptr)(p, sz); +} + +void reftable_free(void *p) +{ + reftable_free_ptr(p); +} + +void *reftable_calloc(size_t sz) +{ + void *p = reftable_malloc(sz); + memset(p, 0, sz); + return p; +} + +void reftable_set_alloc(void *(*malloc)(size_t), + void *(*realloc)(void *, size_t), void (*free)(void *)) +{ + reftable_malloc_ptr = malloc; + reftable_realloc_ptr = realloc; + reftable_free_ptr = free; +} diff --git a/reftable/basics.h b/reftable/basics.h new file mode 100644 index 00000000000..1b003485c9c --- /dev/null +++ b/reftable/basics.h @@ -0,0 +1,53 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include "system.h" + +#include "reftable.h" + +#define true 1 +#define false 0 + +/* Bigendian en/decoding of integers */ + +void put_be24(byte *out, uint32_t i); +uint32_t get_be24(byte *in); +void put_be16(uint8_t *out, uint16_t i); + +/* + find smallest index i in [0, sz) at which f(i) is true, assuming + that f is ascending. Return sz if f(i) is false for all indices. +*/ +int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args); + +/* + Frees a NULL terminated array of malloced strings. The array itself is also + freed. + */ +void free_names(char **a); + +/* parse a newline separated list of names. Empty names are discarded. */ +void parse_names(char *buf, int size, char ***namesp); + +/* compares two NULL-terminated arrays of strings. */ +int names_equal(char **a, char **b); + +/* returns the array size of a NULL-terminated array of strings. */ +int names_length(char **names); + +/* Allocation routines; they invoke the functions set through + * reftable_set_alloc() */ +void *reftable_malloc(size_t sz); +void *reftable_realloc(void *p, size_t sz); +void reftable_free(void *p); +void *reftable_calloc(size_t sz); + +#endif diff --git a/reftable/block.c b/reftable/block.c new file mode 100644 index 00000000000..968b192979a --- /dev/null +++ b/reftable/block.c @@ -0,0 +1,436 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "block.h" + +#include "system.h" + +#include "blocksource.h" +#include "constants.h" +#include "record.h" +#include "reftable.h" +#include "zlib.h" + +int header_size(int version) +{ + switch (version) { + case 1: + return 24; + case 2: + return 28; + } + abort(); +} + +int footer_size(int version) +{ + switch (version) { + case 1: + return 68; + case 2: + return 72; + } + abort(); +} + +int block_writer_register_restart(struct block_writer *w, int n, bool restart, + struct slice key); + +void block_writer_init(struct block_writer *bw, byte typ, byte *buf, + uint32_t block_size, uint32_t header_off, int hash_size) +{ + bw->buf = buf; + bw->hash_size = hash_size; + bw->block_size = block_size; + bw->header_off = header_off; + bw->buf[header_off] = typ; + bw->next = header_off + 4; + bw->restart_interval = 16; + bw->entries = 0; + bw->restart_len = 0; + bw->last_key.len = 0; +} + +byte block_writer_type(struct block_writer *bw) +{ + return bw->buf[bw->header_off]; +} + +/* adds the record to the block. Returns -1 if it does not fit, 0 on + success */ +int block_writer_add(struct block_writer *w, struct record rec) +{ + struct slice empty = { 0 }; + struct slice last = w->entries % w->restart_interval == 0 ? empty : + w->last_key; + struct slice out = { + .buf = w->buf + w->next, + .len = w->block_size - w->next, + }; + + struct slice start = out; + + bool restart = false; + struct slice key = { 0 }; + int n = 0; + + record_key(rec, &key); + n = encode_key(&restart, out, last, key, record_val_type(rec)); + if (n < 0) { + goto err; + } + slice_consume(&out, n); + + n = record_encode(rec, out, w->hash_size); + if (n < 0) { + goto err; + } + slice_consume(&out, n); + + if (block_writer_register_restart(w, start.len - out.len, restart, + key) < 0) { + goto err; + } + + slice_clear(&key); + return 0; + +err: + slice_clear(&key); + return -1; +} + +int block_writer_register_restart(struct block_writer *w, int n, bool restart, + struct slice key) +{ + int rlen = w->restart_len; + if (rlen >= MAX_RESTARTS) { + restart = false; + } + + if (restart) { + rlen++; + } + if (2 + 3 * rlen + n > w->block_size - w->next) { + return -1; + } + if (restart) { + if (w->restart_len == w->restart_cap) { + w->restart_cap = w->restart_cap * 2 + 1; + w->restarts = reftable_realloc( + w->restarts, sizeof(uint32_t) * w->restart_cap); + } + + w->restarts[w->restart_len++] = w->next; + } + + w->next += n; + slice_copy(&w->last_key, key); + w->entries++; + return 0; +} + +int block_writer_finish(struct block_writer *w) +{ + int i = 0; + for (i = 0; i < w->restart_len; i++) { + put_be24(w->buf + w->next, w->restarts[i]); + w->next += 3; + } + + put_be16(w->buf + w->next, w->restart_len); + w->next += 2; + put_be24(w->buf + 1 + w->header_off, w->next); + + if (block_writer_type(w) == BLOCK_TYPE_LOG) { + int block_header_skip = 4 + w->header_off; + struct slice compressed = { 0 }; + int zresult = 0; + uLongf src_len = w->next - block_header_skip; + slice_resize(&compressed, src_len); + + while (1) { + uLongf dest_len = compressed.len; + + zresult = compress2(compressed.buf, &dest_len, + w->buf + block_header_skip, src_len, + 9); + if (zresult == Z_BUF_ERROR) { + slice_resize(&compressed, 2 * compressed.len); + continue; + } + + if (Z_OK != zresult) { + slice_clear(&compressed); + return REFTABLE_ZLIB_ERROR; + } + + memcpy(w->buf + block_header_skip, compressed.buf, + dest_len); + w->next = dest_len + block_header_skip; + slice_clear(&compressed); + break; + } + } + return w->next; +} + +byte block_reader_type(struct block_reader *r) +{ + return r->block.data[r->header_off]; +} + +int block_reader_init(struct block_reader *br, struct reftable_block *block, + uint32_t header_off, uint32_t table_block_size, + int hash_size) +{ + uint32_t full_block_size = table_block_size; + byte typ = block->data[header_off]; + uint32_t sz = get_be24(block->data + header_off + 1); + + if (!is_block_type(typ)) { + return REFTABLE_FORMAT_ERROR; + } + + if (typ == BLOCK_TYPE_LOG) { + struct slice uncompressed = { 0 }; + int block_header_skip = 4 + header_off; + uLongf dst_len = sz - block_header_skip; /* total size of dest + buffer. */ + uLongf src_len = block->len - block_header_skip; + + /* Log blocks specify the *uncompressed* size in their header. + */ + slice_resize(&uncompressed, sz); + + /* Copy over the block header verbatim. It's not compressed. */ + memcpy(uncompressed.buf, block->data, block_header_skip); + + /* Uncompress */ + if (Z_OK != uncompress_return_consumed( + uncompressed.buf + block_header_skip, + &dst_len, block->data + block_header_skip, + &src_len)) { + slice_clear(&uncompressed); + return REFTABLE_ZLIB_ERROR; + } + + if (dst_len + block_header_skip != sz) { + return REFTABLE_FORMAT_ERROR; + } + + /* We're done with the input data. */ + block_source_return_block(block->source, block); + block->data = uncompressed.buf; + block->len = sz; + block->source = malloc_block_source(); + full_block_size = src_len + block_header_skip; + } else if (full_block_size == 0) { + full_block_size = sz; + } else if (sz < full_block_size && sz < block->len && + block->data[sz] != 0) { + /* If the block is smaller than the full block size, it is + padded (data followed by '\0') or the next block is + unaligned. */ + full_block_size = sz; + } + + { + uint16_t restart_count = get_be16(block->data + sz - 2); + uint32_t restart_start = sz - 2 - 3 * restart_count; + + byte *restart_bytes = block->data + restart_start; + + /* transfer ownership. */ + br->block = *block; + block->data = NULL; + block->len = 0; + + br->hash_size = hash_size; + br->block_len = restart_start; + br->full_block_size = full_block_size; + br->header_off = header_off; + br->restart_count = restart_count; + br->restart_bytes = restart_bytes; + } + + return 0; +} + +static uint32_t block_reader_restart_offset(struct block_reader *br, int i) +{ + return get_be24(br->restart_bytes + 3 * i); +} + +void block_reader_start(struct block_reader *br, struct block_iter *it) +{ + it->br = br; + slice_resize(&it->last_key, 0); + it->next_off = br->header_off + 4; +} + +struct restart_find_args { + int error; + struct slice key; + struct block_reader *r; +}; + +static int restart_key_less(size_t idx, void *args) +{ + struct restart_find_args *a = (struct restart_find_args *)args; + uint32_t off = block_reader_restart_offset(a->r, idx); + struct slice in = { + .buf = a->r->block.data + off, + .len = a->r->block_len - off, + }; + + /* the restart key is verbatim in the block, so this could avoid the + alloc for decoding the key */ + struct slice rkey = { 0 }; + struct slice last_key = { 0 }; + byte unused_extra; + int n = decode_key(&rkey, &unused_extra, last_key, in); + if (n < 0) { + a->error = 1; + return -1; + } + + { + int result = slice_compare(a->key, rkey); + slice_clear(&rkey); + return result; + } +} + +void block_iter_copy_from(struct block_iter *dest, struct block_iter *src) +{ + dest->br = src->br; + dest->next_off = src->next_off; + slice_copy(&dest->last_key, src->last_key); +} + +int block_iter_next(struct block_iter *it, struct record rec) +{ + if (it->next_off >= it->br->block_len) { + return 1; + } + + { + struct slice in = { + .buf = it->br->block.data + it->next_off, + .len = it->br->block_len - it->next_off, + }; + struct slice start = in; + struct slice key = { 0 }; + byte extra; + int n = decode_key(&key, &extra, it->last_key, in); + if (n < 0) { + return -1; + } + + slice_consume(&in, n); + n = record_decode(rec, key, extra, in, it->br->hash_size); + if (n < 0) { + return -1; + } + slice_consume(&in, n); + + slice_copy(&it->last_key, key); + it->next_off += start.len - in.len; + slice_clear(&key); + return 0; + } +} + +int block_reader_first_key(struct block_reader *br, struct slice *key) +{ + struct slice empty = { 0 }; + int off = br->header_off + 4; + struct slice in = { + .buf = br->block.data + off, + .len = br->block_len - off, + }; + + byte extra = 0; + int n = decode_key(key, &extra, empty, in); + if (n < 0) { + return n; + } + return 0; +} + +int block_iter_seek(struct block_iter *it, struct slice want) +{ + return block_reader_seek(it->br, it, want); +} + +void block_iter_close(struct block_iter *it) +{ + slice_clear(&it->last_key); +} + +int block_reader_seek(struct block_reader *br, struct block_iter *it, + struct slice want) +{ + struct restart_find_args args = { + .key = want, + .r = br, + }; + + int i = binsearch(br->restart_count, &restart_key_less, &args); + if (args.error) { + return -1; + } + + it->br = br; + if (i > 0) { + i--; + it->next_off = block_reader_restart_offset(br, i); + } else { + it->next_off = br->header_off + 4; + } + + { + struct record rec = new_record(block_reader_type(br)); + struct slice key = { 0 }; + int result = 0; + int err = 0; + struct block_iter next = { 0 }; + while (true) { + block_iter_copy_from(&next, it); + + err = block_iter_next(&next, rec); + if (err < 0) { + result = -1; + goto exit; + } + + record_key(rec, &key); + if (err > 0 || slice_compare(key, want) >= 0) { + result = 0; + goto exit; + } + + block_iter_copy_from(it, &next); + } + + exit: + slice_clear(&key); + slice_clear(&next.last_key); + record_destroy(&rec); + + return result; + } +} + +void block_writer_clear(struct block_writer *bw) +{ + FREE_AND_NULL(bw->restarts); + slice_clear(&bw->last_key); + /* the block is not owned. */ +} diff --git a/reftable/block.h b/reftable/block.h new file mode 100644 index 00000000000..320a2d6753b --- /dev/null +++ b/reftable/block.h @@ -0,0 +1,126 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCK_H +#define BLOCK_H + +#include "basics.h" +#include "record.h" +#include "reftable.h" + +/* + Writes reftable blocks. The block_writer is reused across blocks to minimize + allocation overhead. +*/ +struct block_writer { + byte *buf; + uint32_t block_size; + + /* Offset ofof the global header. Nonzero in the first block only. */ + uint32_t header_off; + + /* How often to restart keys. */ + int restart_interval; + int hash_size; + + /* Offset of next byte to write. */ + uint32_t next; + uint32_t *restarts; + uint32_t restart_len; + uint32_t restart_cap; + + struct slice last_key; + int entries; +}; + +/* + initializes the blockwriter to write `typ` entries, using `buf` as temporary + storage. `buf` is not owned by the block_writer. */ +void block_writer_init(struct block_writer *bw, byte typ, byte *buf, + uint32_t block_size, uint32_t header_off, int hash_size); + +/* + returns the block type (eg. 'r' for ref records. +*/ +byte block_writer_type(struct block_writer *bw); + +/* appends the record, or -1 if it doesn't fit. */ +int block_writer_add(struct block_writer *w, struct record rec); + +/* appends the key restarts, and compress the block if necessary. */ +int block_writer_finish(struct block_writer *w); + +/* clears out internally allocated block_writer members. */ +void block_writer_clear(struct block_writer *bw); + +/* Read a block. */ +struct block_reader { + /* offset of the block header; nonzero for the first block in a + * reftable. */ + uint32_t header_off; + + /* the memory block */ + struct reftable_block block; + int hash_size; + + /* size of the data, excluding restart data. */ + uint32_t block_len; + byte *restart_bytes; + uint16_t restart_count; + + /* size of the data in the file. For log blocks, this is the compressed + * size. */ + uint32_t full_block_size; +}; + +/* Iterate over entries in a block */ +struct block_iter { + /* offset within the block of the next entry to read. */ + uint32_t next_off; + struct block_reader *br; + + /* key for last entry we read. */ + struct slice last_key; +}; + +/* initializes a block reader */ +int block_reader_init(struct block_reader *br, struct reftable_block *bl, + uint32_t header_off, uint32_t table_block_size, + int hash_size); + +/* Position `it` at start of the block */ +void block_reader_start(struct block_reader *br, struct block_iter *it); + +/* Position `it` to the `want` key in the block */ +int block_reader_seek(struct block_reader *br, struct block_iter *it, + struct slice want); + +/* Returns the block type (eg. 'r' for refs) */ +byte block_reader_type(struct block_reader *r); + +/* Decodes the first key in the block */ +int block_reader_first_key(struct block_reader *br, struct slice *key); + +void block_iter_copy_from(struct block_iter *dest, struct block_iter *src); + +/* return < 0 for error, 0 for OK, > 0 for EOF. */ +int block_iter_next(struct block_iter *it, struct record rec); + +/* Seek to `want` with in the block pointed to by `it` */ +int block_iter_seek(struct block_iter *it, struct slice want); + +/* deallocate memory for `it`. The block reader and its block is left intact. */ +void block_iter_close(struct block_iter *it); + +/* size of file header, depending on format version */ +int header_size(int version); + +/* size of file footer, depending on format version */ +int footer_size(int version); + +#endif diff --git a/reftable/blocksource.h b/reftable/blocksource.h new file mode 100644 index 00000000000..1fdf0bf4557 --- /dev/null +++ b/reftable/blocksource.h @@ -0,0 +1,22 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size(struct reftable_block_source source); +int block_source_read_block(struct reftable_block_source source, + struct reftable_block *dest, uint64_t off, + uint32_t size); +void block_source_return_block(struct reftable_block_source source, + struct reftable_block *ret); +void block_source_close(struct reftable_block_source source); + +#endif diff --git a/reftable/constants.h b/reftable/constants.h new file mode 100644 index 00000000000..5eee72c4c11 --- /dev/null +++ b/reftable/constants.h @@ -0,0 +1,21 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif diff --git a/reftable/file.c b/reftable/file.c new file mode 100644 index 00000000000..2bf1135a2e2 --- /dev/null +++ b/reftable/file.c @@ -0,0 +1,99 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "system.h" + +#include "block.h" +#include "iter.h" +#include "record.h" +#include "reftable.h" +#include "tree.h" + +struct file_block_source { + int fd; + uint64_t size; +}; + +static uint64_t file_size(void *b) +{ + return ((struct file_block_source *)b)->size; +} + +static void file_return_block(void *b, struct reftable_block *dest) +{ + memset(dest->data, 0xff, dest->len); + reftable_free(dest->data); +} + +static void file_close(void *b) +{ + int fd = ((struct file_block_source *)b)->fd; + if (fd > 0) { + close(fd); + ((struct file_block_source *)b)->fd = 0; + } + + reftable_free(b); +} + +static int file_read_block(void *v, struct reftable_block *dest, uint64_t off, + uint32_t size) +{ + struct file_block_source *b = (struct file_block_source *)v; + assert(off + size <= b->size); + dest->data = reftable_malloc(size); + if (pread(b->fd, dest->data, size, off) != size) { + return -1; + } + dest->len = size; + return size; +} + +struct reftable_block_source_vtable file_vtable = { + .size = &file_size, + .read_block = &file_read_block, + .return_block = &file_return_block, + .close = &file_close, +}; + +int reftable_block_source_from_file(struct reftable_block_source *bs, + const char *name) +{ + struct stat st = { 0 }; + int err = 0; + int fd = open(name, O_RDONLY); + if (fd < 0) { + if (errno == ENOENT) { + return REFTABLE_NOT_EXIST_ERROR; + } + return -1; + } + + err = fstat(fd, &st); + if (err < 0) { + return -1; + } + + { + struct file_block_source *p = + reftable_calloc(sizeof(struct file_block_source)); + p->size = st.st_size; + p->fd = fd; + + assert(bs->ops == NULL); + bs->ops = &file_vtable; + bs->arg = p; + } + return 0; +} + +int reftable_fd_write(void *arg, byte *data, size_t sz) +{ + int *fdp = (int *)arg; + return write(*fdp, data, sz); +} diff --git a/reftable/iter.c b/reftable/iter.c new file mode 100644 index 00000000000..145fa1393ad --- /dev/null +++ b/reftable/iter.c @@ -0,0 +1,240 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "iter.h" + +#include "system.h" + +#include "block.h" +#include "constants.h" +#include "reader.h" +#include "reftable.h" + +bool iterator_is_null(struct reftable_iterator it) +{ + return it.ops == NULL; +} + +static int empty_iterator_next(void *arg, struct record rec) +{ + return 1; +} + +static void empty_iterator_close(void *arg) +{ +} + +struct reftable_iterator_vtable empty_vtable = { + .next = &empty_iterator_next, + .close = &empty_iterator_close, +}; + +void iterator_set_empty(struct reftable_iterator *it) +{ + assert(it->ops == NULL); + it->iter_arg = NULL; + it->ops = &empty_vtable; +} + +int iterator_next(struct reftable_iterator it, struct record rec) +{ + return it.ops->next(it.iter_arg, rec); +} + +void reftable_iterator_destroy(struct reftable_iterator *it) +{ + if (it->ops == NULL) { + return; + } + it->ops->close(it->iter_arg); + it->ops = NULL; + FREE_AND_NULL(it->iter_arg); +} + +int reftable_iterator_next_ref(struct reftable_iterator it, + struct reftable_ref_record *ref) +{ + struct record rec = { 0 }; + record_from_ref(&rec, ref); + return iterator_next(it, rec); +} + +int reftable_iterator_next_log(struct reftable_iterator it, + struct reftable_log_record *log) +{ + struct record rec = { 0 }; + record_from_log(&rec, log); + return iterator_next(it, rec); +} + +static void filtering_ref_iterator_close(void *iter_arg) +{ + struct filtering_ref_iterator *fri = + (struct filtering_ref_iterator *)iter_arg; + slice_clear(&fri->oid); + reftable_iterator_destroy(&fri->it); +} + +static int filtering_ref_iterator_next(void *iter_arg, struct record rec) +{ + struct filtering_ref_iterator *fri = + (struct filtering_ref_iterator *)iter_arg; + struct reftable_ref_record *ref = + (struct reftable_ref_record *)rec.data; + int err = 0; + while (true) { + err = reftable_iterator_next_ref(fri->it, ref); + if (err != 0) { + break; + } + + if (fri->double_check) { + struct reftable_iterator it = { 0 }; + + err = reftable_reader_seek_ref(fri->r, &it, + ref->ref_name); + if (err == 0) { + err = reftable_iterator_next_ref(it, ref); + } + + reftable_iterator_destroy(&it); + + if (err < 0) { + break; + } + + if (err > 0) { + continue; + } + } + + if ((ref->target_value != NULL && + !memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) || + (ref->value != NULL && + !memcmp(fri->oid.buf, ref->value, fri->oid.len))) { + return 0; + } + } + + reftable_ref_record_clear(ref); + return err; +} + +struct reftable_iterator_vtable filtering_ref_iterator_vtable = { + .next = &filtering_ref_iterator_next, + .close = &filtering_ref_iterator_close, +}; + +void iterator_from_filtering_ref_iterator(struct reftable_iterator *it, + struct filtering_ref_iterator *fri) +{ + assert(it->ops == NULL); + it->iter_arg = fri; + it->ops = &filtering_ref_iterator_vtable; +} + +static void indexed_table_ref_iter_close(void *p) +{ + struct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p; + block_iter_close(&it->cur); + reader_return_block(it->r, &it->block_reader.block); + slice_clear(&it->oid); +} + +static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it) +{ + if (it->offset_idx == it->offset_len) { + it->finished = true; + return 1; + } + + reader_return_block(it->r, &it->block_reader.block); + + { + uint64_t off = it->offsets[it->offset_idx++]; + int err = reader_init_block_reader(it->r, &it->block_reader, + off, BLOCK_TYPE_REF); + if (err < 0) { + return err; + } + if (err > 0) { + /* indexed block does not exist. */ + return REFTABLE_FORMAT_ERROR; + } + } + block_reader_start(&it->block_reader, &it->cur); + return 0; +} + +static int indexed_table_ref_iter_next(void *p, struct record rec) +{ + struct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p; + struct reftable_ref_record *ref = + (struct reftable_ref_record *)rec.data; + + while (true) { + int err = block_iter_next(&it->cur, rec); + if (err < 0) { + return err; + } + + if (err > 0) { + err = indexed_table_ref_iter_next_block(it); + if (err < 0) { + return err; + } + + if (it->finished) { + return 1; + } + continue; + } + + if (!memcmp(it->oid.buf, ref->target_value, it->oid.len) || + !memcmp(it->oid.buf, ref->value, it->oid.len)) { + return 0; + } + } +} + +int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest, + struct reftable_reader *r, byte *oid, + int oid_len, uint64_t *offsets, int offset_len) +{ + struct indexed_table_ref_iter *itr = + reftable_calloc(sizeof(struct indexed_table_ref_iter)); + int err = 0; + + itr->r = r; + slice_resize(&itr->oid, oid_len); + memcpy(itr->oid.buf, oid, oid_len); + + itr->offsets = offsets; + itr->offset_len = offset_len; + + err = indexed_table_ref_iter_next_block(itr); + if (err < 0) { + reftable_free(itr); + } else { + *dest = itr; + } + return err; +} + +struct reftable_iterator_vtable indexed_table_ref_iter_vtable = { + .next = &indexed_table_ref_iter_next, + .close = &indexed_table_ref_iter_close, +}; + +void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it, + struct indexed_table_ref_iter *itr) +{ + assert(it->ops == NULL); + it->iter_arg = itr; + it->ops = &indexed_table_ref_iter_vtable; +} diff --git a/reftable/iter.h b/reftable/iter.h new file mode 100644 index 00000000000..ef1d3964baa --- /dev/null +++ b/reftable/iter.h @@ -0,0 +1,63 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef ITER_H +#define ITER_H + +#include "block.h" +#include "record.h" +#include "slice.h" + +struct reftable_iterator_vtable { + int (*next)(void *iter_arg, struct record rec); + void (*close)(void *iter_arg); +}; + +void iterator_set_empty(struct reftable_iterator *it); +int iterator_next(struct reftable_iterator it, struct record rec); + +/* Returns true for a zeroed out iterator, such as the one returned from + iterator_destroy. */ +bool iterator_is_null(struct reftable_iterator it); + +/* iterator that produces only ref records that point to `oid` */ +struct filtering_ref_iterator { + bool double_check; + struct reftable_reader *r; + struct slice oid; + struct reftable_iterator it; +}; + +void iterator_from_filtering_ref_iterator(struct reftable_iterator *, + struct filtering_ref_iterator *); + +/* iterator that produces only ref records that point to `oid`, + but using the object index. + */ +struct indexed_table_ref_iter { + struct reftable_reader *r; + struct slice oid; + + /* mutable */ + uint64_t *offsets; + + /* Points to the next offset to read. */ + int offset_idx; + int offset_len; + struct block_reader block_reader; + struct block_iter cur; + bool finished; +}; + +void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it, + struct indexed_table_ref_iter *itr); +int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest, + struct reftable_reader *r, byte *oid, + int oid_len, uint64_t *offsets, int offset_len); + +#endif diff --git a/reftable/merged.c b/reftable/merged.c new file mode 100644 index 00000000000..5a1b896e0b0 --- /dev/null +++ b/reftable/merged.c @@ -0,0 +1,327 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "merged.h" + +#include "system.h" + +#include "constants.h" +#include "iter.h" +#include "pq.h" +#include "reader.h" + +static int merged_iter_init(struct merged_iter *mi) +{ + int i = 0; + for (i = 0; i < mi->stack_len; i++) { + struct record rec = new_record(mi->typ); + int err = iterator_next(mi->stack[i], rec); + if (err < 0) { + return err; + } + + if (err > 0) { + reftable_iterator_destroy(&mi->stack[i]); + record_destroy(&rec); + } else { + struct pq_entry e = { + .rec = rec, + .index = i, + }; + merged_iter_pqueue_add(&mi->pq, e); + } + } + + return 0; +} + +static void merged_iter_close(void *p) +{ + struct merged_iter *mi = (struct merged_iter *)p; + int i = 0; + merged_iter_pqueue_clear(&mi->pq); + for (i = 0; i < mi->stack_len; i++) { + reftable_iterator_destroy(&mi->stack[i]); + } + reftable_free(mi->stack); +} + +static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx) +{ + if (iterator_is_null(mi->stack[idx])) { + return 0; + } + + { + struct record rec = new_record(mi->typ); + struct pq_entry e = { + .rec = rec, + .index = idx, + }; + int err = iterator_next(mi->stack[idx], rec); + if (err < 0) { + return err; + } + + if (err > 0) { + reftable_iterator_destroy(&mi->stack[idx]); + record_destroy(&rec); + return 0; + } + + merged_iter_pqueue_add(&mi->pq, e); + } + return 0; +} + +static int merged_iter_next_entry(struct merged_iter *mi, struct record rec) +{ + struct slice entry_key = { 0 }; + struct pq_entry entry = { 0 }; + int err = 0; + + if (merged_iter_pqueue_is_empty(mi->pq)) { + return 1; + } + + entry = merged_iter_pqueue_remove(&mi->pq); + err = merged_iter_advance_subiter(mi, entry.index); + if (err < 0) { + return err; + } + + /* + One can also use reftable as datacenter-local storage, where the ref + database is maintained in globally consistent database (eg. + CockroachDB or Spanner). In this scenario, replication delays together + with compaction may cause newer tables to contain older entries. In + such a deployment, the loop below must be changed to collect all + entries for the same key, and return new the newest one. + */ + record_key(entry.rec, &entry_key); + while (!merged_iter_pqueue_is_empty(mi->pq)) { + struct pq_entry top = merged_iter_pqueue_top(mi->pq); + struct slice k = { 0 }; + int err = 0, cmp = 0; + + record_key(top.rec, &k); + + cmp = slice_compare(k, entry_key); + slice_clear(&k); + + if (cmp > 0) { + break; + } + + merged_iter_pqueue_remove(&mi->pq); + err = merged_iter_advance_subiter(mi, top.index); + if (err < 0) { + return err; + } + record_clear(top.rec); + reftable_free(record_yield(&top.rec)); + } + + record_copy_from(rec, entry.rec, hash_size(mi->hash_id)); + record_clear(entry.rec); + reftable_free(record_yield(&entry.rec)); + slice_clear(&entry_key); + return 0; +} + +static int merged_iter_next(struct merged_iter *mi, struct record rec) +{ + while (true) { + int err = merged_iter_next_entry(mi, rec); + if (err == 0 && mi->suppress_deletions && + record_is_deletion(rec)) { + continue; + } + + return err; + } +} + +static int merged_iter_next_void(void *p, struct record rec) +{ + struct merged_iter *mi = (struct merged_iter *)p; + if (merged_iter_pqueue_is_empty(mi->pq)) { + return 1; + } + + return merged_iter_next(mi, rec); +} + +struct reftable_iterator_vtable merged_iter_vtable = { + .next = &merged_iter_next_void, + .close = &merged_iter_close, +}; + +static void iterator_from_merged_iter(struct reftable_iterator *it, + struct merged_iter *mi) +{ + assert(it->ops == NULL); + it->iter_arg = mi; + it->ops = &merged_iter_vtable; +} + +int reftable_new_merged_table(struct reftable_merged_table **dest, + struct reftable_reader **stack, int n, + uint32_t hash_id) +{ + uint64_t last_max = 0; + uint64_t first_min = 0; + int i = 0; + for (i = 0; i < n; i++) { + struct reftable_reader *r = stack[i]; + if (r->hash_id != hash_id) { + return REFTABLE_FORMAT_ERROR; + } + if (i > 0 && last_max >= reftable_reader_min_update_index(r)) { + return REFTABLE_FORMAT_ERROR; + } + if (i == 0) { + first_min = reftable_reader_min_update_index(r); + } + + last_max = reftable_reader_max_update_index(r); + } + + { + struct reftable_merged_table m = { + .stack = stack, + .stack_len = n, + .min = first_min, + .max = last_max, + .hash_id = hash_id, + }; + + *dest = reftable_calloc(sizeof(struct reftable_merged_table)); + **dest = m; + } + return 0; +} + +void reftable_merged_table_close(struct reftable_merged_table *mt) +{ + int i = 0; + for (i = 0; i < mt->stack_len; i++) { + reftable_reader_free(mt->stack[i]); + } + FREE_AND_NULL(mt->stack); + mt->stack_len = 0; +} + +/* clears the list of subtable, without affecting the readers themselves. */ +void merged_table_clear(struct reftable_merged_table *mt) +{ + FREE_AND_NULL(mt->stack); + mt->stack_len = 0; +} + +void reftable_merged_table_free(struct reftable_merged_table *mt) +{ + if (mt == NULL) { + return; + } + merged_table_clear(mt); + reftable_free(mt); +} + +uint64_t +reftable_merged_table_max_update_index(struct reftable_merged_table *mt) +{ + return mt->max; +} + +uint64_t +reftable_merged_table_min_update_index(struct reftable_merged_table *mt) +{ + return mt->min; +} + +int merged_table_seek_record(struct reftable_merged_table *mt, + struct reftable_iterator *it, struct record rec) +{ + struct reftable_iterator *iters = reftable_calloc( + sizeof(struct reftable_iterator) * mt->stack_len); + struct merged_iter merged = { + .stack = iters, + .typ = record_type(rec), + .hash_id = mt->hash_id, + .suppress_deletions = mt->suppress_deletions, + }; + int n = 0; + int err = 0; + int i = 0; + for (i = 0; i < mt->stack_len && err == 0; i++) { + int e = reader_seek(mt->stack[i], &iters[n], rec); + if (e < 0) { + err = e; + } + if (e == 0) { + n++; + } + } + if (err < 0) { + int i = 0; + for (i = 0; i < n; i++) { + reftable_iterator_destroy(&iters[i]); + } + reftable_free(iters); + return err; + } + + merged.stack_len = n; + err = merged_iter_init(&merged); + if (err < 0) { + merged_iter_close(&merged); + return err; + } + + { + struct merged_iter *p = + reftable_malloc(sizeof(struct merged_iter)); + *p = merged; + iterator_from_merged_iter(it, p); + } + return 0; +} + +int reftable_merged_table_seek_ref(struct reftable_merged_table *mt, + struct reftable_iterator *it, + const char *name) +{ + struct reftable_ref_record ref = { + .ref_name = (char *)name, + }; + struct record rec = { 0 }; + record_from_ref(&rec, &ref); + return merged_table_seek_record(mt, it, rec); +} + +int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt, + struct reftable_iterator *it, + const char *name, uint64_t update_index) +{ + struct reftable_log_record log = { + .ref_name = (char *)name, + .update_index = update_index, + }; + struct record rec = { 0 }; + record_from_log(&rec, &log); + return merged_table_seek_record(mt, it, rec); +} + +int reftable_merged_table_seek_log(struct reftable_merged_table *mt, + struct reftable_iterator *it, + const char *name) +{ + uint64_t max = ~((uint64_t)0); + return reftable_merged_table_seek_log_at(mt, it, name, max); +} diff --git a/reftable/merged.h b/reftable/merged.h new file mode 100644 index 00000000000..a71f36ecbae --- /dev/null +++ b/reftable/merged.h @@ -0,0 +1,38 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct reftable_merged_table { + struct reftable_reader **stack; + int stack_len; + uint32_t hash_id; + bool suppress_deletions; + + uint64_t min; + uint64_t max; +}; + +struct merged_iter { + struct reftable_iterator *stack; + uint32_t hash_id; + int stack_len; + byte typ; + bool suppress_deletions; + struct merged_iter_pqueue pq; +}; + +void merged_table_clear(struct reftable_merged_table *mt); +int merged_table_seek_record(struct reftable_merged_table *mt, + struct reftable_iterator *it, struct record rec); + +#endif diff --git a/reftable/pq.c b/reftable/pq.c new file mode 100644 index 00000000000..bfad96c6695 --- /dev/null +++ b/reftable/pq.c @@ -0,0 +1,115 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "pq.h" + +#include "system.h" + +int pq_less(struct pq_entry a, struct pq_entry b) +{ + struct slice ak = { 0 }; + struct slice bk = { 0 }; + int cmp = 0; + record_key(a.rec, &ak); + record_key(b.rec, &bk); + + cmp = slice_compare(ak, bk); + + slice_clear(&ak); + slice_clear(&bk); + + if (cmp == 0) { + return a.index > b.index; + } + + return cmp < 0; +} + +struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq) +{ + return pq.heap[0]; +} + +bool merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq) +{ + return pq.len == 0; +} + +void merged_iter_pqueue_check(struct merged_iter_pqueue pq) +{ + int i = 0; + for (i = 1; i < pq.len; i++) { + int parent = (i - 1) / 2; + + assert(pq_less(pq.heap[parent], pq.heap[i])); + } +} + +struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq) +{ + int i = 0; + struct pq_entry e = pq->heap[0]; + pq->heap[0] = pq->heap[pq->len - 1]; + pq->len--; + + i = 0; + while (i < pq->len) { + int min = i; + int j = 2 * i + 1; + int k = 2 * i + 2; + if (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) { + min = j; + } + if (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) { + min = k; + } + + if (min == i) { + break; + } + + SWAP(pq->heap[i], pq->heap[min]); + i = min; + } + + return e; +} + +void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e) +{ + int i = 0; + if (pq->len == pq->cap) { + pq->cap = 2 * pq->cap + 1; + pq->heap = reftable_realloc(pq->heap, + pq->cap * sizeof(struct pq_entry)); + } + + pq->heap[pq->len++] = e; + i = pq->len - 1; + while (i > 0) { + int j = (i - 1) / 2; + if (pq_less(pq->heap[j], pq->heap[i])) { + break; + } + + SWAP(pq->heap[j], pq->heap[i]); + + i = j; + } +} + +void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq) +{ + int i = 0; + for (i = 0; i < pq->len; i++) { + record_clear(pq->heap[i].rec); + reftable_free(record_yield(&pq->heap[i].rec)); + } + FREE_AND_NULL(pq->heap); + pq->len = pq->cap = 0; +} diff --git a/reftable/pq.h b/reftable/pq.h new file mode 100644 index 00000000000..b585a52ee14 --- /dev/null +++ b/reftable/pq.h @@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { + int index; + struct record rec; +}; + +int pq_less(struct pq_entry a, struct pq_entry b); + +struct merged_iter_pqueue { + struct pq_entry *heap; + int len; + int cap; +}; + +struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq); +void merged_iter_pqueue_check(struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e); +void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq); + +#endif diff --git a/reftable/reader.c b/reftable/reader.c new file mode 100644 index 00000000000..00913bc3b54 --- /dev/null +++ b/reftable/reader.c @@ -0,0 +1,758 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "reader.h" + +#include "system.h" + +#include "block.h" +#include "constants.h" +#include "iter.h" +#include "record.h" +#include "reftable.h" +#include "tree.h" + +uint64_t block_source_size(struct reftable_block_source source) +{ + return source.ops->size(source.arg); +} + +int block_source_read_block(struct reftable_block_source source, + struct reftable_block *dest, uint64_t off, + uint32_t size) +{ + int result = source.ops->read_block(source.arg, dest, off, size); + dest->source = source; + return result; +} + +void block_source_return_block(struct reftable_block_source source, + struct reftable_block *blockp) +{ + source.ops->return_block(source.arg, blockp); + blockp->data = NULL; + blockp->len = 0; + blockp->source.ops = NULL; + blockp->source.arg = NULL; +} + +void block_source_close(struct reftable_block_source *source) +{ + if (source->ops == NULL) { + return; + } + + source->ops->close(source->arg); + source->ops = NULL; +} + +static struct reftable_reader_offsets * +reader_offsets_for(struct reftable_reader *r, byte typ) +{ + switch (typ) { + case BLOCK_TYPE_REF: + return &r->ref_offsets; + case BLOCK_TYPE_LOG: + return &r->log_offsets; + case BLOCK_TYPE_OBJ: + return &r->obj_offsets; + } + abort(); +} + +static int reader_get_block(struct reftable_reader *r, + struct reftable_block *dest, uint64_t off, + uint32_t sz) +{ + if (off >= r->size) { + return 0; + } + + if (off + sz > r->size) { + sz = r->size - off; + } + + return block_source_read_block(r->source, dest, off, sz); +} + +void reader_return_block(struct reftable_reader *r, struct reftable_block *p) +{ + block_source_return_block(r->source, p); +} + +uint32_t reftable_reader_hash_id(struct reftable_reader *r) +{ + return r->hash_id; +} + +const char *reader_name(struct reftable_reader *r) +{ + return r->name; +} + +static int parse_footer(struct reftable_reader *r, byte *footer, byte *header) +{ + byte *f = footer; + int err = 0; + if (memcmp(f, "REFT", 4)) { + err = REFTABLE_FORMAT_ERROR; + goto exit; + } + f += 4; + + if (memcmp(footer, header, header_size(r->version))) { + err = REFTABLE_FORMAT_ERROR; + goto exit; + } + + f++; + r->block_size = get_be24(f); + + f += 3; + r->min_update_index = get_be64(f); + f += 8; + r->max_update_index = get_be64(f); + f += 8; + + if (r->version == 1) { + r->hash_id = SHA1_ID; + } else { + r->hash_id = get_be32(f); + switch (r->hash_id) { + case SHA1_ID: + break; + case SHA256_ID: + break; + default: + err = REFTABLE_FORMAT_ERROR; + goto exit; + } + f += 4; + } + + r->ref_offsets.index_offset = get_be64(f); + f += 8; + + r->obj_offsets.offset = get_be64(f); + f += 8; + + r->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1); + r->obj_offsets.offset >>= 5; + + r->obj_offsets.index_offset = get_be64(f); + f += 8; + r->log_offsets.offset = get_be64(f); + f += 8; + r->log_offsets.index_offset = get_be64(f); + f += 8; + + { + uint32_t computed_crc = crc32(0, footer, f - footer); + uint32_t file_crc = get_be32(f); + f += 4; + if (computed_crc != file_crc) { + err = REFTABLE_FORMAT_ERROR; + goto exit; + } + } + + { + byte first_block_typ = header[header_size(r->version)]; + r->ref_offsets.present = (first_block_typ == BLOCK_TYPE_REF); + r->ref_offsets.offset = 0; + r->log_offsets.present = (first_block_typ == BLOCK_TYPE_LOG || + r->log_offsets.offset > 0); + r->obj_offsets.present = r->obj_offsets.offset > 0; + } + err = 0; +exit: + return err; +} + +int init_reader(struct reftable_reader *r, struct reftable_block_source source, + const char *name) +{ + struct reftable_block footer = { 0 }; + struct reftable_block header = { 0 }; + int err = 0; + + memset(r, 0, sizeof(struct reftable_reader)); + + /* Need +1 to read type of first block. */ + err = block_source_read_block(source, &header, 0, header_size(2) + 1); + if (err != header_size(2) + 1) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + if (memcmp(header.data, "REFT", 4)) { + err = REFTABLE_FORMAT_ERROR; + goto exit; + } + r->version = header.data[4]; + if (r->version != 1 && r->version != 2) { + err = REFTABLE_FORMAT_ERROR; + goto exit; + } + + r->size = block_source_size(source) - footer_size(r->version); + r->source = source; + r->name = xstrdup(name); + r->hash_id = 0; + + err = block_source_read_block(source, &footer, r->size, + footer_size(r->version)); + if (err != footer_size(r->version)) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + err = parse_footer(r, footer.data, header.data); +exit: + block_source_return_block(r->source, &footer); + block_source_return_block(r->source, &header); + return err; +} + +struct table_iter { + struct reftable_reader *r; + byte typ; + uint64_t block_off; + struct block_iter bi; + bool finished; +}; + +static void table_iter_copy_from(struct table_iter *dest, + struct table_iter *src) +{ + dest->r = src->r; + dest->typ = src->typ; + dest->block_off = src->block_off; + dest->finished = src->finished; + block_iter_copy_from(&dest->bi, &src->bi); +} + +static int table_iter_next_in_block(struct table_iter *ti, struct record rec) +{ + int res = block_iter_next(&ti->bi, rec); + if (res == 0 && record_type(rec) == BLOCK_TYPE_REF) { + ((struct reftable_ref_record *)rec.data)->update_index += + ti->r->min_update_index; + } + + return res; +} + +static void table_iter_block_done(struct table_iter *ti) +{ + if (ti->bi.br == NULL) { + return; + } + reader_return_block(ti->r, &ti->bi.br->block); + FREE_AND_NULL(ti->bi.br); + + ti->bi.last_key.len = 0; + ti->bi.next_off = 0; +} + +static int32_t extract_block_size(byte *data, byte *typ, uint64_t off, + int version) +{ + int32_t result = 0; + + if (off == 0) { + data += header_size(version); + } + + *typ = data[0]; + if (is_block_type(*typ)) { + result = get_be24(data + 1); + } + return result; +} + +int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br, + uint64_t next_off, byte want_typ) +{ + int32_t guess_block_size = r->block_size ? r->block_size : + DEFAULT_BLOCK_SIZE; + struct reftable_block block = { 0 }; + byte block_typ = 0; + int err = 0; + uint32_t header_off = next_off ? 0 : header_size(r->version); + int32_t block_size = 0; + + if (next_off >= r->size) { + return 1; + } + + err = reader_get_block(r, &block, next_off, guess_block_size); + if (err < 0) { + return err; + } + + block_size = extract_block_size(block.data, &block_typ, next_off, + r->version); + if (block_size < 0) { + return block_size; + } + + if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) { + reader_return_block(r, &block); + return 1; + } + + if (block_size > guess_block_size) { + reader_return_block(r, &block); + err = reader_get_block(r, &block, next_off, block_size); + if (err < 0) { + return err; + } + } + + return block_reader_init(br, &block, header_off, r->block_size, + hash_size(r->hash_id)); +} + +static int table_iter_next_block(struct table_iter *dest, + struct table_iter *src) +{ + uint64_t next_block_off = src->block_off + src->bi.br->full_block_size; + struct block_reader br = { 0 }; + int err = 0; + + dest->r = src->r; + dest->typ = src->typ; + dest->block_off = next_block_off; + + err = reader_init_block_reader(src->r, &br, next_block_off, src->typ); + if (err > 0) { + dest->finished = true; + return 1; + } + if (err != 0) { + return err; + } + + { + struct block_reader *brp = + reftable_malloc(sizeof(struct block_reader)); + *brp = br; + + dest->finished = false; + block_reader_start(brp, &dest->bi); + } + return 0; +} + +static int table_iter_next(struct table_iter *ti, struct record rec) +{ + if (record_type(rec) != ti->typ) { + return REFTABLE_API_ERROR; + } + + while (true) { + struct table_iter next = { 0 }; + int err = 0; + if (ti->finished) { + return 1; + } + + err = table_iter_next_in_block(ti, rec); + if (err <= 0) { + return err; + } + + err = table_iter_next_block(&next, ti); + if (err != 0) { + ti->finished = true; + } + table_iter_block_done(ti); + if (err != 0) { + return err; + } + table_iter_copy_from(ti, &next); + block_iter_close(&next.bi); + } +} + +static int table_iter_next_void(void *ti, struct record rec) +{ + return table_iter_next((struct table_iter *)ti, rec); +} + +static void table_iter_close(void *p) +{ + struct table_iter *ti = (struct table_iter *)p; + table_iter_block_done(ti); + block_iter_close(&ti->bi); +} + +struct reftable_iterator_vtable table_iter_vtable = { + .next = &table_iter_next_void, + .close = &table_iter_close, +}; + +static void iterator_from_table_iter(struct reftable_iterator *it, + struct table_iter *ti) +{ + assert(it->ops == NULL); + it->iter_arg = ti; + it->ops = &table_iter_vtable; +} + +static int reader_table_iter_at(struct reftable_reader *r, + struct table_iter *ti, uint64_t off, byte typ) +{ + struct block_reader br = { 0 }; + struct block_reader *brp = NULL; + + int err = reader_init_block_reader(r, &br, off, typ); + if (err != 0) { + return err; + } + + brp = reftable_malloc(sizeof(struct block_reader)); + *brp = br; + ti->r = r; + ti->typ = block_reader_type(brp); + ti->block_off = off; + block_reader_start(brp, &ti->bi); + return 0; +} + +static int reader_start(struct reftable_reader *r, struct table_iter *ti, + byte typ, bool index) +{ + struct reftable_reader_offsets *offs = reader_offsets_for(r, typ); + uint64_t off = offs->offset; + if (index) { + off = offs->index_offset; + if (off == 0) { + return 1; + } + typ = BLOCK_TYPE_INDEX; + } + + return reader_table_iter_at(r, ti, off, typ); +} + +static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti, + struct record want) +{ + struct record rec = new_record(record_type(want)); + struct slice want_key = { 0 }; + struct slice got_key = { 0 }; + struct table_iter next = { 0 }; + int err = -1; + record_key(want, &want_key); + + while (true) { + err = table_iter_next_block(&next, ti); + if (err < 0) { + goto exit; + } + + if (err > 0) { + break; + } + + err = block_reader_first_key(next.bi.br, &got_key); + if (err < 0) { + goto exit; + } + { + int cmp = slice_compare(got_key, want_key); + if (cmp > 0) { + table_iter_block_done(&next); + break; + } + } + + table_iter_block_done(ti); + table_iter_copy_from(ti, &next); + } + + err = block_iter_seek(&ti->bi, want_key); + if (err < 0) { + goto exit; + } + err = 0; + +exit: + block_iter_close(&next.bi); + record_destroy(&rec); + slice_clear(&want_key); + slice_clear(&got_key); + return err; +} + +static int reader_seek_indexed(struct reftable_reader *r, + struct reftable_iterator *it, struct record rec) +{ + struct index_record want_index = { 0 }; + struct record want_index_rec = { 0 }; + struct index_record index_result = { 0 }; + struct record index_result_rec = { 0 }; + struct table_iter index_iter = { 0 }; + struct table_iter next = { 0 }; + int err = 0; + + record_key(rec, &want_index.last_key); + record_from_index(&want_index_rec, &want_index); + record_from_index(&index_result_rec, &index_result); + + err = reader_start(r, &index_iter, record_type(rec), true); + if (err < 0) { + goto exit; + } + + err = reader_seek_linear(r, &index_iter, want_index_rec); + while (true) { + err = table_iter_next(&index_iter, index_result_rec); + table_iter_block_done(&index_iter); + if (err != 0) { + goto exit; + } + + err = reader_table_iter_at(r, &next, index_result.offset, 0); + if (err != 0) { + goto exit; + } + + err = block_iter_seek(&next.bi, want_index.last_key); + if (err < 0) { + goto exit; + } + + if (next.typ == record_type(rec)) { + err = 0; + break; + } + + if (next.typ != BLOCK_TYPE_INDEX) { + err = REFTABLE_FORMAT_ERROR; + break; + } + + table_iter_copy_from(&index_iter, &next); + } + + if (err == 0) { + struct table_iter *malloced = + reftable_calloc(sizeof(struct table_iter)); + table_iter_copy_from(malloced, &next); + iterator_from_table_iter(it, malloced); + } +exit: + block_iter_close(&next.bi); + table_iter_close(&index_iter); + record_clear(want_index_rec); + record_clear(index_result_rec); + return err; +} + +static int reader_seek_internal(struct reftable_reader *r, + struct reftable_iterator *it, struct record rec) +{ + struct reftable_reader_offsets *offs = + reader_offsets_for(r, record_type(rec)); + uint64_t idx = offs->index_offset; + struct table_iter ti = { 0 }; + int err = 0; + if (idx > 0) { + return reader_seek_indexed(r, it, rec); + } + + err = reader_start(r, &ti, record_type(rec), false); + if (err < 0) { + return err; + } + err = reader_seek_linear(r, &ti, rec); + if (err < 0) { + return err; + } + + { + struct table_iter *p = + reftable_malloc(sizeof(struct table_iter)); + *p = ti; + iterator_from_table_iter(it, p); + } + + return 0; +} + +int reader_seek(struct reftable_reader *r, struct reftable_iterator *it, + struct record rec) +{ + byte typ = record_type(rec); + + struct reftable_reader_offsets *offs = reader_offsets_for(r, typ); + if (!offs->present) { + iterator_set_empty(it); + return 0; + } + + return reader_seek_internal(r, it, rec); +} + +int reftable_reader_seek_ref(struct reftable_reader *r, + struct reftable_iterator *it, const char *name) +{ + struct reftable_ref_record ref = { + .ref_name = (char *)name, + }; + struct record rec = { 0 }; + record_from_ref(&rec, &ref); + return reader_seek(r, it, rec); +} + +int reftable_reader_seek_log_at(struct reftable_reader *r, + struct reftable_iterator *it, const char *name, + uint64_t update_index) +{ + struct reftable_log_record log = { + .ref_name = (char *)name, + .update_index = update_index, + }; + struct record rec = { 0 }; + record_from_log(&rec, &log); + return reader_seek(r, it, rec); +} + +int reftable_reader_seek_log(struct reftable_reader *r, + struct reftable_iterator *it, const char *name) +{ + uint64_t max = ~((uint64_t)0); + return reftable_reader_seek_log_at(r, it, name, max); +} + +void reader_close(struct reftable_reader *r) +{ + block_source_close(&r->source); + FREE_AND_NULL(r->name); +} + +int reftable_new_reader(struct reftable_reader **p, + struct reftable_block_source src, char const *name) +{ + struct reftable_reader *rd = + reftable_calloc(sizeof(struct reftable_reader)); + int err = init_reader(rd, src, name); + if (err == 0) { + *p = rd; + } else { + block_source_close(&src); + reftable_free(rd); + } + return err; +} + +void reftable_reader_free(struct reftable_reader *r) +{ + reader_close(r); + reftable_free(r); +} + +static int reftable_reader_refs_for_indexed(struct reftable_reader *r, + struct reftable_iterator *it, + byte *oid) +{ + struct obj_record want = { + .hash_prefix = oid, + .hash_prefix_len = r->object_id_len, + }; + struct record want_rec = { 0 }; + struct reftable_iterator oit = { 0 }; + struct obj_record got = { 0 }; + struct record got_rec = { 0 }; + int err = 0; + + /* Look through the reverse index. */ + record_from_obj(&want_rec, &want); + err = reader_seek(r, &oit, want_rec); + if (err != 0) { + goto exit; + } + + /* read out the obj_record */ + record_from_obj(&got_rec, &got); + err = iterator_next(oit, got_rec); + if (err < 0) { + goto exit; + } + + if (err > 0 || + memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) { + /* didn't find it; return empty iterator */ + iterator_set_empty(it); + err = 0; + goto exit; + } + + { + struct indexed_table_ref_iter *itr = NULL; + err = new_indexed_table_ref_iter(&itr, r, oid, + hash_size(r->hash_id), + got.offsets, got.offset_len); + if (err < 0) { + goto exit; + } + got.offsets = NULL; + iterator_from_indexed_table_ref_iter(it, itr); + } + +exit: + reftable_iterator_destroy(&oit); + record_clear(got_rec); + return err; +} + +static int reftable_reader_refs_for_unindexed(struct reftable_reader *r, + struct reftable_iterator *it, + byte *oid, int oid_len) +{ + struct table_iter *ti = reftable_calloc(sizeof(struct table_iter)); + struct filtering_ref_iterator *filter = NULL; + int err = reader_start(r, ti, BLOCK_TYPE_REF, false); + if (err < 0) { + reftable_free(ti); + return err; + } + + filter = reftable_calloc(sizeof(struct filtering_ref_iterator)); + slice_resize(&filter->oid, oid_len); + memcpy(filter->oid.buf, oid, oid_len); + filter->r = r; + filter->double_check = false; + iterator_from_table_iter(&filter->it, ti); + + iterator_from_filtering_ref_iterator(it, filter); + return 0; +} + +int reftable_reader_refs_for(struct reftable_reader *r, + struct reftable_iterator *it, byte *oid, + int oid_len) +{ + if (r->obj_offsets.present) { + return reftable_reader_refs_for_indexed(r, it, oid); + } + return reftable_reader_refs_for_unindexed(r, it, oid, oid_len); +} + +uint64_t reftable_reader_max_update_index(struct reftable_reader *r) +{ + return r->max_update_index; +} + +uint64_t reftable_reader_min_update_index(struct reftable_reader *r) +{ + return r->min_update_index; +} diff --git a/reftable/reader.h b/reftable/reader.h new file mode 100644 index 00000000000..80d8a5adbaa --- /dev/null +++ b/reftable/reader.h @@ -0,0 +1,68 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef READER_H +#define READER_H + +#include "block.h" +#include "record.h" +#include "reftable.h" + +uint64_t block_source_size(struct reftable_block_source source); + +int block_source_read_block(struct reftable_block_source source, + struct reftable_block *dest, uint64_t off, + uint32_t size); +void block_source_return_block(struct reftable_block_source source, + struct reftable_block *ret); +void block_source_close(struct reftable_block_source *source); + +/* metadata for a block type */ +struct reftable_reader_offsets { + bool present; + uint64_t offset; + uint64_t index_offset; +}; + +/* The state for reading a reftable file. */ +struct reftable_reader { + /* for convience, associate a name with the instance. */ + char *name; + struct reftable_block_source source; + + /* Size of the file, excluding the footer. */ + uint64_t size; + + /* 'sha1' for SHA1, 's256' for SHA-256 */ + uint32_t hash_id; + + uint32_t block_size; + uint64_t min_update_index; + uint64_t max_update_index; + /* Length of the OID keys in the 'o' section */ + int object_id_len; + int version; + + struct reftable_reader_offsets ref_offsets; + struct reftable_reader_offsets obj_offsets; + struct reftable_reader_offsets log_offsets; +}; + +int init_reader(struct reftable_reader *r, struct reftable_block_source source, + const char *name); +int reader_seek(struct reftable_reader *r, struct reftable_iterator *it, + struct record rec); +void reader_close(struct reftable_reader *r); +const char *reader_name(struct reftable_reader *r); +void reader_return_block(struct reftable_reader *r, struct reftable_block *p); + +/* initialize a block reader to read from `r` */ +int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br, + uint64_t next_off, byte want_typ); + +#endif diff --git a/reftable/record.c b/reftable/record.c new file mode 100644 index 00000000000..8b9099b1966 --- /dev/null +++ b/reftable/record.c @@ -0,0 +1,1131 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +/* record.c - methods for different types of records. */ + +#include "record.h" + +#include "system.h" + +#include "constants.h" +#include "reftable.h" + +int get_var_int(uint64_t *dest, struct slice in) +{ + int ptr = 0; + uint64_t val; + + if (in.len == 0) { + return -1; + } + val = in.buf[ptr] & 0x7f; + + while (in.buf[ptr] & 0x80) { + ptr++; + if (ptr > in.len) { + return -1; + } + val = (val + 1) << 7 | (uint64_t)(in.buf[ptr] & 0x7f); + } + + *dest = val; + return ptr + 1; +} + +int put_var_int(struct slice dest, uint64_t val) +{ + byte buf[10] = { 0 }; + int i = 9; + buf[i] = (byte)(val & 0x7f); + i--; + while (true) { + val >>= 7; + if (!val) { + break; + } + val--; + buf[i] = 0x80 | (byte)(val & 0x7f); + i--; + } + + { + int n = sizeof(buf) - i - 1; + if (dest.len < n) { + return -1; + } + memcpy(dest.buf, &buf[i + 1], n); + return n; + } +} + +int is_block_type(byte typ) +{ + switch (typ) { + case BLOCK_TYPE_REF: + case BLOCK_TYPE_LOG: + case BLOCK_TYPE_OBJ: + case BLOCK_TYPE_INDEX: + return true; + } + return false; +} + +static int decode_string(struct slice *dest, struct slice in) +{ + int start_len = in.len; + uint64_t tsize = 0; + int n = get_var_int(&tsize, in); + if (n <= 0) { + return -1; + } + slice_consume(&in, n); + if (in.len < tsize) { + return -1; + } + + slice_resize(dest, tsize + 1); + dest->buf[tsize] = 0; + memcpy(dest->buf, in.buf, tsize); + slice_consume(&in, tsize); + + return start_len - in.len; +} + +static int encode_string(char *str, struct slice s) +{ + struct slice start = s; + int l = strlen(str); + int n = put_var_int(s, l); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + if (s.len < l) { + return -1; + } + memcpy(s.buf, str, l); + slice_consume(&s, l); + + return start.len - s.len; +} + +int encode_key(bool *restart, struct slice dest, struct slice prev_key, + struct slice key, byte extra) +{ + struct slice start = dest; + int prefix_len = common_prefix_size(prev_key, key); + uint64_t suffix_len = key.len - prefix_len; + int n = put_var_int(dest, (uint64_t)prefix_len); + if (n < 0) { + return -1; + } + slice_consume(&dest, n); + + *restart = (prefix_len == 0); + + n = put_var_int(dest, suffix_len << 3 | (uint64_t)extra); + if (n < 0) { + return -1; + } + slice_consume(&dest, n); + + if (dest.len < suffix_len) { + return -1; + } + memcpy(dest.buf, key.buf + prefix_len, suffix_len); + slice_consume(&dest, suffix_len); + + return start.len - dest.len; +} + +int decode_key(struct slice *key, byte *extra, struct slice last_key, + struct slice in) +{ + int start_len = in.len; + uint64_t prefix_len = 0; + uint64_t suffix_len = 0; + int n = get_var_int(&prefix_len, in); + if (n < 0) { + return -1; + } + slice_consume(&in, n); + + if (prefix_len > last_key.len) { + return -1; + } + + n = get_var_int(&suffix_len, in); + if (n <= 0) { + return -1; + } + slice_consume(&in, n); + + *extra = (byte)(suffix_len & 0x7); + suffix_len >>= 3; + + if (in.len < suffix_len) { + return -1; + } + + slice_resize(key, suffix_len + prefix_len); + memcpy(key->buf, last_key.buf, prefix_len); + + memcpy(key->buf + prefix_len, in.buf, suffix_len); + slice_consume(&in, suffix_len); + + return start_len - in.len; +} + +static void reftable_ref_record_key(const void *r, struct slice *dest) +{ + const struct reftable_ref_record *rec = + (const struct reftable_ref_record *)r; + slice_set_string(dest, rec->ref_name); +} + +static void reftable_ref_record_copy_from(void *rec, const void *src_rec, + int hash_size) +{ + struct reftable_ref_record *ref = (struct reftable_ref_record *)rec; + struct reftable_ref_record *src = (struct reftable_ref_record *)src_rec; + assert(hash_size > 0); + + /* This is simple and correct, but we could probably reuse the hash + fields. */ + reftable_ref_record_clear(ref); + if (src->ref_name != NULL) { + ref->ref_name = xstrdup(src->ref_name); + } + + if (src->target != NULL) { + ref->target = xstrdup(src->target); + } + + if (src->target_value != NULL) { + ref->target_value = reftable_malloc(hash_size); + memcpy(ref->target_value, src->target_value, hash_size); + } + + if (src->value != NULL) { + ref->value = reftable_malloc(hash_size); + memcpy(ref->value, src->value, hash_size); + } + ref->update_index = src->update_index; +} + +static char hexdigit(int c) +{ + if (c <= 9) { + return '0' + c; + } + return 'a' + (c - 10); +} + +static void hex_format(char *dest, byte *src, int hash_size) +{ + assert(hash_size > 0); + if (src != NULL) { + int i = 0; + for (i = 0; i < hash_size; i++) { + dest[2 * i] = hexdigit(src[i] >> 4); + dest[2 * i + 1] = hexdigit(src[i] & 0xf); + } + dest[2 * hash_size] = 0; + } +} + +void reftable_ref_record_print(struct reftable_ref_record *ref, + uint32_t hash_id) +{ + char hex[SHA256_SIZE + 1] = { 0 }; + printf("ref{%s(%" PRIu64 ") ", ref->ref_name, ref->update_index); + if (ref->value != NULL) { + hex_format(hex, ref->value, hash_size(hash_id)); + printf("%s", hex); + } + if (ref->target_value != NULL) { + hex_format(hex, ref->target_value, hash_size(hash_id)); + printf(" (T %s)", hex); + } + if (ref->target != NULL) { + printf("=> %s", ref->target); + } + printf("}\n"); +} + +static void reftable_ref_record_clear_void(void *rec) +{ + reftable_ref_record_clear((struct reftable_ref_record *)rec); +} + +void reftable_ref_record_clear(struct reftable_ref_record *ref) +{ + reftable_free(ref->ref_name); + reftable_free(ref->target); + reftable_free(ref->target_value); + reftable_free(ref->value); + memset(ref, 0, sizeof(struct reftable_ref_record)); +} + +static byte reftable_ref_record_val_type(const void *rec) +{ + const struct reftable_ref_record *r = + (const struct reftable_ref_record *)rec; + if (r->value != NULL) { + if (r->target_value != NULL) { + return 2; + } else { + return 1; + } + } else if (r->target != NULL) { + return 3; + } + return 0; +} + +static int reftable_ref_record_encode(const void *rec, struct slice s, + int hash_size) +{ + const struct reftable_ref_record *r = + (const struct reftable_ref_record *)rec; + struct slice start = s; + int n = put_var_int(s, r->update_index); + assert(hash_size > 0); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + + if (r->value != NULL) { + if (s.len < hash_size) { + return -1; + } + memcpy(s.buf, r->value, hash_size); + slice_consume(&s, hash_size); + } + + if (r->target_value != NULL) { + if (s.len < hash_size) { + return -1; + } + memcpy(s.buf, r->target_value, hash_size); + slice_consume(&s, hash_size); + } + + if (r->target != NULL) { + int n = encode_string(r->target, s); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + } + + return start.len - s.len; +} + +static int reftable_ref_record_decode(void *rec, struct slice key, + byte val_type, struct slice in, + int hash_size) +{ + struct reftable_ref_record *r = (struct reftable_ref_record *)rec; + struct slice start = in; + bool seen_value = false; + bool seen_target_value = false; + bool seen_target = false; + + int n = get_var_int(&r->update_index, in); + if (n < 0) { + return n; + } + assert(hash_size > 0); + + slice_consume(&in, n); + + r->ref_name = reftable_realloc(r->ref_name, key.len + 1); + memcpy(r->ref_name, key.buf, key.len); + r->ref_name[key.len] = 0; + + switch (val_type) { + case 1: + case 2: + if (in.len < hash_size) { + return -1; + } + + if (r->value == NULL) { + r->value = reftable_malloc(hash_size); + } + seen_value = true; + memcpy(r->value, in.buf, hash_size); + slice_consume(&in, hash_size); + if (val_type == 1) { + break; + } + if (r->target_value == NULL) { + r->target_value = reftable_malloc(hash_size); + } + seen_target_value = true; + memcpy(r->target_value, in.buf, hash_size); + slice_consume(&in, hash_size); + break; + case 3: { + struct slice dest = { 0 }; + int n = decode_string(&dest, in); + if (n < 0) { + return -1; + } + slice_consume(&in, n); + seen_target = true; + r->target = (char *)slice_as_string(&dest); + } break; + + case 0: + break; + default: + abort(); + break; + } + + if (!seen_target && r->target != NULL) { + FREE_AND_NULL(r->target); + } + if (!seen_target_value && r->target_value != NULL) { + FREE_AND_NULL(r->target_value); + } + if (!seen_value && r->value != NULL) { + FREE_AND_NULL(r->value); + } + + return start.len - in.len; +} + +static bool reftable_ref_record_is_deletion_void(const void *p) +{ + return reftable_ref_record_is_deletion( + (const struct reftable_ref_record *)p); +} + +struct record_vtable reftable_ref_record_vtable = { + .key = &reftable_ref_record_key, + .type = BLOCK_TYPE_REF, + .copy_from = &reftable_ref_record_copy_from, + .val_type = &reftable_ref_record_val_type, + .encode = &reftable_ref_record_encode, + .decode = &reftable_ref_record_decode, + .clear = &reftable_ref_record_clear_void, + .is_deletion = &reftable_ref_record_is_deletion_void, +}; + +static void obj_record_key(const void *r, struct slice *dest) +{ + const struct obj_record *rec = (const struct obj_record *)r; + slice_resize(dest, rec->hash_prefix_len); + memcpy(dest->buf, rec->hash_prefix, rec->hash_prefix_len); +} + +static void obj_record_copy_from(void *rec, const void *src_rec, int hash_size) +{ + struct obj_record *ref = (struct obj_record *)rec; + const struct obj_record *src = (const struct obj_record *)src_rec; + + *ref = *src; + ref->hash_prefix = reftable_malloc(ref->hash_prefix_len); + memcpy(ref->hash_prefix, src->hash_prefix, ref->hash_prefix_len); + + { + int olen = ref->offset_len * sizeof(uint64_t); + ref->offsets = reftable_malloc(olen); + memcpy(ref->offsets, src->offsets, olen); + } +} + +static void obj_record_clear(void *rec) +{ + struct obj_record *ref = (struct obj_record *)rec; + FREE_AND_NULL(ref->hash_prefix); + FREE_AND_NULL(ref->offsets); + memset(ref, 0, sizeof(struct obj_record)); +} + +static byte obj_record_val_type(const void *rec) +{ + struct obj_record *r = (struct obj_record *)rec; + if (r->offset_len > 0 && r->offset_len < 8) { + return r->offset_len; + } + return 0; +} + +static int obj_record_encode(const void *rec, struct slice s, int hash_size) +{ + struct obj_record *r = (struct obj_record *)rec; + struct slice start = s; + int n = 0; + if (r->offset_len == 0 || r->offset_len >= 8) { + n = put_var_int(s, r->offset_len); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + } + if (r->offset_len == 0) { + return start.len - s.len; + } + n = put_var_int(s, r->offsets[0]); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + + { + uint64_t last = r->offsets[0]; + int i = 0; + for (i = 1; i < r->offset_len; i++) { + int n = put_var_int(s, r->offsets[i] - last); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + last = r->offsets[i]; + } + } + return start.len - s.len; +} + +static int obj_record_decode(void *rec, struct slice key, byte val_type, + struct slice in, int hash_size) +{ + struct slice start = in; + struct obj_record *r = (struct obj_record *)rec; + uint64_t count = val_type; + int n = 0; + r->hash_prefix = reftable_malloc(key.len); + memcpy(r->hash_prefix, key.buf, key.len); + r->hash_prefix_len = key.len; + + if (val_type == 0) { + n = get_var_int(&count, in); + if (n < 0) { + return n; + } + + slice_consume(&in, n); + } + + r->offsets = NULL; + r->offset_len = 0; + if (count == 0) { + return start.len - in.len; + } + + r->offsets = reftable_malloc(count * sizeof(uint64_t)); + r->offset_len = count; + + n = get_var_int(&r->offsets[0], in); + if (n < 0) { + return n; + } + slice_consume(&in, n); + + { + uint64_t last = r->offsets[0]; + int j = 1; + while (j < count) { + uint64_t delta = 0; + int n = get_var_int(&delta, in); + if (n < 0) { + return n; + } + slice_consume(&in, n); + + last = r->offsets[j] = (delta + last); + j++; + } + } + return start.len - in.len; +} + +static bool not_a_deletion(const void *p) +{ + return false; +} + +struct record_vtable obj_record_vtable = { + .key = &obj_record_key, + .type = BLOCK_TYPE_OBJ, + .copy_from = &obj_record_copy_from, + .val_type = &obj_record_val_type, + .encode = &obj_record_encode, + .decode = &obj_record_decode, + .clear = &obj_record_clear, + .is_deletion = not_a_deletion, +}; + +void reftable_log_record_print(struct reftable_log_record *log, + uint32_t hash_id) +{ + char hex[SHA256_SIZE + 1] = { 0 }; + + printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n", log->ref_name, + log->update_index, log->name, log->email, log->time, + log->tz_offset); + hex_format(hex, log->old_hash, hash_size(hash_id)); + printf("%s => ", hex); + hex_format(hex, log->new_hash, hash_size(hash_id)); + printf("%s\n\n%s\n}\n", hex, log->message); +} + +static void reftable_log_record_key(const void *r, struct slice *dest) +{ + const struct reftable_log_record *rec = + (const struct reftable_log_record *)r; + int len = strlen(rec->ref_name); + uint64_t ts = 0; + slice_resize(dest, len + 9); + memcpy(dest->buf, rec->ref_name, len + 1); + ts = (~ts) - rec->update_index; + put_be64(dest->buf + 1 + len, ts); +} + +static void reftable_log_record_copy_from(void *rec, const void *src_rec, + int hash_size) +{ + struct reftable_log_record *dst = (struct reftable_log_record *)rec; + const struct reftable_log_record *src = + (const struct reftable_log_record *)src_rec; + + *dst = *src; + if (dst->ref_name != NULL) { + dst->ref_name = xstrdup(dst->ref_name); + } + if (dst->email != NULL) { + dst->email = xstrdup(dst->email); + } + if (dst->name != NULL) { + dst->name = xstrdup(dst->name); + } + if (dst->message != NULL) { + dst->message = xstrdup(dst->message); + } + + if (dst->new_hash != NULL) { + dst->new_hash = reftable_malloc(hash_size); + memcpy(dst->new_hash, src->new_hash, hash_size); + } + if (dst->old_hash != NULL) { + dst->old_hash = reftable_malloc(hash_size); + memcpy(dst->old_hash, src->old_hash, hash_size); + } +} + +static void reftable_log_record_clear_void(void *rec) +{ + struct reftable_log_record *r = (struct reftable_log_record *)rec; + reftable_log_record_clear(r); +} + +void reftable_log_record_clear(struct reftable_log_record *r) +{ + reftable_free(r->ref_name); + reftable_free(r->new_hash); + reftable_free(r->old_hash); + reftable_free(r->name); + reftable_free(r->email); + reftable_free(r->message); + memset(r, 0, sizeof(struct reftable_log_record)); +} + +static byte reftable_log_record_val_type(const void *rec) +{ + const struct reftable_log_record *log = + (const struct reftable_log_record *)rec; + + return reftable_log_record_is_deletion(log) ? 0 : 1; +} + +static byte zero[SHA256_SIZE] = { 0 }; + +static int reftable_log_record_encode(const void *rec, struct slice s, + int hash_size) +{ + struct reftable_log_record *r = (struct reftable_log_record *)rec; + struct slice start = s; + int n = 0; + byte *oldh = r->old_hash; + byte *newh = r->new_hash; + if (reftable_log_record_is_deletion(r)) { + return 0; + } + + if (oldh == NULL) { + oldh = zero; + } + if (newh == NULL) { + newh = zero; + } + + if (s.len < 2 * hash_size) { + return -1; + } + + memcpy(s.buf, oldh, hash_size); + memcpy(s.buf + hash_size, newh, hash_size); + slice_consume(&s, 2 * hash_size); + + n = encode_string(r->name ? r->name : "", s); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + + n = encode_string(r->email ? r->email : "", s); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + + n = put_var_int(s, r->time); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + + if (s.len < 2) { + return -1; + } + + put_be16(s.buf, r->tz_offset); + slice_consume(&s, 2); + + n = encode_string(r->message ? r->message : "", s); + if (n < 0) { + return -1; + } + slice_consume(&s, n); + + return start.len - s.len; +} + +static int reftable_log_record_decode(void *rec, struct slice key, + byte val_type, struct slice in, + int hash_size) +{ + struct slice start = in; + struct reftable_log_record *r = (struct reftable_log_record *)rec; + uint64_t max = 0; + uint64_t ts = 0; + struct slice dest = { 0 }; + int n; + + if (key.len <= 9 || key.buf[key.len - 9] != 0) { + return REFTABLE_FORMAT_ERROR; + } + + r->ref_name = reftable_realloc(r->ref_name, key.len - 8); + memcpy(r->ref_name, key.buf, key.len - 8); + ts = get_be64(key.buf + key.len - 8); + + r->update_index = (~max) - ts; + + if (val_type == 0) { + return 0; + } + + if (in.len < 2 * hash_size) { + return REFTABLE_FORMAT_ERROR; + } + + r->old_hash = reftable_realloc(r->old_hash, hash_size); + r->new_hash = reftable_realloc(r->new_hash, hash_size); + + memcpy(r->old_hash, in.buf, hash_size); + memcpy(r->new_hash, in.buf + hash_size, hash_size); + + slice_consume(&in, 2 * hash_size); + + n = decode_string(&dest, in); + if (n < 0) { + goto error; + } + slice_consume(&in, n); + + r->name = reftable_realloc(r->name, dest.len + 1); + memcpy(r->name, dest.buf, dest.len); + r->name[dest.len] = 0; + + slice_resize(&dest, 0); + n = decode_string(&dest, in); + if (n < 0) { + goto error; + } + slice_consume(&in, n); + + r->email = reftable_realloc(r->email, dest.len + 1); + memcpy(r->email, dest.buf, dest.len); + r->email[dest.len] = 0; + + ts = 0; + n = get_var_int(&ts, in); + if (n < 0) { + goto error; + } + slice_consume(&in, n); + r->time = ts; + if (in.len < 2) { + goto error; + } + + r->tz_offset = get_be16(in.buf); + slice_consume(&in, 2); + + slice_resize(&dest, 0); + n = decode_string(&dest, in); + if (n < 0) { + goto error; + } + slice_consume(&in, n); + + r->message = reftable_realloc(r->message, dest.len + 1); + memcpy(r->message, dest.buf, dest.len); + r->message[dest.len] = 0; + + slice_clear(&dest); + return start.len - in.len; + +error: + slice_clear(&dest); + return REFTABLE_FORMAT_ERROR; +} + +static bool null_streq(char *a, char *b) +{ + char *empty = ""; + if (a == NULL) { + a = empty; + } + if (b == NULL) { + b = empty; + } + return 0 == strcmp(a, b); +} + +static bool zero_hash_eq(byte *a, byte *b, int sz) +{ + if (a == NULL) { + a = zero; + } + if (b == NULL) { + b = zero; + } + return !memcmp(a, b, sz); +} + +bool reftable_log_record_equal(struct reftable_log_record *a, + struct reftable_log_record *b, int hash_size) +{ + return null_streq(a->name, b->name) && null_streq(a->email, b->email) && + null_streq(a->message, b->message) && + zero_hash_eq(a->old_hash, b->old_hash, hash_size) && + zero_hash_eq(a->new_hash, b->new_hash, hash_size) && + a->time == b->time && a->tz_offset == b->tz_offset && + a->update_index == b->update_index; +} + +static bool reftable_log_record_is_deletion_void(const void *p) +{ + return reftable_log_record_is_deletion( + (const struct reftable_log_record *)p); +} + +struct record_vtable reftable_log_record_vtable = { + .key = &reftable_log_record_key, + .type = BLOCK_TYPE_LOG, + .copy_from = &reftable_log_record_copy_from, + .val_type = &reftable_log_record_val_type, + .encode = &reftable_log_record_encode, + .decode = &reftable_log_record_decode, + .clear = &reftable_log_record_clear_void, + .is_deletion = &reftable_log_record_is_deletion_void, +}; + +struct record new_record(byte typ) +{ + struct record rec = { NULL }; + switch (typ) { + case BLOCK_TYPE_REF: { + struct reftable_ref_record *r = + reftable_calloc(sizeof(struct reftable_ref_record)); + record_from_ref(&rec, r); + return rec; + } + + case BLOCK_TYPE_OBJ: { + struct obj_record *r = + reftable_calloc(sizeof(struct obj_record)); + record_from_obj(&rec, r); + return rec; + } + case BLOCK_TYPE_LOG: { + struct reftable_log_record *r = + reftable_calloc(sizeof(struct reftable_log_record)); + record_from_log(&rec, r); + return rec; + } + case BLOCK_TYPE_INDEX: { + struct index_record *r = + reftable_calloc(sizeof(struct index_record)); + record_from_index(&rec, r); + return rec; + } + } + abort(); + return rec; +} + +void *record_yield(struct record *rec) +{ + void *p = rec->data; + rec->data = NULL; + return p; +} + +void record_destroy(struct record *rec) +{ + record_clear(*rec); + reftable_free(record_yield(rec)); +} + +static void index_record_key(const void *r, struct slice *dest) +{ + struct index_record *rec = (struct index_record *)r; + slice_copy(dest, rec->last_key); +} + +static void index_record_copy_from(void *rec, const void *src_rec, + int hash_size) +{ + struct index_record *dst = (struct index_record *)rec; + struct index_record *src = (struct index_record *)src_rec; + + slice_copy(&dst->last_key, src->last_key); + dst->offset = src->offset; +} + +static void index_record_clear(void *rec) +{ + struct index_record *idx = (struct index_record *)rec; + slice_clear(&idx->last_key); +} + +static byte index_record_val_type(const void *rec) +{ + return 0; +} + +static int index_record_encode(const void *rec, struct slice out, int hash_size) +{ + const struct index_record *r = (const struct index_record *)rec; + struct slice start = out; + + int n = put_var_int(out, r->offset); + if (n < 0) { + return n; + } + + slice_consume(&out, n); + + return start.len - out.len; +} + +static int index_record_decode(void *rec, struct slice key, byte val_type, + struct slice in, int hash_size) +{ + struct slice start = in; + struct index_record *r = (struct index_record *)rec; + int n = 0; + + slice_copy(&r->last_key, key); + + n = get_var_int(&r->offset, in); + if (n < 0) { + return n; + } + + slice_consume(&in, n); + return start.len - in.len; +} + +struct record_vtable index_record_vtable = { + .key = &index_record_key, + .type = BLOCK_TYPE_INDEX, + .copy_from = &index_record_copy_from, + .val_type = &index_record_val_type, + .encode = &index_record_encode, + .decode = &index_record_decode, + .clear = &index_record_clear, + .is_deletion = ¬_a_deletion, +}; + +void record_key(struct record rec, struct slice *dest) +{ + rec.ops->key(rec.data, dest); +} + +byte record_type(struct record rec) +{ + return rec.ops->type; +} + +int record_encode(struct record rec, struct slice dest, int hash_size) +{ + return rec.ops->encode(rec.data, dest, hash_size); +} + +void record_copy_from(struct record rec, struct record src, int hash_size) +{ + assert(src.ops->type == rec.ops->type); + + rec.ops->copy_from(rec.data, src.data, hash_size); +} + +byte record_val_type(struct record rec) +{ + return rec.ops->val_type(rec.data); +} + +int record_decode(struct record rec, struct slice key, byte extra, + struct slice src, int hash_size) +{ + return rec.ops->decode(rec.data, key, extra, src, hash_size); +} + +void record_clear(struct record rec) +{ + rec.ops->clear(rec.data); +} + +bool record_is_deletion(struct record rec) +{ + return rec.ops->is_deletion(rec.data); +} + +void record_from_ref(struct record *rec, struct reftable_ref_record *ref_rec) +{ + assert(rec->ops == NULL); + rec->data = ref_rec; + rec->ops = &reftable_ref_record_vtable; +} + +void record_from_obj(struct record *rec, struct obj_record *obj_rec) +{ + assert(rec->ops == NULL); + rec->data = obj_rec; + rec->ops = &obj_record_vtable; +} + +void record_from_index(struct record *rec, struct index_record *index_rec) +{ + assert(rec->ops == NULL); + rec->data = index_rec; + rec->ops = &index_record_vtable; +} + +void record_from_log(struct record *rec, struct reftable_log_record *log_rec) +{ + assert(rec->ops == NULL); + rec->data = log_rec; + rec->ops = &reftable_log_record_vtable; +} + +struct reftable_ref_record *record_as_ref(struct record rec) +{ + assert(record_type(rec) == BLOCK_TYPE_REF); + return (struct reftable_ref_record *)rec.data; +} + +struct reftable_log_record *record_as_log(struct record rec) +{ + assert(record_type(rec) == BLOCK_TYPE_LOG); + return (struct reftable_log_record *)rec.data; +} + +static bool hash_equal(byte *a, byte *b, int hash_size) +{ + if (a != NULL && b != NULL) { + return !memcmp(a, b, hash_size); + } + + return a == b; +} + +static bool str_equal(char *a, char *b) +{ + if (a != NULL && b != NULL) { + return 0 == strcmp(a, b); + } + + return a == b; +} + +bool reftable_ref_record_equal(struct reftable_ref_record *a, + struct reftable_ref_record *b, int hash_size) +{ + assert(hash_size > 0); + return 0 == strcmp(a->ref_name, b->ref_name) && + a->update_index == b->update_index && + hash_equal(a->value, b->value, hash_size) && + hash_equal(a->target_value, b->target_value, hash_size) && + str_equal(a->target, b->target); +} + +int reftable_ref_record_compare_name(const void *a, const void *b) +{ + return strcmp(((struct reftable_ref_record *)a)->ref_name, + ((struct reftable_ref_record *)b)->ref_name); +} + +bool reftable_ref_record_is_deletion(const struct reftable_ref_record *ref) +{ + return ref->value == NULL && ref->target == NULL && + ref->target_value == NULL; +} + +int reftable_log_record_compare_key(const void *a, const void *b) +{ + struct reftable_log_record *la = (struct reftable_log_record *)a; + struct reftable_log_record *lb = (struct reftable_log_record *)b; + + int cmp = strcmp(la->ref_name, lb->ref_name); + if (cmp) { + return cmp; + } + if (la->update_index > lb->update_index) { + return -1; + } + return (la->update_index < lb->update_index) ? 1 : 0; +} + +bool reftable_log_record_is_deletion(const struct reftable_log_record *log) +{ + return (log->new_hash == NULL && log->old_hash == NULL && + log->name == NULL && log->email == NULL && + log->message == NULL && log->time == 0 && log->tz_offset == 0 && + log->message == NULL); +} + +int hash_size(uint32_t id) +{ + switch (id) { + case 0: + case SHA1_ID: + return SHA1_SIZE; + case SHA256_ID: + return SHA256_SIZE; + } + abort(); +} diff --git a/reftable/record.h b/reftable/record.h new file mode 100644 index 00000000000..011d9bc56fc --- /dev/null +++ b/reftable/record.h @@ -0,0 +1,121 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef RECORD_H +#define RECORD_H + +#include "reftable.h" +#include "slice.h" + +/* utilities for de/encoding varints */ + +int get_var_int(uint64_t *dest, struct slice in); +int put_var_int(struct slice dest, uint64_t val); + +/* Methods for records. */ +struct record_vtable { + /* encode the key of to a byte slice. */ + void (*key)(const void *rec, struct slice *dest); + + /* The record type of ('r' for ref). */ + byte type; + + void (*copy_from)(void *dest, const void *src, int hash_size); + + /* a value of [0..7], indicating record subvariants (eg. ref vs. symref + * vs ref deletion) */ + byte (*val_type)(const void *rec); + + /* encodes rec into dest, returning how much space was used. */ + int (*encode)(const void *rec, struct slice dest, int hash_size); + + /* decode data from `src` into the record. */ + int (*decode)(void *rec, struct slice key, byte extra, struct slice src, + int hash_size); + + /* deallocate and null the record. */ + void (*clear)(void *rec); + + /* is this a tombstone? */ + bool (*is_deletion)(const void *rec); +}; + +/* record is a generic wrapper for different types of records. */ +struct record { + void *data; + struct record_vtable *ops; +}; + +/* returns true for recognized block types. Block start with the block type. */ +int is_block_type(byte typ); + +/* creates a malloced record of the given type. Dispose with record_destroy */ +struct record new_record(byte typ); + +extern struct record_vtable reftable_ref_record_vtable; + +/* Encode `key` into `dest`. Sets `restart` to indicate a restart. Returns + number of bytes written. */ +int encode_key(bool *restart, struct slice dest, struct slice prev_key, + struct slice key, byte extra); + +/* Decode into `key` and `extra` from `in` */ +int decode_key(struct slice *key, byte *extra, struct slice last_key, + struct slice in); + +/* index_record are used internally to speed up lookups. */ +struct index_record { + uint64_t offset; /* Offset of block */ + struct slice last_key; /* Last key of the block. */ +}; + +/* obj_record stores an object ID => ref mapping. */ +struct obj_record { + byte *hash_prefix; /* leading bytes of the object ID */ + int hash_prefix_len; /* number of leading bytes. Constant + * across a single table. */ + uint64_t *offsets; /* a vector of file offsets. */ + int offset_len; +}; + +/* see struct record_vtable */ + +void record_key(struct record rec, struct slice *dest); +byte record_type(struct record rec); +void record_copy_from(struct record rec, struct record src, int hash_size); +byte record_val_type(struct record rec); +int record_encode(struct record rec, struct slice dest, int hash_size); +int record_decode(struct record rec, struct slice key, byte extra, + struct slice src, int hash_size); +bool record_is_deletion(struct record rec); + +/* zeroes out the embedded record */ +void record_clear(struct record rec); + +/* clear out the record, yielding the record data that was encapsulated. */ +void *record_yield(struct record *rec); + +/* clear and deallocate embedded record, and zero `rec`. */ +void record_destroy(struct record *rec); + +/* initialize generic records from concrete records. The generic record should + * be zeroed out. */ +void record_from_obj(struct record *rec, struct obj_record *objrec); +void record_from_index(struct record *rec, struct index_record *idxrec); +void record_from_ref(struct record *rec, struct reftable_ref_record *refrec); +void record_from_log(struct record *rec, struct reftable_log_record *logrec); +struct reftable_ref_record *record_as_ref(struct record ref); +struct reftable_log_record *record_as_log(struct record ref); + +/* for qsort. */ +int reftable_ref_record_compare_name(const void *a, const void *b); + +/* for qsort. */ +int reftable_log_record_compare_key(const void *a, const void *b); + +#endif diff --git a/reftable/refname.c b/reftable/refname.c new file mode 100644 index 00000000000..c92488924ac --- /dev/null +++ b/reftable/refname.c @@ -0,0 +1,215 @@ +/* + Copyright 2020 Google LLC + + Use of this source code is governed by a BSD-style + license that can be found in the LICENSE file or at + https://developers.google.com/open-source/licenses/bsd +*/ + +#include "system.h" +#include "reftable.h" +#include "basics.h" +#include "refname.h" +#include "slice.h" + +struct find_arg { + char **names; + const char *want; +}; + +static int find_name(size_t k, void *arg) +{ + struct find_arg *f_arg = (struct find_arg *)arg; + + return strcmp(f_arg->names[k], f_arg->want) >= 0; +} + +int modification_has_ref(struct modification *mod, const char *name) +{ + struct reftable_ref_record ref = { 0 }; + int err = 0; + + if (mod->add_len > 0) { + struct find_arg arg = { + .names = mod->add, + .want = name, + }; + int idx = binsearch(mod->add_len, find_name, &arg); + if (idx < mod->add_len && !strcmp(mod->add[idx], name)) { + return 0; + } + } + + if (mod->del_len > 0) { + struct find_arg arg = { + .names = mod->del, + .want = name, + }; + int idx = binsearch(mod->del_len, find_name, &arg); + if (idx < mod->del_len && !strcmp(mod->del[idx], name)) { + return 1; + } + } + + err = reftable_table_read_ref(mod->tab, name, &ref); + reftable_ref_record_clear(&ref); + return err; +} + +static void modification_clear(struct modification *mod) +{ + FREE_AND_NULL(mod->add); + FREE_AND_NULL(mod->del); + mod->add_len = 0; + mod->del_len = 0; +} + +int modification_has_ref_with_prefix(struct modification *mod, + const char *prefix) +{ + struct reftable_iterator it = { NULL }; + struct reftable_ref_record ref = { NULL }; + int err = 0; + + if (mod->add_len > 0) { + struct find_arg arg = { + .names = mod->add, + .want = prefix, + }; + int idx = binsearch(mod->add_len, find_name, &arg); + if (idx < mod->add_len && + !strncmp(prefix, mod->add[idx], strlen(prefix))) { + goto exit; + } + } + + err = reftable_table_seek_ref(mod->tab, &it, prefix); + if (err) { + goto exit; + } + + while (true) { + err = reftable_iterator_next_ref(it, &ref); + if (err) { + goto exit; + } + + if (mod->del_len > 0) { + struct find_arg arg = { + .names = mod->del, + .want = ref.ref_name, + }; + int idx = binsearch(mod->del_len, find_name, &arg); + if (idx < mod->del_len && + !strcmp(ref.ref_name, mod->del[idx])) { + continue; + } + } + + if (strncmp(ref.ref_name, prefix, strlen(prefix))) { + err = 1; + goto exit; + } + err = 0; + goto exit; + } + +exit: + reftable_ref_record_clear(&ref); + reftable_iterator_destroy(&it); + return err; +} + +int validate_ref_name(const char *name) +{ + while (true) { + char *next = strchr(name, '/'); + if (!*name) { + return REFTABLE_REFNAME_ERROR; + } + if (!next) { + return 0; + } + if (next - name == 0 || (next - name == 1 && *name == '.') || + (next - name == 2 && name[0] == '.' && name[1] == '.')) + return REFTABLE_REFNAME_ERROR; + name = next + 1; + } + return 0; +} + +int validate_ref_record_addition(struct reftable_table tab, + struct reftable_ref_record *recs, size_t sz) +{ + struct modification mod = { + .tab = tab, + .add = reftable_calloc(sizeof(char *) * sz), + .del = reftable_calloc(sizeof(char *) * sz), + }; + int i = 0; + int err = 0; + for (; i < sz; i++) { + if (reftable_ref_record_is_deletion(&recs[i])) { + mod.del[mod.del_len++] = recs[i].ref_name; + } else { + mod.add[mod.add_len++] = recs[i].ref_name; + } + } + + err = modification_validate(&mod); + modification_clear(&mod); + return err; +} + +static void slice_trim_component(struct slice *sl) +{ + while (sl->len > 0) { + bool is_slash = (sl->buf[sl->len - 1] == '/'); + sl->len--; + if (is_slash) + break; + } +} + +int modification_validate(struct modification *mod) +{ + struct slice slashed = { 0 }; + int err = 0; + int i = 0; + for (; i < mod->add_len; i++) { + err = validate_ref_name(mod->add[i]); + if (err) { + goto exit; + } + slice_set_string(&slashed, mod->add[i]); + slice_append_string(&slashed, "/"); + + err = modification_has_ref_with_prefix( + mod, slice_as_string(&slashed)); + if (err == 0) { + err = REFTABLE_NAME_CONFLICT; + goto exit; + } + if (err < 0) { + goto exit; + } + + slice_set_string(&slashed, mod->add[i]); + while (slashed.len) { + slice_trim_component(&slashed); + err = modification_has_ref(mod, + slice_as_string(&slashed)); + if (err == 0) { + err = REFTABLE_NAME_CONFLICT; + goto exit; + } + if (err < 0) { + goto exit; + } + } + } + err = 0; +exit: + slice_clear(&slashed); + return err; +} diff --git a/reftable/refname.h b/reftable/refname.h new file mode 100644 index 00000000000..e0c40b7fa9e --- /dev/null +++ b/reftable/refname.h @@ -0,0 +1,39 @@ +/* + Copyright 2020 Google LLC + + Use of this source code is governed by a BSD-style + license that can be found in the LICENSE file or at + https://developers.google.com/open-source/licenses/bsd +*/ +#ifndef REFNAME_H +#define REFNAME_H + +struct modification { + struct reftable_table tab; + + char **add; + size_t add_len; + + char **del; + size_t del_len; +}; + +// -1 = error, 0 = found, 1 = not found +int modification_has_ref(struct modification *mod, const char *name); + +// -1 = error, 0 = found, 1 = not found. +int modification_has_ref_with_prefix(struct modification *mod, + const char *prefix); + +// 0 = OK. +int validate_ref_name(const char *name); + +int validate_ref_record_addition(struct reftable_table tab, + struct reftable_ref_record *recs, size_t sz); + +int modification_validate(struct modification *mod); + +/* illegal name, or dir/file conflict */ +#define REFTABLE_REFNAME_ERROR -9 + +#endif diff --git a/reftable/reftable.c b/reftable/reftable.c new file mode 100644 index 00000000000..d29fbd8ee44 --- /dev/null +++ b/reftable/reftable.c @@ -0,0 +1,91 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "reftable.h" +#include "record.h" +#include "reader.h" +#include "merged.h" + +struct reftable_table_vtable { + int (*seek)(void *tab, struct reftable_iterator *it, struct record); +}; + +static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it, + struct record rec) +{ + return reader_seek((struct reftable_reader *)tab, it, rec); +} + +static struct reftable_table_vtable reader_vtable = { + .seek = reftable_reader_seek_void, +}; + +static int reftable_merged_table_seek_void(void *tab, + struct reftable_iterator *it, + struct record rec) +{ + return merged_table_seek_record((struct reftable_merged_table *)tab, it, + rec); +} + +static struct reftable_table_vtable merged_table_vtable = { + .seek = reftable_merged_table_seek_void, +}; + +int reftable_table_seek_ref(struct reftable_table tab, + struct reftable_iterator *it, const char *name) +{ + struct reftable_ref_record ref = { + .ref_name = (char *)name, + }; + struct record rec = { 0 }; + record_from_ref(&rec, &ref); + return tab.ops->seek(tab.table_arg, it, rec); +} + +void reftable_table_from_reader(struct reftable_table *tab, + struct reftable_reader *reader) +{ + assert(tab->ops == NULL); + tab->ops = &reader_vtable; + tab->table_arg = reader; +} + +void reftable_table_from_merged_table(struct reftable_table *tab, + struct reftable_merged_table *merged) +{ + assert(tab->ops == NULL); + tab->ops = &merged_table_vtable; + tab->table_arg = merged; +} + +int reftable_table_read_ref(struct reftable_table tab, const char *name, + struct reftable_ref_record *ref) +{ + struct reftable_iterator it = { 0 }; + int err = reftable_table_seek_ref(tab, &it, name); + if (err) { + goto exit; + } + + err = reftable_iterator_next_ref(it, ref); + if (err) { + goto exit; + } + + if (strcmp(ref->ref_name, name) || + reftable_ref_record_is_deletion(ref)) { + reftable_ref_record_clear(ref); + err = 1; + goto exit; + } + +exit: + reftable_iterator_destroy(&it); + return err; +} diff --git a/reftable/reftable.h b/reftable/reftable.h new file mode 100644 index 00000000000..968a8c644c2 --- /dev/null +++ b/reftable/reftable.h @@ -0,0 +1,564 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef REFTABLE_H +#define REFTABLE_H + +#include +#include + +void reftable_set_alloc(void *(*malloc)(size_t), + void *(*realloc)(void *, size_t), void (*free)(void *)); + +/**************************************************************** + Basic data types + + Reftables store the state of each ref in struct reftable_ref_record, and they + store a sequence of reflog updates in struct reftable_log_record. + ****************************************************************/ + +/* reftable_ref_record holds a ref database entry target_value */ +struct reftable_ref_record { + char *ref_name; /* Name of the ref, malloced. */ + uint64_t update_index; /* Logical timestamp at which this value is + written */ + uint8_t *value; /* SHA1, or NULL. malloced. */ + uint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */ + char *target; /* symref, or NULL. malloced. */ +}; + +/* returns whether 'ref' represents a deletion */ +int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref); + +/* prints a reftable_ref_record onto stdout */ +void reftable_ref_record_print(struct reftable_ref_record *ref, + uint32_t hash_id); + +/* frees and nulls all pointer values. */ +void reftable_ref_record_clear(struct reftable_ref_record *ref); + +/* returns whether two reftable_ref_records are the same */ +int reftable_ref_record_equal(struct reftable_ref_record *a, + struct reftable_ref_record *b, int hash_size); + +/* reftable_log_record holds a reflog entry */ +struct reftable_log_record { + char *ref_name; + uint64_t update_index; /* logical timestamp of a transactional update. + */ + uint8_t *new_hash; + uint8_t *old_hash; + char *name; + char *email; + uint64_t time; + int16_t tz_offset; + char *message; +}; + +/* returns whether 'ref' represents the deletion of a log record. */ +int reftable_log_record_is_deletion(const struct reftable_log_record *log); + +/* frees and nulls all pointer values. */ +void reftable_log_record_clear(struct reftable_log_record *log); + +/* returns whether two records are equal. */ +int reftable_log_record_equal(struct reftable_log_record *a, + struct reftable_log_record *b, int hash_size); + +/* dumps a reftable_log_record on stdout, for debugging/testing. */ +void reftable_log_record_print(struct reftable_log_record *log, + uint32_t hash_id); + +/**************************************************************** + Error handling + + Error are signaled with negative integer return values. 0 means success. + ****************************************************************/ + +/* different types of errors */ +enum reftable_error { + /* Unexpected file system behavior */ + REFTABLE_IO_ERROR = -2, + + /* Format inconsistency on reading data + */ + REFTABLE_FORMAT_ERROR = -3, + + /* File does not exist. Returned from block_source_from_file(), because + it needs special handling in stack. + */ + REFTABLE_NOT_EXIST_ERROR = -4, + + /* Trying to write out-of-date data. */ + REFTABLE_LOCK_ERROR = -5, + + /* Misuse of the API: + - on writing a record with NULL ref_name. + - on writing a reftable_ref_record outside the table limits + - on writing a ref or log record before the stack's next_update_index + - on reading a reftable_ref_record from log iterator, or vice versa. + */ + REFTABLE_API_ERROR = -6, + + /* Decompression error */ + REFTABLE_ZLIB_ERROR = -7, + + /* Wrote a table without blocks. */ + REFTABLE_EMPTY_TABLE_ERROR = -8, + + /* Dir/file conflict. */ + REFTABLE_NAME_CONFLICT = -9, + + /* Illegal ref name. */ + REFTABLE_REFNAME_ERROR = -10, +}; + +/* convert the numeric error code to a string. The string should not be + * deallocated. */ +const char *reftable_error_str(int err); + +/* + * Convert the numeric error code to an equivalent errno code. + */ +int reftable_error_to_errno(int err); + +/**************************************************************** + Writing + + Writing single reftables + ****************************************************************/ + +/* reftable_write_options sets options for writing a single reftable. */ +struct reftable_write_options { + /* boolean: do not pad out blocks to block size. */ + int unpadded; + + /* the blocksize. Should be less than 2^24. */ + uint32_t block_size; + + /* boolean: do not generate a SHA1 => ref index. */ + int skip_index_objects; + + /* how often to write complete keys in each block. */ + int restart_interval; + + /* 4-byte identifier ("sha1", "s256") of the hash. + * Defaults to SHA1 if unset + */ + uint32_t hash_id; + + /* boolean: do not check ref names for validity or dir/file conflicts. + */ + int skip_name_check; +}; + +/* reftable_block_stats holds statistics for a single block type */ +struct reftable_block_stats { + /* total number of entries written */ + int entries; + /* total number of key restarts */ + int restarts; + /* total number of blocks */ + int blocks; + /* total number of index blocks */ + int index_blocks; + /* depth of the index */ + int max_index_level; + + /* offset of the first block for this type */ + uint64_t offset; + /* offset of the top level index block for this type, or 0 if not + * present */ + uint64_t index_offset; +}; + +/* stats holds overall statistics for a single reftable */ +struct reftable_stats { + /* total number of blocks written. */ + int blocks; + /* stats for ref data */ + struct reftable_block_stats ref_stats; + /* stats for the SHA1 to ref map. */ + struct reftable_block_stats obj_stats; + /* stats for index blocks */ + struct reftable_block_stats idx_stats; + /* stats for log blocks */ + struct reftable_block_stats log_stats; + + /* disambiguation length of shortened object IDs. */ + int object_id_len; +}; + +/* reftable_new_writer creates a new writer */ +struct reftable_writer * +reftable_new_writer(int (*writer_func)(void *, uint8_t *, size_t), + void *writer_arg, struct reftable_write_options *opts); + +/* write to a file descriptor. fdp should be an int* pointing to the fd. */ +int reftable_fd_write(void *fdp, uint8_t *data, size_t size); + +/* Set the range of update indices for the records we will add. When + writing a table into a stack, the min should be at least + reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned. + + For transactional updates, typically min==max. When converting an existing + ref database into a single reftable, this would be a range of update-index + timestamps. + */ +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min, + uint64_t max); + +/* adds a reftable_ref_record. Must be called in ascending + order. The update_index must be within the limits set by + reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. + + It is an error to write a ref record after a log record. + */ +int reftable_writer_add_ref(struct reftable_writer *w, + struct reftable_ref_record *ref); + +/* Convenience function to add multiple refs. Will sort the refs by + name before adding. */ +int reftable_writer_add_refs(struct reftable_writer *w, + struct reftable_ref_record *refs, int n); + +/* adds a reftable_log_record. Must be called in ascending order (with more + recent log entries first.) + */ +int reftable_writer_add_log(struct reftable_writer *w, + struct reftable_log_record *log); + +/* Convenience function to add multiple logs. Will sort the records by + key before adding. */ +int reftable_writer_add_logs(struct reftable_writer *w, + struct reftable_log_record *logs, int n); + +/* reftable_writer_close finalizes the reftable. The writer is retained so + * statistics can be inspected. */ +int reftable_writer_close(struct reftable_writer *w); + +/* writer_stats returns the statistics on the reftable being written. + + This struct becomes invalid when the writer is freed. + */ +const struct reftable_stats *writer_stats(struct reftable_writer *w); + +/* reftable_writer_free deallocates memory for the writer */ +void reftable_writer_free(struct reftable_writer *w); + +/**************************************************************** + * ITERATING + ****************************************************************/ + +/* iterator is the generic interface for walking over data stored in a + reftable. It is generally passed around by value. +*/ +struct reftable_iterator { + struct reftable_iterator_vtable *ops; + void *iter_arg; +}; + +/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0: + end of iteration. +*/ +int reftable_iterator_next_ref(struct reftable_iterator it, + struct reftable_ref_record *ref); + +/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0: + end of iteration. +*/ +int reftable_iterator_next_log(struct reftable_iterator it, + struct reftable_log_record *log); + +/* releases resources associated with an iterator. */ +void reftable_iterator_destroy(struct reftable_iterator *it); + +/**************************************************************** + Reading single tables + + The follow routines are for reading single files. For an application-level + interface, skip ahead to struct reftable_merged_table and struct + reftable_stack. + ****************************************************************/ + +/* block_source is a generic wrapper for a seekable readable file. + It is generally passed around by value. + */ +struct reftable_block_source { + struct reftable_block_source_vtable *ops; + void *arg; +}; + +/* a contiguous segment of bytes. It keeps track of its generating block_source + so it can return itself into the pool. +*/ +struct reftable_block { + uint8_t *data; + int len; + struct reftable_block_source source; +}; + +/* block_source_vtable are the operations that make up block_source */ +struct reftable_block_source_vtable { + /* returns the size of a block source */ + uint64_t (*size)(void *source); + + /* reads a segment from the block source. It is an error to read + beyond the end of the block */ + int (*read_block)(void *source, struct reftable_block *dest, + uint64_t off, uint32_t size); + /* mark the block as read; may return the data back to malloc */ + void (*return_block)(void *source, struct reftable_block *blockp); + + /* release all resources associated with the block source */ + void (*close)(void *source); +}; + +/* opens a file on the file system as a block_source */ +int reftable_block_source_from_file(struct reftable_block_source *block_src, + const char *name); + +/* The reader struct is a handle to an open reftable file. */ +struct reftable_reader; + +/* reftable_new_reader opens a reftable for reading. If successful, returns 0 + * code and sets pp. The name is used for creating a stack. Typically, it is the + * basename of the file. The block source `src` is owned by the reader, and is + * closed on calling reftable_reader_destroy(). + */ +int reftable_new_reader(struct reftable_reader **pp, + struct reftable_block_source src, const char *name); + +/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted + in the table. To seek to the start of the table, use name = "". + + example: + + struct reftable_reader *r = NULL; + int err = reftable_new_reader(&r, src, "filename"); + if (err < 0) { ... } + struct reftable_iterator it = {0}; + err = reftable_reader_seek_ref(r, &it, "refs/heads/master"); + if (err < 0) { ... } + struct reftable_ref_record ref = {0}; + while (1) { + err = reftable_iterator_next_ref(it, &ref); + if (err > 0) { + break; + } + if (err < 0) { + ..error handling.. + } + ..found.. + } + reftable_iterator_destroy(&it); + reftable_ref_record_clear(&ref); + */ +int reftable_reader_seek_ref(struct reftable_reader *r, + struct reftable_iterator *it, const char *name); + +/* returns the hash ID used in this table. */ +uint32_t reftable_reader_hash_id(struct reftable_reader *r); + +/* seek to logs for the given name, older than update_index. To seek to the + start of the table, use name = "". + */ +int reftable_reader_seek_log_at(struct reftable_reader *r, + struct reftable_iterator *it, const char *name, + uint64_t update_index); + +/* seek to newest log entry for given name. */ +int reftable_reader_seek_log(struct reftable_reader *r, + struct reftable_iterator *it, const char *name); + +/* closes and deallocates a reader. */ +void reftable_reader_free(struct reftable_reader *); + +/* return an iterator for the refs pointing to oid */ +int reftable_reader_refs_for(struct reftable_reader *r, + struct reftable_iterator *it, uint8_t *oid, + int oid_len); + +/* return the max_update_index for a table */ +uint64_t reftable_reader_max_update_index(struct reftable_reader *r); + +/* return the min_update_index for a table */ +uint64_t reftable_reader_min_update_index(struct reftable_reader *r); + +/**************************************************************** + Merged tables + + A ref database kept in a sequence of table files. The merged_table presents a + unified view to reading (seeking, iterating) a sequence of immutable tables. + ****************************************************************/ + +/* A merged table is implements seeking/iterating over a stack of tables. */ +struct reftable_merged_table; + +/* reftable_new_merged_table creates a new merged table. It takes ownership of + the stack array. +*/ +int reftable_new_merged_table(struct reftable_merged_table **dest, + struct reftable_reader **stack, int n, + uint32_t hash_id); + +/* returns an iterator positioned just before 'name' */ +int reftable_merged_table_seek_ref(struct reftable_merged_table *mt, + struct reftable_iterator *it, + const char *name); + +/* returns an iterator for log entry, at given update_index */ +int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt, + struct reftable_iterator *it, + const char *name, uint64_t update_index); + +/* like reftable_merged_table_seek_log_at but look for the newest entry. */ +int reftable_merged_table_seek_log(struct reftable_merged_table *mt, + struct reftable_iterator *it, + const char *name); + +/* returns the max update_index covered by this merged table. */ +uint64_t +reftable_merged_table_max_update_index(struct reftable_merged_table *mt); + +/* returns the min update_index covered by this merged table. */ +uint64_t +reftable_merged_table_min_update_index(struct reftable_merged_table *mt); + +/* closes readers for the merged tables */ +void reftable_merged_table_close(struct reftable_merged_table *mt); + +/* releases memory for the merged_table */ +void reftable_merged_table_free(struct reftable_merged_table *m); + +/**************************************************************** + Generic tables + + A unified API for reading tables, either merged tables, or single readers. + ****************************************************************/ + +struct reftable_table { + struct reftable_table_vtable *ops; + void *table_arg; +}; + +int reftable_table_seek_ref(struct reftable_table tab, + struct reftable_iterator *it, const char *name); + +void reftable_table_from_reader(struct reftable_table *tab, + struct reftable_reader *reader); +void reftable_table_from_merged_table(struct reftable_table *tab, + struct reftable_merged_table *table); + +/* convenience function to read a single ref. Returns < 0 for error, 0 + for success, and 1 if ref not found. */ +int reftable_table_read_ref(struct reftable_table tab, const char *name, + struct reftable_ref_record *ref); + +/**************************************************************** + Mutable ref database + + The stack presents an interface to a mutable sequence of reftables. + ****************************************************************/ + +/* a stack is a stack of reftables, which can be mutated by pushing a table to + * the top of the stack */ +struct reftable_stack; + +/* open a new reftable stack. The tables along with the table list will be + stored in 'dir'. Typically, this should be .git/reftables. +*/ +int reftable_new_stack(struct reftable_stack **dest, const char *dir, + struct reftable_write_options config); + +/* returns the update_index at which a next table should be written. */ +uint64_t reftable_stack_next_update_index(struct reftable_stack *st); + +/* holds a transaction to add tables at the top of a stack. */ +struct reftable_addition; + +/* + returns a new transaction to add reftables to the given stack. As a side + effect, the ref database is locked. +*/ +int reftable_stack_new_addition(struct reftable_addition **dest, + struct reftable_stack *st); + +/* Adds a reftable to transaction. */ +int reftable_addition_add(struct reftable_addition *add, + int (*write_table)(struct reftable_writer *wr, + void *arg), + void *arg); + +/* Commits the transaction, releasing the lock. */ +int reftable_addition_commit(struct reftable_addition *add); + +/* Release all non-committed data from the transaction; releases the lock if + * held. */ +void reftable_addition_close(struct reftable_addition *add); + +/* add a new table to the stack. The write_table function must call + reftable_writer_set_limits, add refs and return an error value. */ +int reftable_stack_add(struct reftable_stack *st, + int (*write_table)(struct reftable_writer *wr, + void *write_arg), + void *write_arg); + +/* returns the merged_table for seeking. This table is valid until the + next write or reload, and should not be closed or deleted. +*/ +struct reftable_merged_table * +reftable_stack_merged_table(struct reftable_stack *st); + +/* frees all resources associated with the stack. */ +void reftable_stack_destroy(struct reftable_stack *st); + +/* reloads the stack if necessary. */ +int reftable_stack_reload(struct reftable_stack *st); + +/* Policy for expiring reflog entries. */ +struct reftable_log_expiry_config { + /* Drop entries older than this timestamp */ + uint64_t time; + + /* Drop older entries */ + uint64_t min_update_index; +}; + +/* compacts all reftables into a giant table. Expire reflog entries if config is + * non-NULL */ +int reftable_stack_compact_all(struct reftable_stack *st, + struct reftable_log_expiry_config *config); + +/* heuristically compact unbalanced table stack. */ +int reftable_stack_auto_compact(struct reftable_stack *st); + +/* convenience function to read a single ref. Returns < 0 for error, 0 + for success, and 1 if ref not found. */ +int reftable_stack_read_ref(struct reftable_stack *st, const char *refname, + struct reftable_ref_record *ref); + +/* convenience function to read a single log. Returns < 0 for error, 0 + for success, and 1 if ref not found. */ +int reftable_stack_read_log(struct reftable_stack *st, const char *refname, + struct reftable_log_record *log); + +/* statistics on past compactions. */ +struct reftable_compaction_stats { + uint64_t bytes; /* total number of bytes written */ + uint64_t entries_written; /* total number of entries written, including + failures. */ + int attempts; /* how often we tried to compact */ + int failures; /* failures happen on concurrent updates */ +}; + +/* return statistics for compaction up till now. */ +struct reftable_compaction_stats * +reftable_stack_compaction_stats(struct reftable_stack *st); + +#endif diff --git a/reftable/slice.c b/reftable/slice.c new file mode 100644 index 00000000000..89e649b0d46 --- /dev/null +++ b/reftable/slice.c @@ -0,0 +1,225 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "slice.h" + +#include "system.h" + +#include "reftable.h" + +void slice_set_string(struct slice *s, const char *str) +{ + if (str == NULL) { + s->len = 0; + return; + } + + { + int l = strlen(str); + l++; /* \0 */ + slice_resize(s, l); + memcpy(s->buf, str, l); + s->len = l - 1; + } +} + +void slice_resize(struct slice *s, int l) +{ + if (s->cap < l) { + int c = s->cap * 2; + if (c < l) { + c = l; + } + s->cap = c; + s->buf = reftable_realloc(s->buf, s->cap); + } + s->len = l; +} + +void slice_append_string(struct slice *d, const char *s) +{ + int l1 = d->len; + int l2 = strlen(s); + + slice_resize(d, l2 + l1); + memcpy(d->buf + l1, s, l2); +} + +void slice_append(struct slice *s, struct slice a) +{ + int end = s->len; + slice_resize(s, s->len + a.len); + memcpy(s->buf + end, a.buf, a.len); +} + +void slice_consume(struct slice *s, int n) +{ + s->buf += n; + s->len -= n; +} + +byte *slice_yield(struct slice *s) +{ + byte *p = s->buf; + s->buf = NULL; + s->cap = 0; + s->len = 0; + return p; +} + +void slice_clear(struct slice *s) +{ + reftable_free(slice_yield(s)); +} + +void slice_copy(struct slice *dest, struct slice src) +{ + slice_resize(dest, src.len); + memcpy(dest->buf, src.buf, src.len); +} + +/* return the underlying data as char*. len is left unchanged, but + a \0 is added at the end. */ +const char *slice_as_string(struct slice *s) +{ + if (s->cap == s->len) { + int l = s->len; + slice_resize(s, l + 1); + s->len = l; + } + s->buf[s->len] = 0; + return (const char *)s->buf; +} + +/* return a newly malloced string for this slice */ +char *slice_to_string(struct slice in) +{ + struct slice s = { 0 }; + slice_resize(&s, in.len + 1); + s.buf[in.len] = 0; + memcpy(s.buf, in.buf, in.len); + return (char *)slice_yield(&s); +} + +bool slice_equal(struct slice a, struct slice b) +{ + if (a.len != b.len) { + return 0; + } + return memcmp(a.buf, b.buf, a.len) == 0; +} + +int slice_compare(struct slice a, struct slice b) +{ + int min = a.len < b.len ? a.len : b.len; + int res = memcmp(a.buf, b.buf, min); + if (res != 0) { + return res; + } + if (a.len < b.len) { + return -1; + } else if (a.len > b.len) { + return 1; + } else { + return 0; + } +} + +int slice_write(struct slice *b, byte *data, size_t sz) +{ + if (b->len + sz > b->cap) { + int newcap = 2 * b->cap + 1; + if (newcap < b->len + sz) { + newcap = (b->len + sz); + } + b->buf = reftable_realloc(b->buf, newcap); + b->cap = newcap; + } + + memcpy(b->buf + b->len, data, sz); + b->len += sz; + return sz; +} + +int slice_write_void(void *b, byte *data, size_t sz) +{ + return slice_write((struct slice *)b, data, sz); +} + +static uint64_t slice_size(void *b) +{ + return ((struct slice *)b)->len; +} + +static void slice_return_block(void *b, struct reftable_block *dest) +{ + memset(dest->data, 0xff, dest->len); + reftable_free(dest->data); +} + +static void slice_close(void *b) +{ +} + +static int slice_read_block(void *v, struct reftable_block *dest, uint64_t off, + uint32_t size) +{ + struct slice *b = (struct slice *)v; + assert(off + size <= b->len); + dest->data = reftable_calloc(size); + memcpy(dest->data, b->buf + off, size); + dest->len = size; + return size; +} + +struct reftable_block_source_vtable slice_vtable = { + .size = &slice_size, + .read_block = &slice_read_block, + .return_block = &slice_return_block, + .close = &slice_close, +}; + +void block_source_from_slice(struct reftable_block_source *bs, + struct slice *buf) +{ + assert(bs->ops == NULL); + bs->ops = &slice_vtable; + bs->arg = buf; +} + +static void malloc_return_block(void *b, struct reftable_block *dest) +{ + memset(dest->data, 0xff, dest->len); + reftable_free(dest->data); +} + +struct reftable_block_source_vtable malloc_vtable = { + .return_block = &malloc_return_block, +}; + +struct reftable_block_source malloc_block_source_instance = { + .ops = &malloc_vtable, +}; + +struct reftable_block_source malloc_block_source(void) +{ + return malloc_block_source_instance; +} + +int common_prefix_size(struct slice a, struct slice b) +{ + int p = 0; + while (p < a.len && p < b.len) { + if (a.buf[p] != b.buf[p]) { + break; + } + p++; + } + + return p; +} diff --git a/reftable/slice.h b/reftable/slice.h new file mode 100644 index 00000000000..886f22811c4 --- /dev/null +++ b/reftable/slice.h @@ -0,0 +1,76 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef SLICE_H +#define SLICE_H + +#include "basics.h" +#include "reftable.h" + +/* + provides bounds-checked byte ranges. + To use, initialize as "slice x = {0};" + */ +struct slice { + int len; + int cap; + byte *buf; +}; + +void slice_set_string(struct slice *dest, const char *src); +void slice_append_string(struct slice *dest, const char *src); +/* Set length to 0, but retain buffer */ +void slice_clear(struct slice *slice); + +/* Return a malloced string for `src` */ +char *slice_to_string(struct slice src); + +/* Ensure that `buf` is \0 terminated. */ +const char *slice_as_string(struct slice *src); + +/* Compare slices */ +bool slice_equal(struct slice a, struct slice b); + +/* Return `buf`, clearing out `s` */ +byte *slice_yield(struct slice *s); + +/* Copy bytes */ +void slice_copy(struct slice *dest, struct slice src); + +/* Advance `buf` by `n`, and decrease length. A copy of the slice + should be kept for deallocating the slice. */ +void slice_consume(struct slice *s, int n); + +/* Set length of the slice to `l` */ +void slice_resize(struct slice *s, int l); + +/* Signed comparison */ +int slice_compare(struct slice a, struct slice b); + +/* Append `data` to the `dest` slice. */ +int slice_write(struct slice *dest, byte *data, size_t sz); + +/* Append `add` to `dest. */ +void slice_append(struct slice *dest, struct slice add); + +/* Like slice_write, but suitable for passing to reftable_new_writer + */ +int slice_write_void(void *b, byte *data, size_t sz); + +/* Find the longest shared prefix size of `a` and `b` */ +int common_prefix_size(struct slice a, struct slice b); + +struct reftable_block_source; + +/* Create an in-memory block source for reading reftables */ +void block_source_from_slice(struct reftable_block_source *bs, + struct slice *buf); + +struct reftable_block_source malloc_block_source(void); + +#endif diff --git a/reftable/stack.c b/reftable/stack.c new file mode 100644 index 00000000000..0f44bd258ed --- /dev/null +++ b/reftable/stack.c @@ -0,0 +1,1229 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "stack.h" + +#include "system.h" +#include "merged.h" +#include "reader.h" +#include "refname.h" +#include "reftable.h" +#include "writer.h" + +int reftable_new_stack(struct reftable_stack **dest, const char *dir, + struct reftable_write_options config) +{ + struct reftable_stack *p = + reftable_calloc(sizeof(struct reftable_stack)); + struct slice list_file_name = { 0 }; + int err = 0; + + if (config.hash_id == 0) { + config.hash_id = SHA1_ID; + } + + *dest = NULL; + + slice_set_string(&list_file_name, dir); + slice_append_string(&list_file_name, "/tables.list"); + + p->list_file = slice_to_string(list_file_name); + slice_clear(&list_file_name); + p->reftable_dir = xstrdup(dir); + p->config = config; + + err = reftable_stack_reload(p); + if (err < 0) { + reftable_stack_destroy(p); + } else { + *dest = p; + } + return err; +} + +static int fd_read_lines(int fd, char ***namesp) +{ + off_t size = lseek(fd, 0, SEEK_END); + char *buf = NULL; + int err = 0; + if (size < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + err = lseek(fd, 0, SEEK_SET); + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + buf = reftable_malloc(size + 1); + if (read(fd, buf, size) != size) { + err = REFTABLE_IO_ERROR; + goto exit; + } + buf[size] = 0; + + parse_names(buf, size, namesp); + +exit: + reftable_free(buf); + return err; +} + +int read_lines(const char *filename, char ***namesp) +{ + int fd = open(filename, O_RDONLY, 0644); + int err = 0; + if (fd < 0) { + if (errno == ENOENT) { + *namesp = reftable_calloc(sizeof(char *)); + return 0; + } + + return REFTABLE_IO_ERROR; + } + err = fd_read_lines(fd, namesp); + close(fd); + return err; +} + +struct reftable_merged_table * +reftable_stack_merged_table(struct reftable_stack *st) +{ + return st->merged; +} + +/* Close and free the stack */ +void reftable_stack_destroy(struct reftable_stack *st) +{ + if (st->merged == NULL) { + return; + } + + reftable_merged_table_close(st->merged); + reftable_merged_table_free(st->merged); + st->merged = NULL; + + FREE_AND_NULL(st->list_file); + FREE_AND_NULL(st->reftable_dir); + reftable_free(st); +} + +static struct reftable_reader **stack_copy_readers(struct reftable_stack *st, + int cur_len) +{ + struct reftable_reader **cur = + reftable_calloc(sizeof(struct reftable_reader *) * cur_len); + int i = 0; + for (i = 0; i < cur_len; i++) { + cur[i] = st->merged->stack[i]; + } + return cur; +} + +static int reftable_stack_reload_once(struct reftable_stack *st, char **names, + bool reuse_open) +{ + int cur_len = st->merged == NULL ? 0 : st->merged->stack_len; + struct reftable_reader **cur = stack_copy_readers(st, cur_len); + int err = 0; + int names_len = names_length(names); + struct reftable_reader **new_tables = + reftable_malloc(sizeof(struct reftable_reader *) * names_len); + int new_tables_len = 0; + struct reftable_merged_table *new_merged = NULL; + + struct slice table_path = { 0 }; + + while (*names) { + struct reftable_reader *rd = NULL; + char *name = *names++; + + /* this is linear; we assume compaction keeps the number of + tables under control so this is not quadratic. */ + int j = 0; + for (j = 0; reuse_open && j < cur_len; j++) { + if (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) { + rd = cur[j]; + cur[j] = NULL; + break; + } + } + + if (rd == NULL) { + struct reftable_block_source src = { 0 }; + slice_set_string(&table_path, st->reftable_dir); + slice_append_string(&table_path, "/"); + slice_append_string(&table_path, name); + + err = reftable_block_source_from_file( + &src, slice_as_string(&table_path)); + if (err < 0) { + goto exit; + } + + err = reftable_new_reader(&rd, src, name); + if (err < 0) { + goto exit; + } + } + + new_tables[new_tables_len++] = rd; + } + + /* success! */ + err = reftable_new_merged_table(&new_merged, new_tables, new_tables_len, + st->config.hash_id); + if (err < 0) { + goto exit; + } + + new_tables = NULL; + new_tables_len = 0; + if (st->merged != NULL) { + merged_table_clear(st->merged); + reftable_merged_table_free(st->merged); + } + new_merged->suppress_deletions = true; + st->merged = new_merged; + + { + int i = 0; + for (i = 0; i < cur_len; i++) { + if (cur[i] != NULL) { + reader_close(cur[i]); + reftable_reader_free(cur[i]); + } + } + } + +exit: + slice_clear(&table_path); + { + int i = 0; + for (i = 0; i < new_tables_len; i++) { + reader_close(new_tables[i]); + } + } + reftable_free(new_tables); + reftable_free(cur); + return err; +} + +/* return negative if a before b. */ +static int tv_cmp(struct timeval *a, struct timeval *b) +{ + time_t diff = a->tv_sec - b->tv_sec; + int udiff = a->tv_usec - b->tv_usec; + + if (diff != 0) { + return diff; + } + + return udiff; +} + +static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st, + bool reuse_open) +{ + struct timeval deadline = { 0 }; + int err = gettimeofday(&deadline, NULL); + int64_t delay = 0; + int tries = 0; + if (err < 0) { + return err; + } + + deadline.tv_sec += 3; + while (true) { + char **names = NULL; + char **names_after = NULL; + struct timeval now = { 0 }; + int err = gettimeofday(&now, NULL); + int err2 = 0; + if (err < 0) { + return err; + } + + /* Only look at deadlines after the first few times. This + simplifies debugging in GDB */ + tries++; + if (tries > 3 && tv_cmp(&now, &deadline) >= 0) { + break; + } + + err = read_lines(st->list_file, &names); + if (err < 0) { + free_names(names); + return err; + } + err = reftable_stack_reload_once(st, names, reuse_open); + if (err == 0) { + free_names(names); + break; + } + if (err != REFTABLE_NOT_EXIST_ERROR) { + free_names(names); + return err; + } + + /* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent + writer. Check if there was one by checking if the name list + changed. + */ + err2 = read_lines(st->list_file, &names_after); + if (err2 < 0) { + free_names(names); + return err2; + } + + if (names_equal(names_after, names)) { + free_names(names); + free_names(names_after); + return err; + } + free_names(names); + free_names(names_after); + + delay = delay + (delay * rand()) / RAND_MAX + 1; + sleep_millisec(delay); + } + + return 0; +} + +int reftable_stack_reload(struct reftable_stack *st) +{ + return reftable_stack_reload_maybe_reuse(st, true); +} + +/* -1 = error + 0 = up to date + 1 = changed. */ +static int stack_uptodate(struct reftable_stack *st) +{ + char **names = NULL; + int err = read_lines(st->list_file, &names); + int i = 0; + if (err < 0) { + return err; + } + + for (i = 0; i < st->merged->stack_len; i++) { + if (names[i] == NULL) { + err = 1; + goto exit; + } + + if (strcmp(st->merged->stack[i]->name, names[i])) { + err = 1; + goto exit; + } + } + + if (names[st->merged->stack_len] != NULL) { + err = 1; + goto exit; + } + +exit: + free_names(names); + return err; +} + +int reftable_stack_add(struct reftable_stack *st, + int (*write)(struct reftable_writer *wr, void *arg), + void *arg) +{ + int err = stack_try_add(st, write, arg); + if (err < 0) { + if (err == REFTABLE_LOCK_ERROR) { + /* Ignore error return, we want to propagate + REFTABLE_LOCK_ERROR. + */ + reftable_stack_reload(st); + } + return err; + } + + if (!st->disable_auto_compact) { + return reftable_stack_auto_compact(st); + } + + return 0; +} + +static void format_name(struct slice *dest, uint64_t min, uint64_t max) +{ + char buf[100]; + snprintf(buf, sizeof(buf), "0x%012" PRIx64 "-0x%012" PRIx64, min, max); + slice_set_string(dest, buf); +} + +struct reftable_addition { + int lock_file_fd; + struct slice lock_file_name; + struct reftable_stack *stack; + char **names; + char **new_tables; + int new_tables_len; + uint64_t next_update_index; +}; + +static int reftable_stack_init_addition(struct reftable_addition *add, + struct reftable_stack *st) +{ + int err = 0; + add->stack = st; + + slice_set_string(&add->lock_file_name, st->list_file); + slice_append_string(&add->lock_file_name, ".lock"); + + add->lock_file_fd = open(slice_as_string(&add->lock_file_name), + O_EXCL | O_CREAT | O_WRONLY, 0644); + if (add->lock_file_fd < 0) { + if (errno == EEXIST) { + err = REFTABLE_LOCK_ERROR; + } else { + err = REFTABLE_IO_ERROR; + } + goto exit; + } + err = stack_uptodate(st); + if (err < 0) { + goto exit; + } + + if (err > 1) { + err = REFTABLE_LOCK_ERROR; + goto exit; + } + + add->next_update_index = reftable_stack_next_update_index(st); +exit: + if (err) { + reftable_addition_close(add); + } + return err; +} + +void reftable_addition_close(struct reftable_addition *add) +{ + int i = 0; + struct slice nm = { 0 }; + for (i = 0; i < add->new_tables_len; i++) { + slice_set_string(&nm, add->stack->list_file); + slice_append_string(&nm, "/"); + slice_append_string(&nm, add->new_tables[i]); + unlink(slice_as_string(&nm)); + reftable_free(add->new_tables[i]); + add->new_tables[i] = NULL; + } + reftable_free(add->new_tables); + add->new_tables = NULL; + add->new_tables_len = 0; + + if (add->lock_file_fd > 0) { + close(add->lock_file_fd); + add->lock_file_fd = 0; + } + if (add->lock_file_name.len > 0) { + unlink(slice_as_string(&add->lock_file_name)); + slice_clear(&add->lock_file_name); + } + + free_names(add->names); + add->names = NULL; + slice_clear(&nm); +} + +int reftable_addition_commit(struct reftable_addition *add) +{ + struct slice table_list = { 0 }; + int i = 0; + int err = 0; + if (add->new_tables_len == 0) { + goto exit; + } + + for (i = 0; i < add->stack->merged->stack_len; i++) { + slice_append_string(&table_list, + add->stack->merged->stack[i]->name); + slice_append_string(&table_list, "\n"); + } + for (i = 0; i < add->new_tables_len; i++) { + slice_append_string(&table_list, add->new_tables[i]); + slice_append_string(&table_list, "\n"); + } + + err = write(add->lock_file_fd, table_list.buf, table_list.len); + slice_clear(&table_list); + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + err = close(add->lock_file_fd); + add->lock_file_fd = 0; + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + err = rename(slice_as_string(&add->lock_file_name), + add->stack->list_file); + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + err = reftable_stack_reload(add->stack); + +exit: + reftable_addition_close(add); + return err; +} + +int reftable_stack_new_addition(struct reftable_addition **dest, + struct reftable_stack *st) +{ + int err = 0; + *dest = reftable_malloc(sizeof(**dest)); + err = reftable_stack_init_addition(*dest, st); + if (err) { + reftable_free(*dest); + *dest = NULL; + } + return err; +} + +int stack_try_add(struct reftable_stack *st, + int (*write_table)(struct reftable_writer *wr, void *arg), + void *arg) +{ + struct reftable_addition add = { 0 }; + int err = reftable_stack_init_addition(&add, st); + if (err < 0) { + goto exit; + } + + err = reftable_addition_add(&add, write_table, arg); + if (err < 0) { + goto exit; + } + + err = reftable_addition_commit(&add); +exit: + reftable_addition_close(&add); + return err; +} + +int reftable_addition_add(struct reftable_addition *add, + int (*write_table)(struct reftable_writer *wr, + void *arg), + void *arg) +{ + struct slice temp_tab_file_name = { 0 }; + struct slice tab_file_name = { 0 }; + struct slice next_name = { 0 }; + struct reftable_writer *wr = NULL; + int err = 0; + int tab_fd = 0; + + slice_resize(&next_name, 0); + format_name(&next_name, add->next_update_index, add->next_update_index); + + slice_set_string(&temp_tab_file_name, add->stack->reftable_dir); + slice_append_string(&temp_tab_file_name, "/"); + slice_append(&temp_tab_file_name, next_name); + slice_append_string(&temp_tab_file_name, ".temp.XXXXXX"); + + tab_fd = mkstemp((char *)slice_as_string(&temp_tab_file_name)); + if (tab_fd < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + wr = reftable_new_writer(reftable_fd_write, &tab_fd, + &add->stack->config); + err = write_table(wr, arg); + if (err < 0) { + goto exit; + } + + err = reftable_writer_close(wr); + if (err == REFTABLE_EMPTY_TABLE_ERROR) { + err = 0; + goto exit; + } + if (err < 0) { + goto exit; + } + + err = close(tab_fd); + tab_fd = 0; + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + err = stack_check_addition(add->stack, + slice_as_string(&temp_tab_file_name)); + if (err < 0) { + goto exit; + } + + if (wr->min_update_index < add->next_update_index) { + err = REFTABLE_API_ERROR; + goto exit; + } + + format_name(&next_name, wr->min_update_index, wr->max_update_index); + slice_append_string(&next_name, ".ref"); + + slice_set_string(&tab_file_name, add->stack->reftable_dir); + slice_append_string(&tab_file_name, "/"); + slice_append(&tab_file_name, next_name); + + err = rename(slice_as_string(&temp_tab_file_name), + slice_as_string(&tab_file_name)); + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + + add->new_tables = reftable_realloc(add->new_tables, + sizeof(*add->new_tables) * + (add->new_tables_len + 1)); + add->new_tables[add->new_tables_len] = slice_to_string(next_name); + add->new_tables_len++; +exit: + if (tab_fd > 0) { + close(tab_fd); + tab_fd = 0; + } + if (temp_tab_file_name.len > 0) { + unlink(slice_as_string(&temp_tab_file_name)); + } + + slice_clear(&temp_tab_file_name); + slice_clear(&tab_file_name); + slice_clear(&next_name); + reftable_writer_free(wr); + return err; +} + +uint64_t reftable_stack_next_update_index(struct reftable_stack *st) +{ + int sz = st->merged->stack_len; + if (sz > 0) { + return reftable_reader_max_update_index( + st->merged->stack[sz - 1]) + + 1; + } + return 1; +} + +static int stack_compact_locked(struct reftable_stack *st, int first, int last, + struct slice *temp_tab, + struct reftable_log_expiry_config *config) +{ + struct slice next_name = { 0 }; + int tab_fd = -1; + struct reftable_writer *wr = NULL; + int err = 0; + + format_name(&next_name, + reftable_reader_min_update_index(st->merged->stack[first]), + reftable_reader_max_update_index(st->merged->stack[first])); + + slice_set_string(temp_tab, st->reftable_dir); + slice_append_string(temp_tab, "/"); + slice_append(temp_tab, next_name); + slice_append_string(temp_tab, ".temp.XXXXXX"); + + tab_fd = mkstemp((char *)slice_as_string(temp_tab)); + wr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config); + + err = stack_write_compact(st, wr, first, last, config); + if (err < 0) { + goto exit; + } + err = reftable_writer_close(wr); + if (err < 0) { + goto exit; + } + reftable_writer_free(wr); + + err = close(tab_fd); + tab_fd = 0; + +exit: + if (tab_fd > 0) { + close(tab_fd); + tab_fd = 0; + } + if (err != 0 && temp_tab->len > 0) { + unlink(slice_as_string(temp_tab)); + slice_clear(temp_tab); + } + slice_clear(&next_name); + return err; +} + +int stack_write_compact(struct reftable_stack *st, struct reftable_writer *wr, + int first, int last, + struct reftable_log_expiry_config *config) +{ + int subtabs_len = last - first + 1; + struct reftable_reader **subtabs = reftable_calloc( + sizeof(struct reftable_reader *) * (last - first + 1)); + struct reftable_merged_table *mt = NULL; + int err = 0; + struct reftable_iterator it = { 0 }; + struct reftable_ref_record ref = { 0 }; + struct reftable_log_record log = { 0 }; + + uint64_t entries = 0; + + int i = 0, j = 0; + for (i = first, j = 0; i <= last; i++) { + struct reftable_reader *t = st->merged->stack[i]; + subtabs[j++] = t; + st->stats.bytes += t->size; + } + reftable_writer_set_limits(wr, + st->merged->stack[first]->min_update_index, + st->merged->stack[last]->max_update_index); + + err = reftable_new_merged_table(&mt, subtabs, subtabs_len, + st->config.hash_id); + if (err < 0) { + reftable_free(subtabs); + goto exit; + } + + err = reftable_merged_table_seek_ref(mt, &it, ""); + if (err < 0) { + goto exit; + } + + while (true) { + err = reftable_iterator_next_ref(it, &ref); + if (err > 0) { + err = 0; + break; + } + if (err < 0) { + break; + } + if (first == 0 && reftable_ref_record_is_deletion(&ref)) { + continue; + } + + err = reftable_writer_add_ref(wr, &ref); + if (err < 0) { + break; + } + entries++; + } + reftable_iterator_destroy(&it); + + err = reftable_merged_table_seek_log(mt, &it, ""); + if (err < 0) { + goto exit; + } + + while (true) { + err = reftable_iterator_next_log(it, &log); + if (err > 0) { + err = 0; + break; + } + if (err < 0) { + break; + } + if (first == 0 && reftable_log_record_is_deletion(&log)) { + continue; + } + + if (config != NULL && config->time > 0 && + log.time < config->time) { + continue; + } + + if (config != NULL && config->min_update_index > 0 && + log.update_index < config->min_update_index) { + continue; + } + + err = reftable_writer_add_log(wr, &log); + if (err < 0) { + break; + } + entries++; + } + +exit: + reftable_iterator_destroy(&it); + if (mt != NULL) { + merged_table_clear(mt); + reftable_merged_table_free(mt); + } + reftable_ref_record_clear(&ref); + reftable_log_record_clear(&log); + st->stats.entries_written += entries; + return err; +} + +/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */ +static int stack_compact_range(struct reftable_stack *st, int first, int last, + struct reftable_log_expiry_config *expiry) +{ + struct slice temp_tab_file_name = { 0 }; + struct slice new_table_name = { 0 }; + struct slice lock_file_name = { 0 }; + struct slice ref_list_contents = { 0 }; + struct slice new_table_path = { 0 }; + int err = 0; + bool have_lock = false; + int lock_file_fd = 0; + int compact_count = last - first + 1; + char **delete_on_success = + reftable_calloc(sizeof(char *) * (compact_count + 1)); + char **subtable_locks = + reftable_calloc(sizeof(char *) * (compact_count + 1)); + int i = 0; + int j = 0; + bool is_empty_table = false; + + if (first > last || (expiry == NULL && first == last)) { + err = 0; + goto exit; + } + + st->stats.attempts++; + + slice_set_string(&lock_file_name, st->list_file); + slice_append_string(&lock_file_name, ".lock"); + + lock_file_fd = open(slice_as_string(&lock_file_name), + O_EXCL | O_CREAT | O_WRONLY, 0644); + if (lock_file_fd < 0) { + if (errno == EEXIST) { + err = 1; + } else { + err = REFTABLE_IO_ERROR; + } + goto exit; + } + /* Don't want to write to the lock for now. */ + close(lock_file_fd); + lock_file_fd = 0; + + have_lock = true; + err = stack_uptodate(st); + if (err != 0) { + goto exit; + } + + for (i = first, j = 0; i <= last; i++) { + struct slice subtab_file_name = { 0 }; + struct slice subtab_lock = { 0 }; + slice_set_string(&subtab_file_name, st->reftable_dir); + slice_append_string(&subtab_file_name, "/"); + slice_append_string(&subtab_file_name, + reader_name(st->merged->stack[i])); + + slice_copy(&subtab_lock, subtab_file_name); + slice_append_string(&subtab_lock, ".lock"); + + { + int sublock_file_fd = + open(slice_as_string(&subtab_lock), + O_EXCL | O_CREAT | O_WRONLY, 0644); + if (sublock_file_fd > 0) { + close(sublock_file_fd); + } else if (sublock_file_fd < 0) { + if (errno == EEXIST) { + err = 1; + } else { + err = REFTABLE_IO_ERROR; + } + } + } + + subtable_locks[j] = (char *)slice_as_string(&subtab_lock); + delete_on_success[j] = + (char *)slice_as_string(&subtab_file_name); + j++; + + if (err != 0) { + goto exit; + } + } + + err = unlink(slice_as_string(&lock_file_name)); + if (err < 0) { + goto exit; + } + have_lock = false; + + err = stack_compact_locked(st, first, last, &temp_tab_file_name, + expiry); + /* Compaction + tombstones can create an empty table out of non-empty + * tables. */ + is_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR); + if (is_empty_table) { + err = 0; + } + if (err < 0) { + goto exit; + } + + lock_file_fd = open(slice_as_string(&lock_file_name), + O_EXCL | O_CREAT | O_WRONLY, 0644); + if (lock_file_fd < 0) { + if (errno == EEXIST) { + err = 1; + } else { + err = REFTABLE_IO_ERROR; + } + goto exit; + } + have_lock = true; + + format_name(&new_table_name, st->merged->stack[first]->min_update_index, + st->merged->stack[last]->max_update_index); + slice_append_string(&new_table_name, ".ref"); + + slice_set_string(&new_table_path, st->reftable_dir); + slice_append_string(&new_table_path, "/"); + + slice_append(&new_table_path, new_table_name); + + if (!is_empty_table) { + err = rename(slice_as_string(&temp_tab_file_name), + slice_as_string(&new_table_path)); + if (err < 0) { + err = REFTABLE_IO_ERROR; + goto exit; + } + } + + for (i = 0; i < first; i++) { + slice_append_string(&ref_list_contents, + st->merged->stack[i]->name); + slice_append_string(&ref_list_contents, "\n"); + } + if (!is_empty_table) { + slice_append(&ref_list_contents, new_table_name); + slice_append_string(&ref_list_contents, "\n"); + } + for (i = last + 1; i < st->merged->stack_len; i++) { + slice_append_string(&ref_list_contents, + st->merged->stack[i]->name); + slice_append_string(&ref_list_contents, "\n"); + } + + err = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len); + if (err < 0) { + err = REFTABLE_IO_ERROR; + unlink(slice_as_string(&new_table_path)); + goto exit; + } + err = close(lock_file_fd); + lock_file_fd = 0; + if (err < 0) { + err = REFTABLE_IO_ERROR; + unlink(slice_as_string(&new_table_path)); + goto exit; + } + + err = rename(slice_as_string(&lock_file_name), st->list_file); + if (err < 0) { + err = REFTABLE_IO_ERROR; + unlink(slice_as_string(&new_table_path)); + goto exit; + } + have_lock = false; + + /* Reload the stack before deleting. On windows, we can only delete the + files after we closed them. + */ + err = reftable_stack_reload_maybe_reuse(st, first < last); + + { + char **p = delete_on_success; + while (*p) { + if (strcmp(*p, slice_as_string(&new_table_path))) { + unlink(*p); + } + p++; + } + } + +exit: + free_names(delete_on_success); + { + char **p = subtable_locks; + while (*p) { + unlink(*p); + p++; + } + } + free_names(subtable_locks); + if (lock_file_fd > 0) { + close(lock_file_fd); + lock_file_fd = 0; + } + if (have_lock) { + unlink(slice_as_string(&lock_file_name)); + } + slice_clear(&new_table_name); + slice_clear(&new_table_path); + slice_clear(&ref_list_contents); + slice_clear(&temp_tab_file_name); + slice_clear(&lock_file_name); + return err; +} + +int reftable_stack_compact_all(struct reftable_stack *st, + struct reftable_log_expiry_config *config) +{ + return stack_compact_range(st, 0, st->merged->stack_len - 1, config); +} + +static int stack_compact_range_stats(struct reftable_stack *st, int first, + int last, + struct reftable_log_expiry_config *config) +{ + int err = stack_compact_range(st, first, last, config); + if (err > 0) { + st->stats.failures++; + } + return err; +} + +static int segment_size(struct segment *s) +{ + return s->end - s->start; +} + +int fastlog2(uint64_t sz) +{ + int l = 0; + if (sz == 0) { + return 0; + } + for (; sz; sz /= 2) { + l++; + } + return l - 1; +} + +struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n) +{ + struct segment *segs = reftable_calloc(sizeof(struct segment) * n); + int next = 0; + struct segment cur = { 0 }; + int i = 0; + + if (n == 0) { + *seglen = 0; + return segs; + } + for (i = 0; i < n; i++) { + int log = fastlog2(sizes[i]); + if (cur.log != log && cur.bytes > 0) { + struct segment fresh = { + .start = i, + }; + + segs[next++] = cur; + cur = fresh; + } + + cur.log = log; + cur.end = i + 1; + cur.bytes += sizes[i]; + } + segs[next++] = cur; + *seglen = next; + return segs; +} + +struct segment suggest_compaction_segment(uint64_t *sizes, int n) +{ + int seglen = 0; + struct segment *segs = sizes_to_segments(&seglen, sizes, n); + struct segment min_seg = { + .log = 64, + }; + int i = 0; + for (i = 0; i < seglen; i++) { + if (segment_size(&segs[i]) == 1) { + continue; + } + + if (segs[i].log < min_seg.log) { + min_seg = segs[i]; + } + } + + while (min_seg.start > 0) { + int prev = min_seg.start - 1; + if (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) { + break; + } + + min_seg.start = prev; + min_seg.bytes += sizes[prev]; + } + + reftable_free(segs); + return min_seg; +} + +static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) +{ + uint64_t *sizes = + reftable_calloc(sizeof(uint64_t) * st->merged->stack_len); + int version = (st->config.hash_id == SHA1_ID) ? 1 : 2; + int overhead = header_size(version) - 1; + int i = 0; + for (i = 0; i < st->merged->stack_len; i++) { + sizes[i] = st->merged->stack[i]->size - overhead; + } + return sizes; +} + +int reftable_stack_auto_compact(struct reftable_stack *st) +{ + uint64_t *sizes = stack_table_sizes_for_compaction(st); + struct segment seg = + suggest_compaction_segment(sizes, st->merged->stack_len); + reftable_free(sizes); + if (segment_size(&seg) > 0) { + return stack_compact_range_stats(st, seg.start, seg.end - 1, + NULL); + } + + return 0; +} + +struct reftable_compaction_stats * +reftable_stack_compaction_stats(struct reftable_stack *st) +{ + return &st->stats; +} + +int reftable_stack_read_ref(struct reftable_stack *st, const char *refname, + struct reftable_ref_record *ref) +{ + struct reftable_table tab = { NULL }; + reftable_table_from_merged_table(&tab, reftable_stack_merged_table(st)); + return reftable_table_read_ref(tab, refname, ref); +} + +int reftable_stack_read_log(struct reftable_stack *st, const char *refname, + struct reftable_log_record *log) +{ + struct reftable_iterator it = { 0 }; + struct reftable_merged_table *mt = reftable_stack_merged_table(st); + int err = reftable_merged_table_seek_log(mt, &it, refname); + if (err) { + goto exit; + } + + err = reftable_iterator_next_log(it, log); + if (err) { + goto exit; + } + + if (strcmp(log->ref_name, refname) || + reftable_log_record_is_deletion(log)) { + err = 1; + goto exit; + } + +exit: + if (err) { + reftable_log_record_clear(log); + } + reftable_iterator_destroy(&it); + return err; +} + +int stack_check_addition(struct reftable_stack *st, const char *new_tab_name) +{ + int err = 0; + struct reftable_block_source src = { 0 }; + struct reftable_reader *rd = NULL; + struct reftable_table tab = { NULL }; + struct reftable_ref_record *refs = NULL; + struct reftable_iterator it = { NULL }; + int cap = 0; + int len = 0; + int i = 0; + + if (st->config.skip_name_check) { + return 0; + } + + err = reftable_block_source_from_file(&src, new_tab_name); + if (err < 0) { + goto exit; + } + + err = reftable_new_reader(&rd, src, new_tab_name); + if (err < 0) { + goto exit; + } + + err = reftable_reader_seek_ref(rd, &it, ""); + if (err > 0) { + err = 0; + goto exit; + } + if (err < 0) { + goto exit; + } + + while (true) { + struct reftable_ref_record ref = { 0 }; + err = reftable_iterator_next_ref(it, &ref); + if (err > 0) { + break; + } + if (err < 0) { + goto exit; + } + + if (len >= cap) { + cap = 2 * cap + 1; + refs = reftable_realloc(refs, cap * sizeof(refs[0])); + } + + refs[len++] = ref; + } + + reftable_table_from_merged_table(&tab, reftable_stack_merged_table(st)); + + err = validate_ref_record_addition(tab, refs, len); + + for (i = 0; i < len; i++) { + reftable_ref_record_clear(&refs[i]); + } + +exit: + free(refs); + reftable_iterator_destroy(&it); + reftable_reader_free(rd); + return err; +} diff --git a/reftable/stack.h b/reftable/stack.h new file mode 100644 index 00000000000..48924c40f4b --- /dev/null +++ b/reftable/stack.h @@ -0,0 +1,45 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef STACK_H +#define STACK_H + +#include "reftable.h" +#include "system.h" + +struct reftable_stack { + char *list_file; + char *reftable_dir; + bool disable_auto_compact; + + struct reftable_write_options config; + + struct reftable_merged_table *merged; + struct reftable_compaction_stats stats; +}; + +int read_lines(const char *filename, char ***lines); +int stack_try_add(struct reftable_stack *st, + int (*write_table)(struct reftable_writer *wr, void *arg), + void *arg); +int stack_write_compact(struct reftable_stack *st, struct reftable_writer *wr, + int first, int last, + struct reftable_log_expiry_config *config); +int fastlog2(uint64_t sz); +int stack_check_addition(struct reftable_stack *st, const char *new_tab_name); + +struct segment { + int start, end; + int log; + uint64_t bytes; +}; + +struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n); +struct segment suggest_compaction_segment(uint64_t *sizes, int n); + +#endif diff --git a/reftable/system.h b/reftable/system.h new file mode 100644 index 00000000000..9e2c4b8d9ac --- /dev/null +++ b/reftable/system.h @@ -0,0 +1,54 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef SYSTEM_H +#define SYSTEM_H + +#if 1 /* REFTABLE_IN_GITCORE */ + +#include "git-compat-util.h" +#include "cache.h" +#include + +#else + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "compat.h" + +#endif /* REFTABLE_IN_GITCORE */ + +#define SHA1_ID 0x73686131 +#define SHA256_ID 0x73323536 +#define SHA1_SIZE 20 +#define SHA256_SIZE 32 + +typedef uint8_t byte; +typedef int bool; + +/* This is uncompress2, which is only available in zlib as of 2017. + * + * TODO: in git-core, this should fallback to uncompress2 if it is available. + */ +int uncompress_return_consumed(Bytef *dest, uLongf *destLen, + const Bytef *source, uLong *sourceLen); +int hash_size(uint32_t id); + +#endif diff --git a/reftable/tree.c b/reftable/tree.c new file mode 100644 index 00000000000..0341c865569 --- /dev/null +++ b/reftable/tree.c @@ -0,0 +1,67 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "tree.h" + +#include "basics.h" +#include "system.h" + +struct tree_node *tree_search(void *key, struct tree_node **rootp, + int (*compare)(const void *, const void *), + int insert) +{ + if (*rootp == NULL) { + if (!insert) { + return NULL; + } else { + struct tree_node *n = + reftable_calloc(sizeof(struct tree_node)); + n->key = key; + *rootp = n; + return *rootp; + } + } + + { + int res = compare(key, (*rootp)->key); + if (res < 0) { + return tree_search(key, &(*rootp)->left, compare, + insert); + } else if (res > 0) { + return tree_search(key, &(*rootp)->right, compare, + insert); + } + } + return *rootp; +} + +void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key), + void *arg) +{ + if (t->left != NULL) { + infix_walk(t->left, action, arg); + } + action(arg, t->key); + if (t->right != NULL) { + infix_walk(t->right, action, arg); + } +} + +void tree_free(struct tree_node *t) +{ + if (t == NULL) { + return; + } + if (t->left != NULL) { + tree_free(t->left); + } + if (t->right != NULL) { + tree_free(t->right); + } + reftable_free(t); +} diff --git a/reftable/tree.h b/reftable/tree.h new file mode 100644 index 00000000000..954512e9a3a --- /dev/null +++ b/reftable/tree.h @@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef TREE_H +#define TREE_H + +/* tree_node is a generic binary search tree. */ +struct tree_node { + void *key; + struct tree_node *left, *right; +}; + +/* looks for `key` in `rootp` using `compare` as comparison function. If insert + is set, insert the key if it's not found. Else, return NULL. +*/ +struct tree_node *tree_search(void *key, struct tree_node **rootp, + int (*compare)(const void *, const void *), + int insert); + +/* performs an infix walk of the tree. */ +void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key), + void *arg); + +/* + deallocates the tree nodes recursively. Keys should be deallocated separately + by walking over the tree. */ +void tree_free(struct tree_node *t); + +#endif diff --git a/reftable/update.sh b/reftable/update.sh new file mode 100755 index 00000000000..ef32aeda515 --- /dev/null +++ b/reftable/update.sh @@ -0,0 +1,24 @@ +#!/bin/sh + +set -eu + +# Override this to import from somewhere else, say "../reftable". +SRC=${SRC:-origin} +BRANCH=${BRANCH:-master} + +((git --git-dir reftable-repo/.git fetch ${SRC} ${BRANCH}:import && cd reftable-repo && git checkout -f $(git rev-parse import) ) || + git clone https://github.com/google/reftable reftable-repo) + +cp reftable-repo/c/*.[ch] reftable/ +cp reftable-repo/c/include/*.[ch] reftable/ +cp reftable-repo/LICENSE reftable/ +git --git-dir reftable-repo/.git show --no-patch HEAD \ + > reftable/VERSION + +mv reftable/system.h reftable/system.h~ +sed 's|if REFTABLE_IN_GITCORE|if 1 /* REFTABLE_IN_GITCORE */|' < reftable/system.h~ > reftable/system.h + +# Remove unittests and compatibility hacks we don't need here. +rm reftable/*_test.c reftable/test_framework.* reftable/compat.* + +git add reftable/*.[ch] reftable/LICENSE reftable/VERSION diff --git a/reftable/writer.c b/reftable/writer.c new file mode 100644 index 00000000000..bb70eba49da --- /dev/null +++ b/reftable/writer.c @@ -0,0 +1,661 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "writer.h" + +#include "system.h" + +#include "block.h" +#include "constants.h" +#include "record.h" +#include "reftable.h" +#include "tree.h" + +static struct reftable_block_stats * +writer_reftable_block_stats(struct reftable_writer *w, byte typ) +{ + switch (typ) { + case 'r': + return &w->stats.ref_stats; + case 'o': + return &w->stats.obj_stats; + case 'i': + return &w->stats.idx_stats; + case 'g': + return &w->stats.log_stats; + } + assert(false); + return NULL; +} + +/* write data, queuing the padding for the next write. Returns negative for + * error. */ +static int padded_write(struct reftable_writer *w, byte *data, size_t len, + int padding) +{ + int n = 0; + if (w->pending_padding > 0) { + byte *zeroed = reftable_calloc(w->pending_padding); + int n = w->write(w->write_arg, zeroed, w->pending_padding); + if (n < 0) { + return n; + } + + w->pending_padding = 0; + reftable_free(zeroed); + } + + w->pending_padding = padding; + n = w->write(w->write_arg, data, len); + if (n < 0) { + return n; + } + n += padding; + return 0; +} + +static void options_set_defaults(struct reftable_write_options *opts) +{ + if (opts->restart_interval == 0) { + opts->restart_interval = 16; + } + + if (opts->hash_id == 0) { + opts->hash_id = SHA1_ID; + } + if (opts->block_size == 0) { + opts->block_size = DEFAULT_BLOCK_SIZE; + } +} + +static int writer_version(struct reftable_writer *w) +{ + return (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2; +} + +static int writer_write_header(struct reftable_writer *w, byte *dest) +{ + memcpy((char *)dest, "REFT", 4); + + dest[4] = writer_version(w); + + put_be24(dest + 5, w->opts.block_size); + put_be64(dest + 8, w->min_update_index); + put_be64(dest + 16, w->max_update_index); + if (writer_version(w) == 2) { + put_be32(dest + 24, w->opts.hash_id); + } + return header_size(writer_version(w)); +} + +static void writer_reinit_block_writer(struct reftable_writer *w, byte typ) +{ + int block_start = 0; + if (w->next == 0) { + block_start = header_size(writer_version(w)); + } + + block_writer_init(&w->block_writer_data, typ, w->block, + w->opts.block_size, block_start, + hash_size(w->opts.hash_id)); + w->block_writer = &w->block_writer_data; + w->block_writer->restart_interval = w->opts.restart_interval; +} + +struct reftable_writer * +reftable_new_writer(int (*writer_func)(void *, byte *, size_t), + void *writer_arg, struct reftable_write_options *opts) +{ + struct reftable_writer *wp = + reftable_calloc(sizeof(struct reftable_writer)); + options_set_defaults(opts); + if (opts->block_size >= (1 << 24)) { + /* TODO - error return? */ + abort(); + } + wp->block = reftable_calloc(opts->block_size); + wp->write = writer_func; + wp->write_arg = writer_arg; + wp->opts = *opts; + writer_reinit_block_writer(wp, BLOCK_TYPE_REF); + + return wp; +} + +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min, + uint64_t max) +{ + w->min_update_index = min; + w->max_update_index = max; +} + +void reftable_writer_free(struct reftable_writer *w) +{ + reftable_free(w->block); + reftable_free(w); +} + +struct obj_index_tree_node { + struct slice hash; + uint64_t *offsets; + int offset_len; + int offset_cap; +}; + +static int obj_index_tree_node_compare(const void *a, const void *b) +{ + return slice_compare(((const struct obj_index_tree_node *)a)->hash, + ((const struct obj_index_tree_node *)b)->hash); +} + +static void writer_index_hash(struct reftable_writer *w, struct slice hash) +{ + uint64_t off = w->next; + + struct obj_index_tree_node want = { .hash = hash }; + + struct tree_node *node = tree_search(&want, &w->obj_index_tree, + &obj_index_tree_node_compare, 0); + struct obj_index_tree_node *key = NULL; + if (node == NULL) { + key = reftable_calloc(sizeof(struct obj_index_tree_node)); + slice_copy(&key->hash, hash); + tree_search((void *)key, &w->obj_index_tree, + &obj_index_tree_node_compare, 1); + } else { + key = node->key; + } + + if (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) { + return; + } + + if (key->offset_len == key->offset_cap) { + key->offset_cap = 2 * key->offset_cap + 1; + key->offsets = reftable_realloc( + key->offsets, sizeof(uint64_t) * key->offset_cap); + } + + key->offsets[key->offset_len++] = off; +} + +static int writer_add_record(struct reftable_writer *w, struct record rec) +{ + int result = -1; + struct slice key = { 0 }; + int err = 0; + record_key(rec, &key); + if (slice_compare(w->last_key, key) >= 0) { + goto exit; + } + + slice_copy(&w->last_key, key); + if (w->block_writer == NULL) { + writer_reinit_block_writer(w, record_type(rec)); + } + + assert(block_writer_type(w->block_writer) == record_type(rec)); + + if (block_writer_add(w->block_writer, rec) == 0) { + result = 0; + goto exit; + } + + err = writer_flush_block(w); + if (err < 0) { + result = err; + goto exit; + } + + writer_reinit_block_writer(w, record_type(rec)); + err = block_writer_add(w->block_writer, rec); + if (err < 0) { + result = err; + goto exit; + } + + result = 0; +exit: + slice_clear(&key); + return result; +} + +int reftable_writer_add_ref(struct reftable_writer *w, + struct reftable_ref_record *ref) +{ + struct record rec = { 0 }; + struct reftable_ref_record copy = *ref; + int err = 0; + + if (ref->ref_name == NULL) { + return REFTABLE_API_ERROR; + } + if (ref->update_index < w->min_update_index || + ref->update_index > w->max_update_index) { + return REFTABLE_API_ERROR; + } + + record_from_ref(&rec, ©); + copy.update_index -= w->min_update_index; + err = writer_add_record(w, rec); + if (err < 0) { + return err; + } + + if (!w->opts.skip_index_objects && ref->value != NULL) { + struct slice h = { + .buf = ref->value, + .len = hash_size(w->opts.hash_id), + }; + + writer_index_hash(w, h); + } + if (!w->opts.skip_index_objects && ref->target_value != NULL) { + struct slice h = { + .buf = ref->target_value, + .len = hash_size(w->opts.hash_id), + }; + writer_index_hash(w, h); + } + return 0; +} + +int reftable_writer_add_refs(struct reftable_writer *w, + struct reftable_ref_record *refs, int n) +{ + int err = 0; + int i = 0; + QSORT(refs, n, reftable_ref_record_compare_name); + for (i = 0; err == 0 && i < n; i++) { + err = reftable_writer_add_ref(w, &refs[i]); + } + return err; +} + +int reftable_writer_add_log(struct reftable_writer *w, + struct reftable_log_record *log) +{ + if (log->ref_name == NULL) { + return REFTABLE_API_ERROR; + } + + if (w->block_writer != NULL && + block_writer_type(w->block_writer) == BLOCK_TYPE_REF) { + int err = writer_finish_public_section(w); + if (err < 0) { + return err; + } + } + + w->next -= w->pending_padding; + w->pending_padding = 0; + + { + struct record rec = { 0 }; + int err; + record_from_log(&rec, log); + err = writer_add_record(w, rec); + return err; + } +} + +int reftable_writer_add_logs(struct reftable_writer *w, + struct reftable_log_record *logs, int n) +{ + int err = 0; + int i = 0; + QSORT(logs, n, reftable_log_record_compare_key); + for (i = 0; err == 0 && i < n; i++) { + err = reftable_writer_add_log(w, &logs[i]); + } + return err; +} + +static int writer_finish_section(struct reftable_writer *w) +{ + byte typ = block_writer_type(w->block_writer); + uint64_t index_start = 0; + int max_level = 0; + int threshold = w->opts.unpadded ? 1 : 3; + int before_blocks = w->stats.idx_stats.blocks; + int err = writer_flush_block(w); + int i = 0; + if (err < 0) { + return err; + } + + while (w->index_len > threshold) { + struct index_record *idx = NULL; + int idx_len = 0; + + max_level++; + index_start = w->next; + writer_reinit_block_writer(w, BLOCK_TYPE_INDEX); + + idx = w->index; + idx_len = w->index_len; + + w->index = NULL; + w->index_len = 0; + w->index_cap = 0; + for (i = 0; i < idx_len; i++) { + struct record rec = { 0 }; + record_from_index(&rec, idx + i); + if (block_writer_add(w->block_writer, rec) == 0) { + continue; + } + + { + int err = writer_flush_block(w); + if (err < 0) { + return err; + } + } + + writer_reinit_block_writer(w, BLOCK_TYPE_INDEX); + + err = block_writer_add(w->block_writer, rec); + assert(err == 0); + } + for (i = 0; i < idx_len; i++) { + slice_clear(&idx[i].last_key); + } + reftable_free(idx); + } + + writer_clear_index(w); + + err = writer_flush_block(w); + if (err < 0) { + return err; + } + + { + struct reftable_block_stats *bstats = + writer_reftable_block_stats(w, typ); + bstats->index_blocks = + w->stats.idx_stats.blocks - before_blocks; + bstats->index_offset = index_start; + bstats->max_index_level = max_level; + } + + /* Reinit lastKey, as the next section can start with any key. */ + w->last_key.len = 0; + + return 0; +} + +struct common_prefix_arg { + struct slice *last; + int max; +}; + +static void update_common(void *void_arg, void *key) +{ + struct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg; + struct obj_index_tree_node *entry = (struct obj_index_tree_node *)key; + if (arg->last != NULL) { + int n = common_prefix_size(entry->hash, *arg->last); + if (n > arg->max) { + arg->max = n; + } + } + arg->last = &entry->hash; +} + +struct write_record_arg { + struct reftable_writer *w; + int err; +}; + +static void write_object_record(void *void_arg, void *key) +{ + struct write_record_arg *arg = (struct write_record_arg *)void_arg; + struct obj_index_tree_node *entry = (struct obj_index_tree_node *)key; + struct obj_record obj_rec = { + .hash_prefix = entry->hash.buf, + .hash_prefix_len = arg->w->stats.object_id_len, + .offsets = entry->offsets, + .offset_len = entry->offset_len, + }; + struct record rec = { 0 }; + if (arg->err < 0) { + goto exit; + } + + record_from_obj(&rec, &obj_rec); + arg->err = block_writer_add(arg->w->block_writer, rec); + if (arg->err == 0) { + goto exit; + } + + arg->err = writer_flush_block(arg->w); + if (arg->err < 0) { + goto exit; + } + + writer_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ); + arg->err = block_writer_add(arg->w->block_writer, rec); + if (arg->err == 0) { + goto exit; + } + obj_rec.offset_len = 0; + arg->err = block_writer_add(arg->w->block_writer, rec); + + /* Should be able to write into a fresh block. */ + assert(arg->err == 0); + +exit:; +} + +static void object_record_free(void *void_arg, void *key) +{ + struct obj_index_tree_node *entry = (struct obj_index_tree_node *)key; + + FREE_AND_NULL(entry->offsets); + slice_clear(&entry->hash); + reftable_free(entry); +} + +static int writer_dump_object_index(struct reftable_writer *w) +{ + struct write_record_arg closure = { .w = w }; + struct common_prefix_arg common = { 0 }; + if (w->obj_index_tree != NULL) { + infix_walk(w->obj_index_tree, &update_common, &common); + } + w->stats.object_id_len = common.max + 1; + + writer_reinit_block_writer(w, BLOCK_TYPE_OBJ); + + if (w->obj_index_tree != NULL) { + infix_walk(w->obj_index_tree, &write_object_record, &closure); + } + + if (closure.err < 0) { + return closure.err; + } + return writer_finish_section(w); +} + +int writer_finish_public_section(struct reftable_writer *w) +{ + byte typ = 0; + int err = 0; + + if (w->block_writer == NULL) { + return 0; + } + + typ = block_writer_type(w->block_writer); + err = writer_finish_section(w); + if (err < 0) { + return err; + } + if (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects && + w->stats.ref_stats.index_blocks > 0) { + err = writer_dump_object_index(w); + if (err < 0) { + return err; + } + } + + if (w->obj_index_tree != NULL) { + infix_walk(w->obj_index_tree, &object_record_free, NULL); + tree_free(w->obj_index_tree); + w->obj_index_tree = NULL; + } + + w->block_writer = NULL; + return 0; +} + +int reftable_writer_close(struct reftable_writer *w) +{ + byte footer[72]; + byte *p = footer; + int err = writer_finish_public_section(w); + int empty_table = w->next == 0; + if (err != 0) { + goto exit; + } + w->pending_padding = 0; + if (empty_table) { + /* Empty tables need a header anyway. */ + byte header[28]; + int n = writer_write_header(w, header); + err = padded_write(w, header, n, 0); + if (err < 0) { + goto exit; + } + } + + p += writer_write_header(w, footer); + put_be64(p, w->stats.ref_stats.index_offset); + p += 8; + put_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len); + p += 8; + put_be64(p, w->stats.obj_stats.index_offset); + p += 8; + + put_be64(p, w->stats.log_stats.offset); + p += 8; + put_be64(p, w->stats.log_stats.index_offset); + p += 8; + + put_be32(p, crc32(0, footer, p - footer)); + p += 4; + + err = padded_write(w, footer, footer_size(writer_version(w)), 0); + if (err < 0) { + goto exit; + } + + if (empty_table) { + err = REFTABLE_EMPTY_TABLE_ERROR; + goto exit; + } + +exit: + /* free up memory. */ + block_writer_clear(&w->block_writer_data); + writer_clear_index(w); + slice_clear(&w->last_key); + return err; +} + +void writer_clear_index(struct reftable_writer *w) +{ + int i = 0; + for (i = 0; i < w->index_len; i++) { + slice_clear(&w->index[i].last_key); + } + + FREE_AND_NULL(w->index); + w->index_len = 0; + w->index_cap = 0; +} + +const int debug = 0; + +static int writer_flush_nonempty_block(struct reftable_writer *w) +{ + byte typ = block_writer_type(w->block_writer); + struct reftable_block_stats *bstats = + writer_reftable_block_stats(w, typ); + uint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0; + int raw_bytes = block_writer_finish(w->block_writer); + int padding = 0; + int err = 0; + if (raw_bytes < 0) { + return raw_bytes; + } + + if (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) { + padding = w->opts.block_size - raw_bytes; + } + + if (block_typ_off > 0) { + bstats->offset = block_typ_off; + } + + bstats->entries += w->block_writer->entries; + bstats->restarts += w->block_writer->restart_len; + bstats->blocks++; + w->stats.blocks++; + + if (debug) { + fprintf(stderr, "block %c off %" PRIu64 " sz %d (%d)\n", typ, + w->next, raw_bytes, + get_be24(w->block + w->block_writer->header_off + 1)); + } + + if (w->next == 0) { + writer_write_header(w, w->block); + } + + err = padded_write(w, w->block, raw_bytes, padding); + if (err < 0) { + return err; + } + + if (w->index_cap == w->index_len) { + w->index_cap = 2 * w->index_cap + 1; + w->index = reftable_realloc( + w->index, sizeof(struct index_record) * w->index_cap); + } + + { + struct index_record ir = { + .offset = w->next, + }; + slice_copy(&ir.last_key, w->block_writer->last_key); + w->index[w->index_len] = ir; + } + + w->index_len++; + w->next += padding + raw_bytes; + w->block_writer = NULL; + return 0; +} + +int writer_flush_block(struct reftable_writer *w) +{ + if (w->block_writer == NULL) { + return 0; + } + if (w->block_writer->entries == 0) { + return 0; + } + return writer_flush_nonempty_block(w); +} + +const struct reftable_stats *writer_stats(struct reftable_writer *w) +{ + return &w->stats; +} diff --git a/reftable/writer.h b/reftable/writer.h new file mode 100644 index 00000000000..afc15d8f31c --- /dev/null +++ b/reftable/writer.h @@ -0,0 +1,60 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef WRITER_H +#define WRITER_H + +#include "basics.h" +#include "block.h" +#include "reftable.h" +#include "slice.h" +#include "tree.h" + +struct reftable_writer { + int (*write)(void *, byte *, size_t); + void *write_arg; + int pending_padding; + struct slice last_key; + + /* offset of next block to write. */ + uint64_t next; + uint64_t min_update_index, max_update_index; + struct reftable_write_options opts; + + /* memory buffer for writing */ + byte *block; + + /* writer for the current section. NULL or points to + * block_writer_data */ + struct block_writer *block_writer; + + struct block_writer block_writer_data; + + /* pending index records for the current section */ + struct index_record *index; + int index_len; + int index_cap; + + /* + tree for use with tsearch; used to populate the 'o' inverse OID + map */ + struct tree_node *obj_index_tree; + + struct reftable_stats stats; +}; + +/* finishes a block, and writes it to storage */ +int writer_flush_block(struct reftable_writer *w); + +/* deallocates memory related to the index */ +void writer_clear_index(struct reftable_writer *w); + +/* finishes writing a 'r' (refs) or 'g' (reflogs) section */ +int writer_finish_public_section(struct reftable_writer *w); + +#endif diff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c new file mode 100644 index 00000000000..3e0b0f24f1c --- /dev/null +++ b/reftable/zlib-compat.c @@ -0,0 +1,92 @@ +/* taken from zlib's uncompr.c + + commit cacf7f1d4e3d44d871b605da3b647f07d718623f + Author: Mark Adler + Date: Sun Jan 15 09:18:46 2017 -0800 + + zlib 1.2.11 + +*/ + +/* + * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +#include "system.h" + +/* clang-format off */ + +/* =========================================================================== + Decompresses the source buffer into the destination buffer. *sourceLen is + the byte length of the source buffer. Upon entry, *destLen is the total size + of the destination buffer, which must be large enough to hold the entire + uncompressed data. (The size of the uncompressed data must have been saved + previously by the compressor and transmitted to the decompressor by some + mechanism outside the scope of this compression library.) Upon exit, + *destLen is the size of the decompressed data and *sourceLen is the number + of source bytes consumed. Upon return, source + *sourceLen points to the + first unused input byte. + + uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough + memory, Z_BUF_ERROR if there was not enough room in the output buffer, or + Z_DATA_ERROR if the input data was corrupted, including if the input data is + an incomplete zlib stream. +*/ +int ZEXPORT uncompress_return_consumed ( + Bytef *dest, + uLongf *destLen, + const Bytef *source, + uLong *sourceLen) { + z_stream stream; + int err; + const uInt max = (uInt)-1; + uLong len, left; + Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */ + + len = *sourceLen; + if (*destLen) { + left = *destLen; + *destLen = 0; + } + else { + left = 1; + dest = buf; + } + + stream.next_in = (z_const Bytef *)source; + stream.avail_in = 0; + stream.zalloc = (alloc_func)0; + stream.zfree = (free_func)0; + stream.opaque = (voidpf)0; + + err = inflateInit(&stream); + if (err != Z_OK) return err; + + stream.next_out = dest; + stream.avail_out = 0; + + do { + if (stream.avail_out == 0) { + stream.avail_out = left > (uLong)max ? max : (uInt)left; + left -= stream.avail_out; + } + if (stream.avail_in == 0) { + stream.avail_in = len > (uLong)max ? max : (uInt)len; + len -= stream.avail_in; + } + err = inflate(&stream, Z_NO_FLUSH); + } while (err == Z_OK); + + *sourceLen -= len + stream.avail_in; + if (dest != buf) + *destLen = stream.total_out; + else if (stream.total_out && err == Z_BUF_ERROR) + left = 1; + + inflateEnd(&stream); + return err == Z_STREAM_END ? Z_OK : + err == Z_NEED_DICT ? Z_DATA_ERROR : + err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR : + err; +} From patchwork Mon May 4 19:03:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527449 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E87C81 for ; Mon, 4 May 2020 19:04:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5CD192073B for ; Mon, 4 May 2020 19:04:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PJYwItq5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727878AbgEDTEH (ORCPT ); Mon, 4 May 2020 15:04:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727824AbgEDTEC (ORCPT ); Mon, 4 May 2020 15:04:02 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52112C061A10 for ; Mon, 4 May 2020 12:04:01 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id h4so682820wmb.4 for ; Mon, 04 May 2020 12:04:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=61pZEYYpoDlzVrSZzekS26q9ynTznn64Rmho/GNcpVs=; b=PJYwItq5FrkK+ZzhNHq2dqOaXvlhS+sFrPmgm4nMBlaSwLw58xvZGaEUwA5AYcHAuY RaQXlAN9KpyZLcAJ2SsjP2B5wxfqeMzUmmzYGaquiBBqnSSSCW7cVoL3WBeagyDJ0dql ZGzidM0mJBJ+/tVej37EQC4nufeHwHtJjyiVyRtvIGZUBf8AeCdU8UOs/JvNaq7c8qcP yPjE74z5GWq1XdTVNgJ+YVhxdYzSNIkQe/dE/AMThAGwahkNrPFf/quCUa/YH24ItTI+ yCx/tETUUkQ7v1kH0WbcXP57V8j9R/YDLQ+XLx92RCuPGVCLZU2hmWyphUohq4BHHdTM kc2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=61pZEYYpoDlzVrSZzekS26q9ynTznn64Rmho/GNcpVs=; b=sZb5/WnJO3F/arXjiOehZ/BDfTBj61TXXARrU+M1DH+CdbovzjSPBeOSQGXhBWsHnw i3y3MLWdRUT4k/4G7cck250ztl0hyUdpZw2qA5Jz0EC2t23QHcF7Oo11iKYbiZwy2qel Uvz8K8TsOYl5zFqI/pVWZnDx71reI7qx6tWl54biax3Qa8vLDo+mlrv9Ov5Cyr/y4TFA oxNFHyWhNbIXX4zqrNj0Nv2QD06DkgA70qfiNzEuvAuG8ghtF9Q4Pp7C+0QV/oud+CsX U4K4juKY6TYttBAVBxyP9FB8AjebFoiVJeFOztOG4kRIDUmV54sL7DPuCEWgKAFW83zP lG8w== X-Gm-Message-State: AGi0Pub0AT4ZH8kkJOpd+SSJ/F0GjYY/E8SC2DK6Lt31B+YiuqYMndFC 2pG3N1sevR0/6Z6eX5a3dBumpxj8 X-Google-Smtp-Source: APiQypJK0vBU1Nl+J6Ge9h5bSyzZ/irF13CQvzNGmVMo5N5gVR2xyD9+syaJ2Sjhd608aSm8MLgrqA== X-Received: by 2002:a7b:c306:: with SMTP id k6mr15605252wmj.40.1588619038553; Mon, 04 May 2020 12:03:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l6sm20872596wrb.75.2020.05.04.12.03.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:58 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:45 +0000 Subject: [PATCH v11 09/12] Reftable support for git-core Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys For background, see the previous commit introducing the library. TODO: * Resolve spots marked with XXX * Support worktrees (t0002-gitfile "linked repo" testcase) Example use: see t/t0031-reftable.sh Signed-off-by: Han-Wen Nienhuys Signed-off-by: Johannes Schindelin Co-authored-by: Jeff King --- .../technical/repository-version.txt | 7 + Makefile | 27 +- builtin/clone.c | 3 +- builtin/init-db.c | 56 +- cache.h | 6 +- refs.c | 27 +- refs.h | 3 + refs/refs-internal.h | 1 + refs/reftable-backend.c | 1045 +++++++++++++++++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- t/t0031-reftable.sh | 109 ++ 13 files changed, 1271 insertions(+), 30 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100755 t/t0031-reftable.sh diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt index 7844ef30ffd..72576235833 100644 --- a/Documentation/technical/repository-version.txt +++ b/Documentation/technical/repository-version.txt @@ -100,3 +100,10 @@ If set, by default "git config" reads from both "config" and multiple working directory mode, "config" file is shared while "config.worktree" is per-working directory (i.e., it's in GIT_COMMON_DIR/worktrees//config.worktree) + +==== `refStorage` + +Specifies the file format for the ref database. Values are `files` +(for the traditional packed + loose ref format) and `reftable` for the +binary reftable format. See https://github.com/google/reftable for +more information. diff --git a/Makefile b/Makefile index 3d3a39fc192..f4848a382f4 100644 --- a/Makefile +++ b/Makefile @@ -810,6 +810,7 @@ TEST_SHELL_PATH = $(SHELL_PATH) LIB_FILE = libgit.a XDIFF_LIB = xdiff/lib.a VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a GENERATED_H += config-list.h GENERATED_H += command-list.h @@ -962,6 +963,7 @@ LIB_OBJS += ref-filter.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o @@ -1165,7 +1167,7 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) +GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION) @@ -2360,11 +2362,29 @@ VCSSVN_OBJS += vcs-svn/sliding_window.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o +REFTABLE_OBJS += reftable/refname.o +REFTABLE_OBJS += reftable/reftable.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o +REFTABLE_OBJS += reftable/writer.o +REFTABLE_OBJS += reftable/zlib-compat.o + + TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ $(FUZZ_OBJS) \ + $(REFTABLE_OBJS) \ common-main.o \ git.o ifndef NO_CURL @@ -2505,6 +2525,9 @@ $(XDIFF_LIB): $(XDIFF_OBJS) $(VCSSVN_LIB): $(VCSSVN_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ +$(REFTABLE_LIB): $(REFTABLE_OBJS) + $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE @@ -3120,7 +3143,7 @@ cocciclean: clean: profile-clean coverage-clean cocciclean $(RM) *.res $(RM) $(OBJECTS) - $(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB) + $(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB) $(REFTABLE_LIB) $(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X $(RM) $(TEST_PROGRAMS) $(RM) $(FUZZ_PROGRAMS) diff --git a/builtin/clone.c b/builtin/clone.c index cb48a291caf..4d0cf065e4a 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1108,7 +1108,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix) } } - init_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, INIT_DB_QUIET); + init_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, + DEFAULT_REF_STORAGE, INIT_DB_QUIET); if (real_git_dir) git_dir = real_git_dir; diff --git a/builtin/init-db.c b/builtin/init-db.c index 0b7222e7188..b7053b9e370 100644 --- a/builtin/init-db.c +++ b/builtin/init-db.c @@ -178,7 +178,8 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree) return 1; } -void initialize_repository_version(int hash_algo) +void initialize_repository_version(int hash_algo, + const char *ref_storage_format) { char repo_version_string[10]; int repo_version = GIT_REPO_VERSION; @@ -188,7 +189,8 @@ void initialize_repository_version(int hash_algo) die(_("The hash algorithm %s is not supported in this build."), hash_algos[hash_algo].name); #endif - if (hash_algo != GIT_HASH_SHA1) + if (hash_algo != GIT_HASH_SHA1 || + !strcmp(ref_storage_format, "reftable")) repo_version = GIT_REPO_VERSION_READ; /* This forces creation of new config file */ @@ -238,6 +240,7 @@ static int create_default_files(const char *template_path, is_bare_repository_cfg = init_is_bare_repository; if (init_shared_repository != -1) set_shared_repository(init_shared_repository); + the_repository->ref_storage_format = xstrdup(fmt->ref_storage); /* * We would have created the above under user's umask -- under @@ -247,6 +250,24 @@ static int create_default_files(const char *template_path, adjust_shared_perm(get_git_dir()); } + /* + * Check to see if .git/HEAD exists; this must happen before + * initializing the ref db, because we want to see if there is an + * existing HEAD. + */ + path = git_path_buf(&buf, "HEAD"); + reinit = (!access(path, R_OK) || + readlink(path, junk, sizeof(junk) - 1) != -1); + + /* + * refs/heads is a file when using reftable. We can't reinitialize with + * a reftable because it will overwrite HEAD + */ + if (reinit && (!strcmp(fmt->ref_storage, "reftable")) == + is_directory(git_path_buf(&buf, "refs/heads"))) { + die("cannot switch ref storage format."); + } + /* * We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. @@ -261,15 +282,12 @@ static int create_default_files(const char *template_path, * Create the default symlink from ".git/HEAD" to the "master" * branch, if it does not exist yet. */ - path = git_path_buf(&buf, "HEAD"); - reinit = (!access(path, R_OK) - || readlink(path, junk, sizeof(junk)-1) != -1); if (!reinit) { if (create_symref("HEAD", "refs/heads/master", NULL) < 0) exit(1); } - initialize_repository_version(fmt->hash_algo); + initialize_repository_version(fmt->hash_algo, fmt->ref_storage); /* Check filemode trustability */ path = git_path_buf(&buf, "config"); @@ -383,7 +401,8 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash } int init_db(const char *git_dir, const char *real_git_dir, - const char *template_dir, int hash, unsigned int flags) + const char *template_dir, int hash, const char *ref_storage_format, + unsigned int flags) { int reinit; int exist_ok = flags & INIT_DB_EXIST_OK; @@ -422,6 +441,7 @@ int init_db(const char *git_dir, const char *real_git_dir, * is an attempt to reinitialize new repository with an old tool. */ check_repository_format(&repo_fmt); + repo_fmt.ref_storage = xstrdup(ref_storage_format); validate_hash_algorithm(&repo_fmt, hash); @@ -450,6 +470,8 @@ int init_db(const char *git_dir, const char *real_git_dir, git_config_set("receive.denyNonFastforwards", "true"); } + git_config_set("extensions.refStorage", ref_storage_format); + if (!(flags & INIT_DB_QUIET)) { int len = strlen(git_dir); @@ -523,6 +545,7 @@ static const char *const init_db_usage[] = { int cmd_init_db(int argc, const char **argv, const char *prefix) { const char *git_dir; + const char *ref_storage_format = DEFAULT_REF_STORAGE; const char *real_git_dir = NULL; const char *work_tree; const char *template_dir = NULL; @@ -530,15 +553,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix) const char *object_format = NULL; int hash_algo = GIT_HASH_UNKNOWN; const struct option init_db_options[] = { - OPT_STRING(0, "template", &template_dir, N_("template-directory"), - N_("directory from which templates will be used")), + OPT_STRING(0, "template", &template_dir, + N_("template-directory"), + N_("directory from which templates will be used")), OPT_SET_INT(0, "bare", &is_bare_repository_cfg, - N_("create a bare repository"), 1), + N_("create a bare repository"), 1), { OPTION_CALLBACK, 0, "shared", &init_shared_repository, - N_("permissions"), - N_("specify that the git repository is to be shared amongst several users"), - PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0}, + N_("permissions"), + N_("specify that the git repository is to be shared amongst several users"), + PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 }, OPT_BIT('q', "quiet", &flags, N_("be quiet"), INIT_DB_QUIET), + OPT_STRING(0, "ref-storage", &ref_storage_format, N_("backend"), + N_("the ref storage format to use")), OPT_STRING(0, "separate-git-dir", &real_git_dir, N_("gitdir"), N_("separate git dir from working tree")), OPT_STRING(0, "object-format", &object_format, N_("hash"), @@ -648,9 +674,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix) } UNLEAK(real_git_dir); + UNLEAK(ref_storage_format); UNLEAK(git_dir); UNLEAK(work_tree); flags |= INIT_DB_EXIST_OK; - return init_db(git_dir, real_git_dir, template_dir, hash_algo, flags); + return init_db(git_dir, real_git_dir, template_dir, hash_algo, + ref_storage_format, flags); } diff --git a/cache.h b/cache.h index 0f0485ecfe2..8cb884773c3 100644 --- a/cache.h +++ b/cache.h @@ -628,8 +628,9 @@ int path_inside_repo(const char *prefix, const char *path); int init_db(const char *git_dir, const char *real_git_dir, const char *template_dir, int hash_algo, - unsigned int flags); -void initialize_repository_version(int hash_algo); + const char *ref_storage_format, unsigned int flags); +void initialize_repository_version(int hash_algo, + const char *ref_storage_format); void sanitize_stdfds(void); int daemonize(void); @@ -1043,6 +1044,7 @@ struct repository_format { int is_bare; int hash_algo; char *work_tree; + char *ref_storage; struct string_list unknown_extensions; }; diff --git a/refs.c b/refs.c index 4db27379661..299a5db8bf1 100644 --- a/refs.c +++ b/refs.c @@ -17,10 +17,16 @@ #include "argv-array.h" #include "repository.h" +const char *default_ref_storage(void) +{ + const char *test = getenv("GIT_TEST_REFTABLE"); + return test ? "reftable" : "files"; +} + /* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable; static struct ref_storage_be *find_ref_storage_backend(const char *name) { @@ -1792,13 +1798,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map, * Create, record, and return a ref_store instance for the specified * gitdir. */ -static struct ref_store *ref_store_init(const char *gitdir, +static struct ref_store *ref_store_init(const char *gitdir, const char *be_name, unsigned int flags) { - const char *be_name = "files"; - struct ref_storage_be *be = find_ref_storage_backend(be_name); + struct ref_storage_be *be; struct ref_store *refs; + be = find_ref_storage_backend(be_name); if (!be) BUG("reference backend %s is unknown", be_name); @@ -1814,7 +1820,11 @@ struct ref_store *get_main_ref_store(struct repository *r) if (!r->gitdir) BUG("attempting to get main_ref_store outside of repository"); - r->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS); + r->refs_private = ref_store_init(r->gitdir, + r->ref_storage_format ? + r->ref_storage_format : + DEFAULT_REF_STORAGE, + REF_STORE_ALL_CAPS); return r->refs_private; } @@ -1869,7 +1879,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done; /* assume that add_submodule_odb() has been called */ - refs = ref_store_init(submodule_sb.buf, + refs = ref_store_init(submodule_sb.buf, DEFAULT_REF_STORAGE, /* XXX */ REF_STORE_READ | REF_STORE_ODB); register_ref_store_map(&submodule_ref_stores, "submodule", refs, submodule); @@ -1883,6 +1893,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) struct ref_store *get_worktree_ref_store(const struct worktree *wt) { + const char *format = DEFAULT_REF_STORAGE; /* XXX */ struct ref_store *refs; const char *id; @@ -1896,9 +1907,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt) if (wt->id) refs = ref_store_init(git_common_path("worktrees/%s", wt->id), - REF_STORE_ALL_CAPS); + format, REF_STORE_ALL_CAPS); else - refs = ref_store_init(get_git_common_dir(), + refs = ref_store_init(get_git_common_dir(), format, REF_STORE_ALL_CAPS); if (refs) diff --git a/refs.h b/refs.h index 9421c5b8465..15f6d78ee84 100644 --- a/refs.h +++ b/refs.h @@ -9,6 +9,9 @@ struct string_list; struct string_list_item; struct worktree; +/* Returns the ref storage backend to use by default. */ +const char *default_ref_storage(void); + /* * Resolve a reference, recursively following symbolic refererences. * diff --git a/refs/refs-internal.h b/refs/refs-internal.h index 3490aac3a40..cafe5b97376 100644 --- a/refs/refs-internal.h +++ b/refs/refs-internal.h @@ -661,6 +661,7 @@ struct ref_storage_be { }; extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable; extern struct ref_storage_be refs_be_packed; /* diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c new file mode 100644 index 00000000000..3ede7e837c1 --- /dev/null +++ b/refs/reftable-backend.c @@ -0,0 +1,1045 @@ +#include "../cache.h" +#include "../config.h" +#include "../refs.h" +#include "refs-internal.h" +#include "../iterator.h" +#include "../lockfile.h" +#include "../chdir-notify.h" + +#include "../reftable/reftable.h" + +extern struct ref_storage_be refs_be_reftable; + +struct git_reftable_ref_store { + struct ref_store base; + unsigned int store_flags; + + int err; + char *repo_dir; + char *reftable_dir; + struct reftable_stack *stack; +}; + +static void clear_reftable_log_record(struct reftable_log_record *log) +{ + log->old_hash = NULL; + log->new_hash = NULL; + log->message = NULL; + log->ref_name = NULL; + reftable_log_record_clear(log); +} + +static void fill_reftable_log_record(struct reftable_log_record *log) +{ + const char *info = git_committer_info(0); + struct ident_split split = { NULL }; + int result = split_ident_line(&split, info, strlen(info)); + int sign = 1; + assert(0 == result); + + reftable_log_record_clear(log); + log->name = + xstrndup(split.name_begin, split.name_end - split.name_begin); + log->email = + xstrndup(split.mail_begin, split.mail_end - split.mail_begin); + log->time = atol(split.date_begin); + if (*split.tz_begin == '-') { + sign = -1; + split.tz_begin++; + } + if (*split.tz_begin == '+') { + sign = 1; + split.tz_begin++; + } + + log->tz_offset = sign * atoi(split.tz_begin); +} + +static struct ref_store *git_reftable_ref_store_create(const char *path, + unsigned int store_flags) +{ + struct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs)); + struct ref_store *ref_store = (struct ref_store *)refs; + struct reftable_write_options cfg = { + .block_size = 4096, + .hash_id = the_hash_algo->format_id, + }; + struct strbuf sb = STRBUF_INIT; + + base_ref_store_init(ref_store, &refs_be_reftable); + refs->store_flags = store_flags; + refs->repo_dir = xstrdup(path); + strbuf_addf(&sb, "%s/reftable", path); + refs->reftable_dir = xstrdup(sb.buf); + strbuf_reset(&sb); + + refs->err = reftable_new_stack(&refs->stack, refs->reftable_dir, cfg); + strbuf_release(&sb); + return ref_store; +} + +static int reftable_init_db(struct ref_store *ref_store, struct strbuf *err) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct strbuf sb = STRBUF_INIT; + + safe_create_dir(refs->reftable_dir, 1); + + strbuf_addf(&sb, "%s/HEAD", refs->repo_dir); + write_file(sb.buf, "ref: refs/.invalid"); + strbuf_reset(&sb); + + strbuf_addf(&sb, "%s/refs", refs->repo_dir); + safe_create_dir(sb.buf, 1); + strbuf_reset(&sb); + + strbuf_addf(&sb, "%s/refs/heads", refs->repo_dir); + write_file(sb.buf, "this repository uses the reftable format"); + + return 0; +} + +struct git_reftable_iterator { + struct ref_iterator base; + struct reftable_iterator iter; + struct reftable_ref_record ref; + struct object_id oid; + struct ref_store *ref_store; + unsigned int flags; + int err; + const char *prefix; +}; + +static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator) +{ + struct git_reftable_iterator *ri = + (struct git_reftable_iterator *)ref_iterator; + while (ri->err == 0) { + ri->err = reftable_iterator_next_ref(ri->iter, &ri->ref); + if (ri->err) { + break; + } + + ri->base.refname = ri->ref.ref_name; + if (ri->prefix != NULL && + strncmp(ri->prefix, ri->ref.ref_name, strlen(ri->prefix))) { + ri->err = 1; + break; + } + if (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY && + ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE) + continue; + + ri->base.flags = 0; + if (ri->ref.value != NULL) { + hashcpy(ri->oid.hash, ri->ref.value); + } else if (ri->ref.target != NULL) { + int out_flags = 0; + const char *resolved = refs_resolve_ref_unsafe( + ri->ref_store, ri->ref.ref_name, + RESOLVE_REF_READING, &ri->oid, &out_flags); + ri->base.flags = out_flags; + if (resolved == NULL && + !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) && + (ri->base.flags & REF_ISBROKEN)) { + continue; + } + } + + ri->base.oid = &ri->oid; + if (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) && + !ref_resolves_to_object(ri->base.refname, ri->base.oid, + ri->base.flags)) { + continue; + } + + break; + } + + if (ri->err > 0) { + return ITER_DONE; + } + if (ri->err < 0) { + return ITER_ERROR; + } + + return ITER_OK; +} + +static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator, + struct object_id *peeled) +{ + struct git_reftable_iterator *ri = + (struct git_reftable_iterator *)ref_iterator; + if (ri->ref.target_value != NULL) { + hashcpy(peeled->hash, ri->ref.target_value); + return 0; + } + + return -1; +} + +static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator) +{ + struct git_reftable_iterator *ri = + (struct git_reftable_iterator *)ref_iterator; + reftable_ref_record_clear(&ri->ref); + reftable_iterator_destroy(&ri->iter); + return 0; +} + +static struct ref_iterator_vtable reftable_ref_iterator_vtable = { + reftable_ref_iterator_advance, reftable_ref_iterator_peel, + reftable_ref_iterator_abort +}; + +static struct ref_iterator * +reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix, + unsigned int flags) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri)); + struct reftable_merged_table *mt = NULL; + + if (refs->err < 0) { + ri->err = refs->err; + } else { + mt = reftable_stack_merged_table(refs->stack); + ri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix); + } + + base_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1); + ri->prefix = prefix; + ri->base.oid = &ri->oid; + ri->flags = flags; + ri->ref_store = ref_store; + return &ri->base; +} + +static int reftable_transaction_prepare(struct ref_store *ref_store, + struct ref_transaction *transaction, + struct strbuf *err) +{ + return 0; +} + +static int reftable_transaction_abort(struct ref_store *ref_store, + struct ref_transaction *transaction, + struct strbuf *err) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + (void)refs; + return 0; +} + +static int reftable_check_old_oid(struct ref_store *refs, const char *refname, + struct object_id *want_oid) +{ + struct object_id out_oid; + int out_flags = 0; + const char *resolved = refs_resolve_ref_unsafe( + refs, refname, RESOLVE_REF_READING, &out_oid, &out_flags); + if (is_null_oid(want_oid) != (resolved == NULL)) { + return REFTABLE_LOCK_ERROR; + } + + if (resolved != NULL && !oideq(&out_oid, want_oid)) { + return REFTABLE_LOCK_ERROR; + } + + return 0; +} + +static int ref_update_cmp(const void *a, const void *b) +{ + return strcmp(((struct ref_update *)a)->refname, + ((struct ref_update *)b)->refname); +} + +static int write_transaction_table(struct reftable_writer *writer, void *arg) +{ + struct ref_transaction *transaction = (struct ref_transaction *)arg; + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)transaction->ref_store; + uint64_t ts = reftable_stack_next_update_index(refs->stack); + int err = 0; + int i = 0; + struct reftable_log_record *logs = + calloc(transaction->nr, sizeof(*logs)); + struct ref_update **sorted = + malloc(transaction->nr * sizeof(struct ref_update *)); + COPY_ARRAY(sorted, transaction->updates, transaction->nr); + QSORT(sorted, transaction->nr, ref_update_cmp); + reftable_writer_set_limits(writer, ts, ts); + + for (i = 0; i < transaction->nr; i++) { + struct ref_update *u = sorted[i]; + if (u->flags & REF_HAVE_OLD) { + err = reftable_check_old_oid(transaction->ref_store, + u->refname, &u->old_oid); + if (err < 0) { + goto exit; + } + } + } + + for (i = 0; i < transaction->nr; i++) { + struct ref_update *u = sorted[i]; + struct reftable_log_record *log = &logs[i]; + fill_reftable_log_record(log); + log->ref_name = (char *)u->refname; + log->old_hash = u->old_oid.hash; + log->new_hash = u->new_oid.hash; + log->update_index = ts; + log->message = u->msg; + + if (u->flags & REF_HAVE_NEW) { + struct object_id out_oid; + int out_flags = 0; + /* Memory owned by refs_resolve_ref_unsafe, no need to + * free(). */ + const char *resolved = refs_resolve_ref_unsafe( + transaction->ref_store, u->refname, 0, &out_oid, + &out_flags); + struct reftable_ref_record ref = { NULL }; + struct object_id peeled; + int peel_error = peel_object(&u->new_oid, &peeled); + + ref.ref_name = + (char *)(resolved ? resolved : u->refname); + log->ref_name = ref.ref_name; + + if (!is_null_oid(&u->new_oid)) { + ref.value = u->new_oid.hash; + } + ref.update_index = ts; + if (!peel_error) { + ref.target_value = peeled.hash; + } + + err = reftable_writer_add_ref(writer, &ref); + if (err < 0) { + goto exit; + } + } + } + + for (i = 0; i < transaction->nr; i++) { + err = reftable_writer_add_log(writer, &logs[i]); + clear_reftable_log_record(&logs[i]); + if (err < 0) { + goto exit; + } + } + +exit: + free(logs); + free(sorted); + return err; +} + +static int reftable_transaction_commit(struct ref_store *ref_store, + struct ref_transaction *transaction, + struct strbuf *errmsg) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + int err = 0; + if (refs->err < 0) { + return refs->err; + } + + err = reftable_stack_add(refs->stack, &write_transaction_table, + transaction); + if (err < 0) { + strbuf_addf(errmsg, "reftable: transaction failure: %s", + reftable_error_str(err)); + return -1; + } + + return 0; +} + +static int reftable_transaction_finish(struct ref_store *ref_store, + struct ref_transaction *transaction, + struct strbuf *err) +{ + return reftable_transaction_commit(ref_store, transaction, err); +} + +struct write_delete_refs_arg { + struct reftable_stack *stack; + struct string_list *refnames; + const char *logmsg; + unsigned int flags; +}; + +static int write_delete_refs_table(struct reftable_writer *writer, void *argv) +{ + struct write_delete_refs_arg *arg = + (struct write_delete_refs_arg *)argv; + uint64_t ts = reftable_stack_next_update_index(arg->stack); + int err = 0; + int i = 0; + + reftable_writer_set_limits(writer, ts, ts); + for (i = 0; i < arg->refnames->nr; i++) { + struct reftable_ref_record ref = { + .ref_name = (char *)arg->refnames->items[i].string, + .update_index = ts, + }; + err = reftable_writer_add_ref(writer, &ref); + if (err < 0) { + return err; + } + } + + for (i = 0; i < arg->refnames->nr; i++) { + struct reftable_log_record log = { NULL }; + struct reftable_ref_record current = { NULL }; + fill_reftable_log_record(&log); + log.message = xstrdup(arg->logmsg); + log.new_hash = NULL; + log.old_hash = NULL; + log.update_index = ts; + log.ref_name = (char *)arg->refnames->items[i].string; + + if (reftable_stack_read_ref(arg->stack, log.ref_name, + ¤t) == 0) { + log.old_hash = current.value; + } + err = reftable_writer_add_log(writer, &log); + log.old_hash = NULL; + reftable_ref_record_clear(¤t); + + clear_reftable_log_record(&log); + if (err < 0) { + return err; + } + } + return 0; +} + +static int reftable_delete_refs(struct ref_store *ref_store, const char *msg, + struct string_list *refnames, + unsigned int flags) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct write_delete_refs_arg arg = { + .stack = refs->stack, + .refnames = refnames, + .logmsg = msg, + .flags = flags, + }; + if (refs->err < 0) { + return refs->err; + } + + return reftable_stack_add(refs->stack, &write_delete_refs_table, &arg); +} + +static int reftable_pack_refs(struct ref_store *ref_store, unsigned int flags) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + if (refs->err < 0) { + return refs->err; + } + return reftable_stack_compact_all(refs->stack, NULL); +} + +struct write_create_symref_arg { + struct git_reftable_ref_store *refs; + const char *refname; + const char *target; + const char *logmsg; +}; + +static int write_create_symref_table(struct reftable_writer *writer, void *arg) +{ + struct write_create_symref_arg *create = + (struct write_create_symref_arg *)arg; + uint64_t ts = reftable_stack_next_update_index(create->refs->stack); + int err = 0; + + struct reftable_ref_record ref = { + .ref_name = (char *)create->refname, + .target = (char *)create->target, + .update_index = ts, + }; + reftable_writer_set_limits(writer, ts, ts); + err = reftable_writer_add_ref(writer, &ref); + if (err < 0) { + return err; + } + + { + struct reftable_log_record log = { NULL }; + struct object_id new_oid; + struct object_id old_oid; + struct reftable_ref_record current = { NULL }; + reftable_stack_read_ref(create->refs->stack, create->refname, + ¤t); + + fill_reftable_log_record(&log); + log.ref_name = current.ref_name; + if (refs_resolve_ref_unsafe( + (struct ref_store *)create->refs, create->refname, + RESOLVE_REF_READING, &old_oid, NULL) != NULL) { + log.old_hash = old_oid.hash; + } + + if (refs_resolve_ref_unsafe((struct ref_store *)create->refs, + create->target, RESOLVE_REF_READING, + &new_oid, NULL) != NULL) { + log.new_hash = new_oid.hash; + } + + if (log.old_hash != NULL || log.new_hash != NULL) { + reftable_writer_add_log(writer, &log); + } + log.ref_name = NULL; + log.old_hash = NULL; + log.new_hash = NULL; + clear_reftable_log_record(&log); + } + return 0; +} + +static int reftable_create_symref(struct ref_store *ref_store, + const char *refname, const char *target, + const char *logmsg) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct write_create_symref_arg arg = { .refs = refs, + .refname = refname, + .target = target, + .logmsg = logmsg }; + if (refs->err < 0) { + return refs->err; + } + return reftable_stack_add(refs->stack, &write_create_symref_table, + &arg); +} + +struct write_rename_arg { + struct reftable_stack *stack; + const char *oldname; + const char *newname; + const char *logmsg; +}; + +static int write_rename_table(struct reftable_writer *writer, void *argv) +{ + struct write_rename_arg *arg = (struct write_rename_arg *)argv; + uint64_t ts = reftable_stack_next_update_index(arg->stack); + struct reftable_ref_record ref = { NULL }; + int err = reftable_stack_read_ref(arg->stack, arg->oldname, &ref); + + if (err) { + goto exit; + } + + /* XXX do ref renames overwrite the target? */ + if (reftable_stack_read_ref(arg->stack, arg->newname, &ref) == 0) { + goto exit; + } + + free(ref.ref_name); + ref.ref_name = strdup(arg->newname); + reftable_writer_set_limits(writer, ts, ts); + ref.update_index = ts; + + { + struct reftable_ref_record todo[2] = { { NULL } }; + todo[0].ref_name = (char *)arg->oldname; + todo[0].update_index = ts; + /* leave todo[0] empty */ + todo[1] = ref; + todo[1].update_index = ts; + + err = reftable_writer_add_refs(writer, todo, 2); + if (err < 0) { + goto exit; + } + } + + if (ref.value != NULL) { + struct reftable_log_record todo[2] = { { NULL } }; + fill_reftable_log_record(&todo[0]); + fill_reftable_log_record(&todo[1]); + + todo[0].ref_name = (char *)arg->oldname; + todo[0].update_index = ts; + todo[0].message = (char *)arg->logmsg; + todo[0].old_hash = ref.value; + todo[0].new_hash = NULL; + + todo[1].ref_name = (char *)arg->newname; + todo[1].update_index = ts; + todo[1].old_hash = NULL; + todo[1].new_hash = ref.value; + todo[1].message = (char *)arg->logmsg; + + err = reftable_writer_add_logs(writer, todo, 2); + + clear_reftable_log_record(&todo[0]); + clear_reftable_log_record(&todo[1]); + + if (err < 0) { + goto exit; + } + + } else { + /* XXX symrefs? */ + } + +exit: + reftable_ref_record_clear(&ref); + return err; +} + +static int reftable_rename_ref(struct ref_store *ref_store, + const char *oldrefname, const char *newrefname, + const char *logmsg) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct write_rename_arg arg = { + .stack = refs->stack, + .oldname = oldrefname, + .newname = newrefname, + .logmsg = logmsg, + }; + if (refs->err < 0) { + return refs->err; + } + + return reftable_stack_add(refs->stack, &write_rename_table, &arg); +} + +static int reftable_copy_ref(struct ref_store *ref_store, + const char *oldrefname, const char *newrefname, + const char *logmsg) +{ + BUG("reftable reference store does not support copying references"); +} + +struct reftable_reflog_ref_iterator { + struct ref_iterator base; + struct reftable_iterator iter; + struct reftable_log_record log; + struct object_id oid; + char *last_name; +}; + +static int +reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator) +{ + struct reftable_reflog_ref_iterator *ri = + (struct reftable_reflog_ref_iterator *)ref_iterator; + + while (1) { + int err = reftable_iterator_next_log(ri->iter, &ri->log); + if (err > 0) { + return ITER_DONE; + } + if (err < 0) { + return ITER_ERROR; + } + + ri->base.refname = ri->log.ref_name; + if (ri->last_name != NULL && + !strcmp(ri->log.ref_name, ri->last_name)) { + /* we want the refnames that we have reflogs for, so we + * skip if we've already produced this name. This could + * be faster by seeking directly to + * reflog@update_index==0. + */ + continue; + } + + free(ri->last_name); + ri->last_name = xstrdup(ri->log.ref_name); + hashcpy(ri->oid.hash, ri->log.new_hash); + return ITER_OK; + } +} + +static int reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator, + struct object_id *peeled) +{ + BUG("not supported."); + return -1; +} + +static int reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator) +{ + struct reftable_reflog_ref_iterator *ri = + (struct reftable_reflog_ref_iterator *)ref_iterator; + reftable_log_record_clear(&ri->log); + reftable_iterator_destroy(&ri->iter); + return 0; +} + +static struct ref_iterator_vtable reftable_reflog_ref_iterator_vtable = { + reftable_reflog_ref_iterator_advance, reftable_reflog_ref_iterator_peel, + reftable_reflog_ref_iterator_abort +}; + +static struct ref_iterator * +reftable_reflog_iterator_begin(struct ref_store *ref_store) +{ + struct reftable_reflog_ref_iterator *ri = xcalloc(sizeof(*ri), 1); + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + + struct reftable_merged_table *mt = + reftable_stack_merged_table(refs->stack); + int err = reftable_merged_table_seek_log(mt, &ri->iter, ""); + if (err < 0) { + free(ri); + return NULL; + } + + base_ref_iterator_init(&ri->base, &reftable_reflog_ref_iterator_vtable, + 1); + ri->base.oid = &ri->oid; + + return (struct ref_iterator *)ri; +} + +static int +reftable_for_each_reflog_ent_newest_first(struct ref_store *ref_store, + const char *refname, + each_reflog_ent_fn fn, void *cb_data) +{ + struct reftable_iterator it = { NULL }; + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct reftable_merged_table *mt = NULL; + int err = 0; + struct reftable_log_record log = { NULL }; + + if (refs->err < 0) { + return refs->err; + } + + mt = reftable_stack_merged_table(refs->stack); + err = reftable_merged_table_seek_log(mt, &it, refname); + while (err == 0) { + err = reftable_iterator_next_log(it, &log); + if (err != 0) { + break; + } + + if (strcmp(log.ref_name, refname)) { + break; + } + + { + struct object_id old_oid; + struct object_id new_oid; + const char *full_committer = ""; + + hashcpy(old_oid.hash, log.old_hash); + hashcpy(new_oid.hash, log.new_hash); + + full_committer = fmt_ident(log.name, log.email, + WANT_COMMITTER_IDENT, + /*date*/ NULL, + IDENT_NO_DATE); + if (fn(&old_oid, &new_oid, full_committer, log.time, + log.tz_offset, log.message, cb_data)) { + err = -1; + break; + } + } + } + + reftable_log_record_clear(&log); + reftable_iterator_destroy(&it); + if (err > 0) { + err = 0; + } + return err; +} + +static int +reftable_for_each_reflog_ent_oldest_first(struct ref_store *ref_store, + const char *refname, + each_reflog_ent_fn fn, void *cb_data) +{ + struct reftable_iterator it = { NULL }; + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct reftable_merged_table *mt = NULL; + struct reftable_log_record *logs = NULL; + int cap = 0; + int len = 0; + int err = 0; + int i = 0; + + if (refs->err < 0) { + return refs->err; + } + mt = reftable_stack_merged_table(refs->stack); + err = reftable_merged_table_seek_log(mt, &it, refname); + + while (err == 0) { + struct reftable_log_record log = { NULL }; + err = reftable_iterator_next_log(it, &log); + if (err != 0) { + break; + } + + if (strcmp(log.ref_name, refname)) { + break; + } + + if (len == cap) { + cap = 2 * cap + 1; + logs = realloc(logs, cap * sizeof(*logs)); + } + + logs[len++] = log; + } + + for (i = len; i--;) { + struct reftable_log_record *log = &logs[i]; + struct object_id old_oid; + struct object_id new_oid; + const char *full_committer = ""; + + hashcpy(old_oid.hash, log->old_hash); + hashcpy(new_oid.hash, log->new_hash); + + full_committer = fmt_ident(log->name, log->email, + WANT_COMMITTER_IDENT, NULL, + IDENT_NO_DATE); + if (!fn(&old_oid, &new_oid, full_committer, log->time, + log->tz_offset, log->message, cb_data)) { + err = -1; + break; + } + } + + for (i = 0; i < len; i++) { + reftable_log_record_clear(&logs[i]); + } + free(logs); + + reftable_iterator_destroy(&it); + if (err > 0) { + err = 0; + } + return err; +} + +static int reftable_reflog_exists(struct ref_store *ref_store, + const char *refname) +{ + /* always exists. */ + return 1; +} + +static int reftable_create_reflog(struct ref_store *ref_store, + const char *refname, int force_create, + struct strbuf *err) +{ + return 0; +} + +static int reftable_delete_reflog(struct ref_store *ref_store, + const char *refname) +{ + return 0; +} + +struct reflog_expiry_arg { + struct git_reftable_ref_store *refs; + struct reftable_log_record *tombstones; + int len; + int cap; +}; + +static void clear_log_tombstones(struct reflog_expiry_arg *arg) +{ + int i = 0; + for (; i < arg->len; i++) { + reftable_log_record_clear(&arg->tombstones[i]); + } + + FREE_AND_NULL(arg->tombstones); +} + +static void add_log_tombstone(struct reflog_expiry_arg *arg, + const char *refname, uint64_t ts) +{ + struct reftable_log_record tombstone = { + .ref_name = xstrdup(refname), + .update_index = ts, + }; + if (arg->len == arg->cap) { + arg->cap = 2 * arg->cap + 1; + arg->tombstones = + realloc(arg->tombstones, arg->cap * sizeof(tombstone)); + } + arg->tombstones[arg->len++] = tombstone; +} + +static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv) +{ + struct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv; + uint64_t ts = reftable_stack_next_update_index(arg->refs->stack); + int i = 0; + reftable_writer_set_limits(writer, ts, ts); + for (i = 0; i < arg->len; i++) { + int err = reftable_writer_add_log(writer, &arg->tombstones[i]); + if (err) { + return err; + } + } + return 0; +} + +static int reftable_reflog_expire(struct ref_store *ref_store, + const char *refname, + const struct object_id *oid, + unsigned int flags, + reflog_expiry_prepare_fn prepare_fn, + reflog_expiry_should_prune_fn should_prune_fn, + reflog_expiry_cleanup_fn cleanup_fn, + void *policy_cb_data) +{ + /* + For log expiry, we write tombstones in place of the expired entries, + This means that the entries are still retrievable by delving into the + stack, and expiring entries paradoxically takes extra memory. + + This memory is only reclaimed when some operation issues a + reftable_pack_refs(), which will compact the entire stack and get rid + of deletion entries. + + It would be better if the refs backend supported an API that sets a + criterion for all refs, passing the criterion to pack_refs(). + */ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct reftable_merged_table *mt = NULL; + struct reflog_expiry_arg arg = { + .refs = refs, + }; + struct reftable_log_record log = { NULL }; + struct reftable_iterator it = { NULL }; + int err = 0; + if (refs->err < 0) { + return refs->err; + } + + mt = reftable_stack_merged_table(refs->stack); + err = reftable_merged_table_seek_log(mt, &it, refname); + if (err < 0) { + return err; + } + + while (1) { + struct object_id ooid; + struct object_id noid; + + int err = reftable_iterator_next_log(it, &log); + if (err < 0) { + return err; + } + + if (err > 0 || strcmp(log.ref_name, refname)) { + break; + } + hashcpy(ooid.hash, log.old_hash); + hashcpy(noid.hash, log.new_hash); + + if (should_prune_fn(&ooid, &noid, log.email, + (timestamp_t)log.time, log.tz_offset, + log.message, policy_cb_data)) { + add_log_tombstone(&arg, refname, log.update_index); + } + } + reftable_log_record_clear(&log); + reftable_iterator_destroy(&it); + err = reftable_stack_add(refs->stack, &write_reflog_expiry_table, &arg); + clear_log_tombstones(&arg); + return err; +} + +static int reftable_read_raw_ref(struct ref_store *ref_store, + const char *refname, struct object_id *oid, + struct strbuf *referent, unsigned int *type) +{ + struct git_reftable_ref_store *refs = + (struct git_reftable_ref_store *)ref_store; + struct reftable_ref_record ref = { NULL }; + int err = 0; + if (refs->err < 0) { + return refs->err; + } + + err = reftable_stack_read_ref(refs->stack, refname, &ref); + if (err > 0) { + errno = ENOENT; + err = -1; + goto exit; + } + if (err < 0) { + errno = reftable_error_to_errno(err); + err = -1; + goto exit; + } + if (ref.target != NULL) { + /* XXX recurse? */ + strbuf_reset(referent); + strbuf_addstr(referent, ref.target); + *type |= REF_ISSYMREF; + } else if (ref.value != NULL) { + hashcpy(oid->hash, ref.value); + } else { + *type |= REF_ISBROKEN; + errno = EINVAL; + err = -1; + } +exit: + reftable_ref_record_clear(&ref); + return err; +} + +struct ref_storage_be refs_be_reftable = { + &refs_be_files, + "reftable", + git_reftable_ref_store_create, + reftable_init_db, + reftable_transaction_prepare, + reftable_transaction_finish, + reftable_transaction_abort, + reftable_transaction_commit, + + reftable_pack_refs, + reftable_create_symref, + reftable_delete_refs, + reftable_rename_ref, + reftable_copy_ref, + + reftable_ref_iterator_begin, + reftable_read_raw_ref, + + reftable_reflog_iterator_begin, + reftable_for_each_reflog_ent_newest_first, + reftable_for_each_reflog_ent_oldest_first, + reftable_reflog_exists, + reftable_create_reflog, + reftable_delete_reflog, + reftable_reflog_expire +}; diff --git a/repository.c b/repository.c index 6f7f6f002b1..087760bc184 100644 --- a/repository.c +++ b/repository.c @@ -178,6 +178,8 @@ int repo_init(struct repository *repo, if (worktree) repo_set_worktree(repo, worktree); + repo->ref_storage_format = xstrdup_or_null(format.ref_storage); + clear_repository_format(&format); return 0; diff --git a/repository.h b/repository.h index 6534fbb7b31..f57b73f4a27 100644 --- a/repository.h +++ b/repository.h @@ -74,6 +74,9 @@ struct repository { */ struct ref_store *refs_private; + /* The format to use for the ref database. */ + char *ref_storage_format; + /* * Contains path to often used file names. */ diff --git a/setup.c b/setup.c index 65fe5ecefbe..2ef970f9f88 100644 --- a/setup.c +++ b/setup.c @@ -468,9 +468,11 @@ static int check_repo_format(const char *var, const char *value, void *vdata) if (!value) return config_error_nonbool(var); data->partial_clone = xstrdup(value); - } else if (!strcmp(ext, "worktreeconfig")) + } else if (!strcmp(ext, "worktreeconfig")) { data->worktree_config = git_config_bool(var, value); - else + } else if (!strcmp(ext, "refstorage")) { + data->ref_storage = xstrdup(value); + } else string_list_append(&data->unknown_extensions, ext); } @@ -559,6 +561,7 @@ void clear_repository_format(struct repository_format *format) string_list_clear(&format->unknown_extensions, 0); free(format->work_tree); free(format->partial_clone); + free(format->ref_storage); init_repository_format(format); } @@ -1204,8 +1207,11 @@ const char *setup_git_directory_gently(int *nongit_ok) gitdir = DEFAULT_GIT_DIR_ENVIRONMENT; setup_git_env(gitdir); } - if (startup_info->have_repository) + if (startup_info->have_repository) { repo_set_hash_algo(the_repository, repo_fmt.hash_algo); + the_repository->ref_storage_format = + xstrdup_or_null(repo_fmt.ref_storage); + } } strbuf_release(&dir); diff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh new file mode 100755 index 00000000000..7510d24a932 --- /dev/null +++ b/t/t0031-reftable.sh @@ -0,0 +1,109 @@ +#!/bin/sh +# +# Copyright (c) 2020 Google LLC +# + +test_description='reftable basics' + +. ./test-lib.sh + +INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + +initialize () { + rm -rf .git && + git init --ref-storage=reftable && + mv .git/hooks .git/hooks-disabled +} + +test_expect_success 'delete ref' ' + initialize && + test_commit file && + SHA=$(git show-ref -s --verify HEAD) && + test_write_lines "$SHA refs/heads/master" "$SHA refs/tags/file" >expect && + git show-ref > actual && + ! git update-ref -d refs/tags/file $INVALID_SHA1 && + test_cmp expect actual && + git update-ref -d refs/tags/file $SHA && + test_write_lines "$SHA refs/heads/master" >expect && + git show-ref > actual && + test_cmp expect actual +' + +test_expect_success 'basic operation of reftable storage: commit, reflog, repack' ' + initialize && + test_commit file && + test_write_lines refs/heads/master refs/tags/file >expect && + git show-ref && + git show-ref | cut -f2 -d" " > actual && + test_cmp actual expect && + for count in $(test_seq 1 10) + do + test_commit "number $count" file.t $count number-$count || + return 1 + done && + git pack-refs && + ls -1 .git/reftable >table-files && + test_line_count = 2 table-files && + git reflog refs/heads/master >output && + test_line_count = 11 output && + grep "commit (initial): file" output && + grep "commit: number 10" output && + git gc && + git reflog refs/heads/master >output && + test_line_count = 0 output +' + +# This matches show-ref's output +print_ref() { + echo "$(git rev-parse "$1") $1" +} + +test_expect_success 'peeled tags are stored' ' + initialize && + test_commit file && + git tag -m "annotated tag" test_tag HEAD && + { + print_ref "refs/heads/master" && + print_ref "refs/tags/file" && + print_ref "refs/tags/test_tag" && + print_ref "refs/tags/test_tag^{}" + } >expect && + git show-ref -d >actual && + test_cmp expect actual +' + +test_expect_success 'show-ref works on fresh repo' ' + initialize && + rm -rf .git && + git init --ref-storage=reftable && + >expect && + ! git show-ref > actual && + test_cmp expect actual +' + +test_expect_success 'checkout unborn branch' ' + initialize && + git checkout -b master +' + + +test_expect_success 'dir/file conflict' ' + initialize && + test_commit file && + ! git branch master/forbidden +' + + +test_expect_success 'do not clobber existing repo' ' + rm -rf .git && + git init --ref-storage=files && + cat .git/HEAD > expect && + test_commit file && + (git init --ref-storage=reftable || true) && + cat .git/HEAD > actual && + test_cmp expect actual +' + + +test_done + From patchwork Mon May 4 19:03:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527443 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C98981 for ; Mon, 4 May 2020 19:04:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 537DB20721 for ; Mon, 4 May 2020 19:04:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MySOvlu9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727849AbgEDTED (ORCPT ); Mon, 4 May 2020 15:04:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727828AbgEDTEB (ORCPT ); Mon, 4 May 2020 15:04:01 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A170FC061A0F for ; Mon, 4 May 2020 12:04:00 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id h4so682792wmb.4 for ; Mon, 04 May 2020 12:04:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Vup9MT8sfUcv6xkbQAWR68a1iVgFiIUVwHGrIXy026Q=; b=MySOvlu9Vl0jXfVuSCNV5tdDfhml3zjLM+OO54s9nDKuG4N+lDtQ6ixg1jFTXeUJAq ngH3Lk/iroQGujPEbWaRZDZTKliKiQvL0d8/0wJgHAl/602OrbtC9oNhE1gTojhKULyu QgYXPJcpHpW0Rsea2avO/a7yPYAY286iL0iLqKc+b/3RPhMN0eYPTyJdJFYObWyPgPdX izfjEtW5uceuUbVoB3s3mCa/DCMOiKd2/3SSB8JPoo/+ciszHOch+9CtsSJ4ewiynif6 fOyqO+oE9k8W2M4N+uCSNHWTXBcrnV1kz4kKkxMhVhlhiFkofs/s6q9okYu6STUVeAPx olZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Vup9MT8sfUcv6xkbQAWR68a1iVgFiIUVwHGrIXy026Q=; b=Y6F+8IDXFO86q8x6jDObUHxMbB0rgfgOwJt3L0Ed0rtFZm7oTrdtUWL5bd+e6aImJ7 Dfn8Zm3YzT57T2a1LatvOIHz9nbDoQZatw8MpprlmlMEPxlB21/xTlGSJrXyQNCVac89 x4rS5/g3ccurGDZEp7uT3+XHWDCybCr1fdrJh4BdJMR3PTd5GO69tf91DOTgQ60mkUsc 70Koz9dIskEsazrQcS6dr0se1ityS4qlOzROGc0O+QN2TPx40DjpS4f07SokLGw+rSxR Zu9BFL7PrSxxuD/axDtTl+wPjaF0+s/tEm6BAU29xT+eGB73ysXZOdWptaxxarqDefr0 O16w== X-Gm-Message-State: AGi0PuYQ2XlFE3/3Gf+3Wk3yfVRjjuTUhNNE2Z2IUhg5rtP7t7kwE+t1 kH6LUA24vIula4f6LraNi71hSlj3 X-Google-Smtp-Source: APiQypInO7z+eklQqJq/itGmZnzNNVngN9IdnlvQkDeau5BaQguankjpQQipJFIIBKH5xYCP/5r57w== X-Received: by 2002:a1c:bc08:: with SMTP id m8mr15966073wmf.119.1588619039222; Mon, 04 May 2020 12:03:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w6sm13431303wrt.39.2020.05.04.12.03.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:58 -0700 (PDT) Message-Id: <513d585f0f8ad33948ea72b1f8cdb87125840cf0.1588619029.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Date: Mon, 04 May 2020 19:03:46 +0000 Subject: [PATCH v11 10/12] vcxproj: adjust for the reftable changes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin This allows Git to be compiled via Visual Studio again after integrating the `hn/reftable` branch. Signed-off-by: Johannes Schindelin --- config.mak.uname | 2 +- contrib/buildsystems/Generators/Vcxproj.pm | 11 ++++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/config.mak.uname b/config.mak.uname index 5ad43c80b1a..484aec15ac2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -710,7 +710,7 @@ vcxproj: # Make .vcxproj files and add them unset QUIET_GEN QUIET_BUILT_IN; \ perl contrib/buildsystems/generate -g Vcxproj - git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj + git add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj # Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file (echo '' && \ diff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm index 5c666f9ac03..33a08d31652 100644 --- a/contrib/buildsystems/Generators/Vcxproj.pm +++ b/contrib/buildsystems/Generators/Vcxproj.pm @@ -77,7 +77,7 @@ sub createProject { my $libs_release = "\n "; my $libs_debug = "\n "; if (!$static_library) { - $libs_release = join(";", sort(grep /^(?!libgit\.lib|xdiff\/lib\.lib|vcs-svn\/lib\.lib)/, @{$$build_structure{"$prefix${name}_LIBS"}})); + $libs_release = join(";", sort(grep /^(?!libgit\.lib|xdiff\/lib\.lib|vcs-svn\/lib\.lib|reftable\/libreftable\.lib)/, @{$$build_structure{"$prefix${name}_LIBS"}})); $libs_debug = $libs_release; $libs_debug =~ s/zlib\.lib/zlibd\.lib/g; $libs_debug =~ s/libcurl\.lib/libcurl-d\.lib/g; @@ -231,6 +231,7 @@ sub createProject { EOM if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') { my $uuid_libgit = $$build_structure{"LIBS_libgit_GUID"}; + my $uuid_libreftable = $$build_structure{"LIBS_reftable/libreftable_GUID"}; my $uuid_xdiff_lib = $$build_structure{"LIBS_xdiff/lib_GUID"}; print F << "EOM"; @@ -240,6 +241,14 @@ sub createProject { false EOM + if (!($name =~ /xdiff|libreftable/)) { + print F << "EOM"; + + $uuid_libreftable + false + +EOM + } if (!($name =~ 'xdiff')) { print F << "EOM"; From patchwork Mon May 4 19:03:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527447 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB1A781 for ; Mon, 4 May 2020 19:04:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A365B2073E for ; Mon, 4 May 2020 19:04:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ml5kD+KU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726900AbgEDTEF (ORCPT ); Mon, 4 May 2020 15:04:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727835AbgEDTEB (ORCPT ); Mon, 4 May 2020 15:04:01 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83932C061A41 for ; Mon, 4 May 2020 12:04:01 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id y24so709613wma.4 for ; Mon, 04 May 2020 12:04:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EIkPMG4kRISjawN0lo4ItJKCbCJkBbHZ40TpscAnGuQ=; b=ml5kD+KUNIbQuZnB9Lk3riYE0NZRR7P2OtJuGAmY0GuThAPBnLcswqCKkkwzCxmmZ5 pI6cppJWlMVbohZz5RLDUWpU3pK87kGqn91nkczfV/lE/SjDnkXZEypDFnjVZBTLHd7Z TMgj2ZiOks6PWAnF6okIyg7uuYjCJIknsDnMH+1V4SnM8gBGumLn7VzbcU3D+TfaG5dO t/lkaP5M847v5yX/yc5T6TMwNlTaf/VR/rMhP6c+NzH2762Ywx/mDq8DGTJkJ5GTUvmP pzzzuJUGIFerOe26pJf8vgwTRYqGhKNJMN5lZ85IaHpiZTdkb5PKOzylYm2Y400QKcZ6 oHGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EIkPMG4kRISjawN0lo4ItJKCbCJkBbHZ40TpscAnGuQ=; b=no45SdmV+XUYLzeMkcbLb341VCBqLzHb+EU00t/XfUydhYO4RYPLOCc4R73gbydUw4 qCWLNTv6ELPt+A+JH+H3hl0O8Bc3sjwmenB6XjwD/Mu8QWBVfi5ppCSUGA+WJrayNnzN r4Ch4lso/CvQ5hC2fLcZBIL4GtC5D/KwPWrR9o20cxrl8Bemv5L20jCWZsNFFH8ckhcX 5tM0xi6fxCNOMCTTPWG8TTdn1RD9O6zFAsQSs9wyEkMgZmeqZPumElVblny/AsKxYYrf 6YhTaXHp/D+oiHigxlcefB/RZNy2yDpTSea/1nZHyX+9Eqzdm63v0uRj5HOmpn1msZRU SZqA== X-Gm-Message-State: AGi0PuZZY/447r+csrUOLwwkExnyigRyu//yE0pUl9GhquA+95Rr9LKO oFazFIX47zkqaDpJtH10OFXeJ+BO X-Google-Smtp-Source: APiQypJIr6gzvC0qBAYuRVv8RMDJoD4ZSBKcOIykrpa4DBRw9lQuKaIx+maAKG7rdRrFmapIRBweSQ== X-Received: by 2002:a1c:dc8b:: with SMTP id t133mr16587657wmg.117.1588619040060; Mon, 04 May 2020 12:04:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o6sm21791767wrw.63.2020.05.04.12.03.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:03:59 -0700 (PDT) Message-Id: <846fe29fa4be9b1195e68d30bafba19c963a01f2.1588619029.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:47 +0000 Subject: [PATCH v11 11/12] Add some reftable testing infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys * Add GIT_TEST_REFTABLE environment var to control default ref storage * Add test_prerequisite REFTABLE. Skip t/t3210-pack-refs.sh for REFTABLE. Signed-off-by: Han-Wen Nienhuys --- builtin/clone.c | 2 +- builtin/init-db.c | 2 +- refs.c | 6 +++--- t/t3210-pack-refs.sh | 6 ++++++ t/test-lib.sh | 5 +++++ 5 files changed, 16 insertions(+), 5 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 4d0cf065e4a..780c5807415 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1109,7 +1109,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) } init_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, - DEFAULT_REF_STORAGE, INIT_DB_QUIET); + default_ref_storage(), INIT_DB_QUIET); if (real_git_dir) git_dir = real_git_dir; diff --git a/builtin/init-db.c b/builtin/init-db.c index b7053b9e370..da5b4670c84 100644 --- a/builtin/init-db.c +++ b/builtin/init-db.c @@ -545,7 +545,7 @@ static const char *const init_db_usage[] = { int cmd_init_db(int argc, const char **argv, const char *prefix) { const char *git_dir; - const char *ref_storage_format = DEFAULT_REF_STORAGE; + const char *ref_storage_format = default_ref_storage(); const char *real_git_dir = NULL; const char *work_tree; const char *template_dir = NULL; diff --git a/refs.c b/refs.c index 299a5db8bf1..b9b3e7e7070 100644 --- a/refs.c +++ b/refs.c @@ -1823,7 +1823,7 @@ struct ref_store *get_main_ref_store(struct repository *r) r->refs_private = ref_store_init(r->gitdir, r->ref_storage_format ? r->ref_storage_format : - DEFAULT_REF_STORAGE, + default_ref_storage(), REF_STORE_ALL_CAPS); return r->refs_private; } @@ -1879,7 +1879,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done; /* assume that add_submodule_odb() has been called */ - refs = ref_store_init(submodule_sb.buf, DEFAULT_REF_STORAGE, /* XXX */ + refs = ref_store_init(submodule_sb.buf, default_ref_storage(), REF_STORE_READ | REF_STORE_ODB); register_ref_store_map(&submodule_ref_stores, "submodule", refs, submodule); @@ -1893,7 +1893,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) struct ref_store *get_worktree_ref_store(const struct worktree *wt) { - const char *format = DEFAULT_REF_STORAGE; /* XXX */ + const char *format = default_ref_storage(); struct ref_store *refs; const char *id; diff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh index f41b2afb996..edaef2c175a 100755 --- a/t/t3210-pack-refs.sh +++ b/t/t3210-pack-refs.sh @@ -11,6 +11,12 @@ semantic is still the same. ' . ./test-lib.sh +if test_have_prereq REFTABLE +then + skip_all='skipping pack-refs tests; incompatible with reftable' + test_done +fi + test_expect_success 'enable reflogs' ' git config core.logallrefupdates true ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 1b221951a8e..b2b16979407 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1514,6 +1514,11 @@ FreeBSD) ;; esac +if test -n "$GIT_TEST_REFTABLE" +then + test_set_prereq REFTABLE +fi + ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1 test -z "$NO_PERL" && test_set_prereq PERL test -z "$NO_PTHREADS" && test_set_prereq PTHREADS From patchwork Mon May 4 19:03:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11527453 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A0C6B81 for ; Mon, 4 May 2020 19:04:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7FCEB2073B for ; Mon, 4 May 2020 19:04:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UDn4vHM4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727890AbgEDTEI (ORCPT ); Mon, 4 May 2020 15:04:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1727845AbgEDTED (ORCPT ); Mon, 4 May 2020 15:04:03 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8894CC0610D6 for ; Mon, 4 May 2020 12:04:02 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id v4so705148wme.1 for ; Mon, 04 May 2020 12:04:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WAvJwU/XeJRtWLWA3rNMorqhg/uR9TciG0No5VqYjE0=; b=UDn4vHM41KH/DJvGPyGJOWC4McOay1e9uerGABD8xi2e0+9uVt0frfgxzLtduLO+pi P6MDLES8pRtKKpZ8p90pIei6F78nVCO1ztsbpFSeL7F92FA/Qa6Y9DxR5A7qEZjD9wiT 0Ozeu5iUw8GMfYBrvj6LA4qn+w+NH2bX12xVjesIk3AUvOcFyr+QlgKtuq8wOqhWl3Md lpyMpVN0yNGE9ng4M17zLAuGhO2ck1Uq4G3iZj8AApJSc25N/MEeKMS7m86JibeLLEUB m1gz1ZVIWO51irrnWrg8Ldf+MenHw3uEvQpB3SFn3BeGcZ5w0vrzmHSuh5KnbevMJbMh DjKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WAvJwU/XeJRtWLWA3rNMorqhg/uR9TciG0No5VqYjE0=; b=rm+OeY+yqWlpY0eJauF0cGedUI9zLfMfZCzhEonVbBhVqPDVMwscZb4qp/t4jwD0P8 HkZWElznl9ROpSAWrZo+i0BEKr1tuTlKsFGcQYoS3RcCZ+8q3w+2lCbrVG+0QcSlbNR6 sNy7tnLGsW3mWmcVQXIxeEtM9+SqNS135malyj+0b1Dx6AEdiCQgSeiaZs/seZ9RcrQv rOoVe3/NBCwPOA3XQl4TNDOIYsCrbTriG7Iy7QW9LWy5AlNG8pkqwF/ou6fNawSBDwHd IFOB5rd9eBUcoacLyCIh8ZT0l1Iget1NuxO+uBVwh8ws6Tbg8l/ooI+VI9Euy9YHqxZ4 RCcw== X-Gm-Message-State: AGi0PuaE1m8+S1dhdLOQABa2ukyMgKbDrQ+AIU0lkosMy0CfhSMjAYzu qyRCHaXGdTsh+YELDf+1xUdcVspl X-Google-Smtp-Source: APiQypIgO6MM6rwtZykAH5foqARYy0GEfrBuL34jLwYPLVOP/nRd5rFmhRQVdQzHWnw7JIpGz2bnQQ== X-Received: by 2002:a1c:2e91:: with SMTP id u139mr15729868wmu.18.1588619040758; Mon, 04 May 2020 12:04:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e5sm19798797wru.92.2020.05.04.12.04.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 May 2020 12:04:00 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Han-Wen Nienhuys via GitGitGadget" Date: Mon, 04 May 2020 19:03:48 +0000 Subject: [PATCH v11 12/12] t: use update-ref and show-ref to reading/writing refs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Han-Wen Nienhuys , Han-Wen Nienhuys Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Han-Wen Nienhuys Reading and writing .git/refs/* assumes that refs are stored in the 'files' ref backend. Signed-off-by: Han-Wen Nienhuys --- t/t0002-gitfile.sh | 2 +- t/t1400-update-ref.sh | 32 ++++++++++++++++---------------- t/t1506-rev-parse-diagnosis.sh | 2 +- t/t6050-replace.sh | 2 +- t/t9020-remote-svn.sh | 4 ++-- 5 files changed, 21 insertions(+), 21 deletions(-) diff --git a/t/t0002-gitfile.sh b/t/t0002-gitfile.sh index 0aa9908ea12..960ed150cb5 100755 --- a/t/t0002-gitfile.sh +++ b/t/t0002-gitfile.sh @@ -62,7 +62,7 @@ test_expect_success 'check commit-tree' ' ' test_expect_success 'check rev-list' ' - echo $SHA >"$REAL/HEAD" && + git update-ref "HEAD" "$SHA" && test "$SHA" = "$(git rev-list HEAD)" ' diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index e1197ac8189..27171f82612 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -37,15 +37,15 @@ test_expect_success setup ' test_expect_success "create $m" ' git update-ref $m $A && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' test_expect_success "create $m with oldvalue verification" ' git update-ref $m $B $A && - test $B = $(cat .git/$m) + test $B = $(git show-ref -s --verify $m) ' test_expect_success "fail to delete $m with stale ref" ' test_must_fail git update-ref -d $m $A && - test $B = "$(cat .git/$m)" + test $B = "$(git show-ref -s --verify $m)" ' test_expect_success "delete $m" ' test_when_finished "rm -f .git/$m" && @@ -56,7 +56,7 @@ test_expect_success "delete $m" ' test_expect_success "delete $m without oldvalue verification" ' test_when_finished "rm -f .git/$m" && git update-ref $m $A && - test $A = $(cat .git/$m) && + test $A = $(git show-ref -s --verify $m) && git update-ref -d $m && test_path_is_missing .git/$m ' @@ -69,15 +69,15 @@ test_expect_success "fail to create $n" ' test_expect_success "create $m (by HEAD)" ' git update-ref HEAD $A && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' test_expect_success "create $m (by HEAD) with oldvalue verification" ' git update-ref HEAD $B $A && - test $B = $(cat .git/$m) + test $B = $(git show-ref -s --verify $m) ' test_expect_success "fail to delete $m (by HEAD) with stale ref" ' test_must_fail git update-ref -d HEAD $A && - test $B = $(cat .git/$m) + test $B = $(git show-ref -s --verify $m) ' test_expect_success "delete $m (by HEAD)" ' test_when_finished "rm -f .git/$m" && @@ -178,14 +178,14 @@ test_expect_success '--no-create-reflog overrides core.logAllRefUpdates=always' test_expect_success "create $m (by HEAD)" ' git update-ref HEAD $A && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' test_expect_success 'pack refs' ' git pack-refs --all ' test_expect_success "move $m (by HEAD)" ' git update-ref HEAD $B $A && - test $B = $(cat .git/$m) + test $B = $(git show-ref -s --verify $m) ' test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" ' test_when_finished "rm -f .git/$m" && @@ -255,7 +255,7 @@ test_expect_success '(not) change HEAD with wrong SHA1' ' ' test_expect_success "(not) changed .git/$m" ' test_when_finished "rm -f .git/$m" && - ! test $B = $(cat .git/$m) + ! test $B = $(git show-ref -s --verify $m) ' rm -f .git/logs/refs/heads/master @@ -263,19 +263,19 @@ test_expect_success "create $m (logged by touch)" ' test_config core.logAllRefUpdates false && GIT_COMMITTER_DATE="2005-05-26 23:30" \ git update-ref --create-reflog HEAD $A -m "Initial Creation" && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' test_expect_success "update $m (logged by touch)" ' test_config core.logAllRefUpdates false && GIT_COMMITTER_DATE="2005-05-26 23:31" \ git update-ref HEAD $B $A -m "Switch" && - test $B = $(cat .git/$m) + test $B = $(git show-ref -s --verify $m) ' test_expect_success "set $m (logged by touch)" ' test_config core.logAllRefUpdates false && GIT_COMMITTER_DATE="2005-05-26 23:41" \ git update-ref HEAD $A && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' test_expect_success 'empty directory removal' ' @@ -319,19 +319,19 @@ test_expect_success "create $m (logged by config)" ' test_config core.logAllRefUpdates true && GIT_COMMITTER_DATE="2005-05-26 23:32" \ git update-ref HEAD $A -m "Initial Creation" && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' test_expect_success "update $m (logged by config)" ' test_config core.logAllRefUpdates true && GIT_COMMITTER_DATE="2005-05-26 23:33" \ git update-ref HEAD'" $B $A "'-m "Switch" && - test $B = $(cat .git/$m) + test $B = $(git show-ref -s --verify $m) ' test_expect_success "set $m (logged by config)" ' test_config core.logAllRefUpdates true && GIT_COMMITTER_DATE="2005-05-26 23:43" \ git update-ref HEAD $A && - test $A = $(cat .git/$m) + test $A = $(git show-ref -s --verify $m) ' cat >expect <expect && git rev-parse foobar -- >actual && diff --git a/t/t6050-replace.sh b/t/t6050-replace.sh index e7e64e085dd..c80dc10b8f1 100755 --- a/t/t6050-replace.sh +++ b/t/t6050-replace.sh @@ -135,7 +135,7 @@ test_expect_success 'tag replaced commit' ' test_expect_success '"git fsck" works' ' git fsck master >fsck_master.out && test_i18ngrep "dangling commit $R" fsck_master.out && - test_i18ngrep "dangling tag $(cat .git/refs/tags/mytag)" fsck_master.out && + test_i18ngrep "dangling tag $(git show-ref -s refs/tags/mytag)" fsck_master.out && test -z "$(git fsck)" ' diff --git a/t/t9020-remote-svn.sh b/t/t9020-remote-svn.sh index 6fca08e5e35..9fcfa969a9b 100755 --- a/t/t9020-remote-svn.sh +++ b/t/t9020-remote-svn.sh @@ -48,8 +48,8 @@ test_expect_success REMOTE_SVN 'simple fetch' ' ' test_debug ' - cat .git/refs/svn/svnsim/master - cat .git/refs/remotes/svnsim/master + git show-ref -s refs/svn/svnsim/master + git show-ref -s refs/remotes/svnsim/master ' test_expect_success REMOTE_SVN 'repeated fetch, nothing shall change' '