From patchwork Sun Sep 5 23:51:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Olsen X-Patchwork-Id: 12476171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE137C433FE for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 76EC160F8F for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234072AbhIEXwr (ORCPT ); Sun, 5 Sep 2021 19:52:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232565AbhIEXwq (ORCPT ); Sun, 5 Sep 2021 19:52:46 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB517C061757 for ; Sun, 5 Sep 2021 16:51:42 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id z4so7118062wrr.6 for ; Sun, 05 Sep 2021 16:51:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yOrIfMGP2yLVzDzxj+jcvCmTtxUMKfqzq2ZBB86tTBY=; b=d4JUNz7yQqsw5M5fBBmc8bqIEKVy2VIrGiIDAyd6/5lwW30rmGCQi0md6bKseaqZFz UytzNsLqwLm4uL1VvS7QbQWCVt28AgKY0cfx9qMoBAVBfgeRX2mDFF2E/F7246nmONHr J67pTWTDIUdHJ6V+47izLgbX9jOLHG1xthtA+znqTB3m1vWUeieYifQCgch6Vs2WFlIa DZSiJKjdi60c1GaApGDzkNfkRYyF54jDIk+Ecz65QpPN8ZHl5glhDcdz0r2mPZKOXzrT 9Qh/EchV5DejDROEaciX9szjOejMbS/WiTG43LvB3GhWfDx7q4YUTKwGDO7qpEZDywr0 ZZPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yOrIfMGP2yLVzDzxj+jcvCmTtxUMKfqzq2ZBB86tTBY=; b=rR2FuahWCDfGbbRyj6KiBC3myqh/2Uxgqhi8LcJ4vU6UAXRWFRleihQM1w9gTBEO3w 5EPIXPM9SPmkvDcvP1c8eB7xXc6QjxLjHmx60wYXc7bbJjg+RdWfa5TkmoRNokZDHzhm KpVp3H2BYQmoT5A9Z0SAwe5ITx3dLYsGPekxDkj3j3YPlc+QZIk4SLwbg3SDVLbkrghS ybyCvKt0Pkxtwws0wxhBtUMobida+skMNEQTnonOZvzTaieakhXFniSTK728ixjHq+7o SPt257Nt0nSCmWZCJ4eX1W+cieSXDlPTlJJ//dr/TfGW3y7VRLyJ5GXOO6gurI0eVovt HeAQ== X-Gm-Message-State: AOAM532nU4dA1CivcixLmOATbtXlo1k86Fud9VrD9aYbKyYua5oJ45yK tN4uq6QvWaPlI9hRR4sTzOgufVfLVXs= X-Google-Smtp-Source: ABdhPJzJyyJ82orwuDu/7fG7KR06ddGIRtGqSKi8zWxii4ph62Z6OPJU0CP7oYIzjcraDBVB7Mfjjw== X-Received: by 2002:adf:9e01:: with SMTP id u1mr10460318wre.420.1630885901293; Sun, 05 Sep 2021 16:51:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w29sm6168136wra.88.2021.09.05.16.51.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Sep 2021 16:51:40 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sun, 05 Sep 2021 23:51:36 +0000 Subject: [PATCH 1/4] Compile-time extensions for list-object-filter Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Andrew Olsen , Andrew Olsen Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Andrew Olsen From: Andrew Olsen Adds an extension: option to list-object-filters, these are implemented by static libraries that must be compiled into Git. C code changes only - Makefile changes follow. Signed-off-by: Andrew Olsen --- .gitignore | 1 + generate-list-objects-filter-extensions.sh | 53 ++++++++++ list-objects-filter-extensions.h | 107 +++++++++++++++++++++ list-objects-filter-options.c | 47 +++++++++ list-objects-filter-options.h | 6 ++ list-objects-filter.c | 84 ++++++++++++++++ 6 files changed, 298 insertions(+) create mode 100755 generate-list-objects-filter-extensions.sh create mode 100644 list-objects-filter-extensions.h diff --git a/.gitignore b/.gitignore index 311841f9bed..3564cb01ad7 100644 --- a/.gitignore +++ b/.gitignore @@ -190,6 +190,7 @@ /gitweb/static/gitweb.min.* /config-list.h /command-list.h +/list-objects-filter-extensions.c *.tar.gz *.dsc *.deb diff --git a/generate-list-objects-filter-extensions.sh b/generate-list-objects-filter-extensions.sh new file mode 100755 index 00000000000..422b1ce837f --- /dev/null +++ b/generate-list-objects-filter-extensions.sh @@ -0,0 +1,53 @@ +#!/bin/sh + +if [ $# -gt 0 ]; then + + # ARGS has one argument per line + ARGS=$(echo "$@" | xargs printf '%s\n') + + # Every argument should be path to a filter extension library. + INVALID_ARGS=$(echo "$ARGS" | grep -v '\.a$') + if [ -n "$INVALID_ARGS" ] ; then + printf "Error: all arguments must be paths to .a files: \n%s\n" \ + "${INVALID_ARGS}" >&2 + exit 1 + fi + + # qux/foo.a -> foo + NAMES=$(echo "$ARGS" | sed -e 's!.*/!!' -e 's!.a$!!') + + # Filter extension names must be valid C symbols so they can be linked by name. + INVALID_NAMES=$(echo "$NAMES" | grep -v '^[A-Za-z0-9_]\+$') + if [ -n "$INVALID_NAMES" ] ; then + printf "Error: all library names must also be valid C symbols: \n%s\n" \ + "${INVALID_NAMES}" >&2 + exit 1 + fi + + # foo -> filter_extension_foo + EXTS=$(echo "$NAMES" | sed -e 's!^!filter_extension_!') + + # filter_extension_foo -> [\t]filter_extension_foo, + DECLARATIONS=$(echo "$EXTS" | sed -e 's!^!\t!' -e 's!$!,!') + + # filter_extension_foo -> [\t]&filter_extension_foo, + ARRAY=$(echo "$EXTS" | sed -e 's!^!\t\&!' -e 's!$!,!') +fi + +echo '/* Automatically generated by generate-list-objects-filter-extensions.sh */' +echo +echo '#include "git-compat-util.h"' +echo '#include "list-objects-filter-extensions.h"' +echo + +if [ $# -gt 0 ]; then + echo 'extern const struct filter_extension' + echo "${DECLARATIONS%?}" + echo ';' + echo +fi + +echo 'const struct filter_extension *filter_extensions[] = {' +echo "${ARRAY}" +echo ' NULL,' +echo '};' \ No newline at end of file diff --git a/list-objects-filter-extensions.h b/list-objects-filter-extensions.h new file mode 100644 index 00000000000..35ebe1ead31 --- /dev/null +++ b/list-objects-filter-extensions.h @@ -0,0 +1,107 @@ +#ifndef GIT_LIST_OBJECTS_FILTER_EXTENSIONS_H +#define GIT_LIST_OBJECTS_FILTER_EXTENSIONS_H + +/** + * The List-Objects-Filter Extensions API can be used to develop filter + * extensions for git-upload-pack/git-rev-list/etc. + * + * See contrib/filter-extensions/README.md for more details and examples. + * + * The API defines three functions to implement a filter operation. Note that + * each filter implementing this API must compiled into Git as a static library. + * There is some plumbing in the Makefile to help with this via + * FILTER_EXTENSIONS. + * + * 1. You write a filter and compile it into your custom build of git. + * See list_objects_filter_ext_filter_fn. + * 2. A filter request is received that specifically names the filter extension + * that you have written, ie: "--filter=extension:[=]" + * 3. Your list_objects_filter_ext_init_fn() is called. + * 4. Your list_objects_filter_ext_filter_fn() is called for each object + * at least once. + * 5. Your list_objects_filter_ext_free_fn() is called. + */ + +#include "list-objects-filter.h" + + +/* Whether to add or remove a specific object from any current omitset. */ +enum list_objects_filter_omit { + LOFO_KEEP = -1, + LOFO_IGNORE = 0, + LOFO_OMIT = 1, +}; + +/* + * This is a corollary to `list_objects_filter__init()` and constructs the + * filter, parsing and validating any user-provided `filter_arg` (via + * `--filter=extension:=`). Use `context` for any filter-allocated + * context data. + * + * Return 0 on success and non-zero on error. + */ +typedef +int list_objects_filter_ext_init_fn( + const struct repository *r, + const char* filter_arg, + void **context +); + +/* + * This is a corollary to `list_objects_filter__free()`, destroying the filter + * and any filter-allocated context data. + */ +typedef +void list_objects_filter_ext_free_fn( + const struct repository *r, + void *context +); + +/* + * This is a corollary to `list_objects_filter__filter_object()`, and + * decides how to handle the object `obj`. + * + * omit provides a flag determining whether to explicitly add or remove + * the object from any current omitset. + */ +typedef +enum list_objects_filter_result list_objects_filter_ext_filter_fn( + const struct repository *r, + const enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + enum list_objects_filter_omit *omit, + void *context +); + +/* + * To implement a filter extension called "mine", you should define + * a const struct filter_extension called filter_extension_mine, + * in the following manner: + * + * const struct filter_extension filter_extension_mine = { + * "mine", + * &my_init_fn, + * &my_filter_object_fn, + * &my_free_fn + * }; + * + * See contrib/filter-extensions/README.md for more details and examples. + */ + +struct filter_extension { + const char *name; + list_objects_filter_ext_init_fn* init_fn; + list_objects_filter_ext_filter_fn* filter_object_fn; + list_objects_filter_ext_free_fn* free_fn; +}; + +/* + * The filter_extensions array is defined in list_objects_filter_extensions.c + * which is generated at compile time from the FILTER_EXTENSIONS variable. + */ +extern const struct filter_extension *filter_extensions[]; + + +#endif /* GIT_LIST_OBJECTS_FILTER_EXTENSIONS_H */ diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index fd8d59f653a..e92499f29c2 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -15,6 +15,11 @@ static int parse_combine_filter( const char *arg, struct strbuf *errbuf); +static int parse_extension_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf); + const char *list_object_filter_config_name(enum list_objects_filter_choice c) { switch (c) { @@ -31,6 +36,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c) return "sparse:oid"; case LOFC_OBJECT_TYPE: return "object:type"; + case LOFC_EXTENSION: + return "extension"; case LOFC_COMBINE: return "combine"; case LOFC__COUNT: @@ -91,6 +98,9 @@ static int gently_parse_list_objects_filter( filter_options->choice = LOFC_SPARSE_OID; return 0; + } else if (skip_prefix(arg, "extension:", &v0)) { + return parse_extension_filter(filter_options, v0, errbuf); + } else if (skip_prefix(arg, "sparse:path=", &v0)) { if (errbuf) { strbuf_addstr( @@ -209,6 +219,41 @@ cleanup: return result; } +static int parse_extension_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf) +{ + int result = 0; + struct strbuf **params = strbuf_split_str(arg, '=', 2); + + if (!params[0]) { + strbuf_addstr(errbuf, _("expected 'extension:[=]'")); + result = 1; + goto cleanup; + } + + if (params[1]) { + // This extension has a parameter. Remove trailing "=" from the name. + size_t last = params[0]->len - 1; + assert(params[0]->buf[last] == '='); + strbuf_remove(params[0], last, 1); + + filter_options->extension_value = xstrdup(params[1]->buf); + } + + filter_options->extension_name = xstrdup(params[0]->buf); + filter_options->choice = LOFC_EXTENSION; + +cleanup: + strbuf_list_free(params); + if (result) { + list_objects_filter_release(filter_options); + memset(filter_options, 0, sizeof(*filter_options)); + } + return result; +} + static int allow_unencoded(char ch) { if (ch <= ' ' || ch == '%' || ch == '+') @@ -349,6 +394,8 @@ void list_objects_filter_release( return; string_list_clear(&filter_options->filter_spec, /*free_util=*/0); free(filter_options->sparse_oid_name); + free(filter_options->extension_name); + free(filter_options->extension_value); for (sub = 0; sub < filter_options->sub_nr; sub++) list_objects_filter_release(&filter_options->sub[sub]); free(filter_options->sub); diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index da5b6737e27..df3e360324e 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -15,6 +15,7 @@ enum list_objects_filter_choice { LOFC_TREE_DEPTH, LOFC_SPARSE_OID, LOFC_OBJECT_TYPE, + LOFC_EXTENSION, LOFC_COMBINE, LOFC__COUNT /* must be last */ }; @@ -58,6 +59,11 @@ struct list_objects_filter_options { unsigned long tree_exclude_depth; enum object_type object_type; + /* LOFC_EXTENSION values */ + + char *extension_name; + char *extension_value; + /* LOFC_COMBINE values */ /* This array contains all the subfilters which this filter combines. */ diff --git a/list-objects-filter.c b/list-objects-filter.c index 1c1ee3d1bb1..037c674b1c3 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -10,6 +10,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "list-objects-filter-extensions.h" #include "oidmap.h" #include "oidset.h" #include "object-store.h" @@ -620,6 +621,88 @@ static void filter_object_type__init( filter->free_fn = free; } +/* + * A filter which passes the objects to a compile-time extension. + * The extension needs to implement the filter_extension interface + * defined in list-objects-filter-extension.h. + * See contrib/filter-extensions/README.md + */ + +struct filter_extension_data { + const struct filter_extension *extension; + void *context; +}; + +static enum list_objects_filter_result filter_extension_filter_object( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct oidset *omits, + void *filter_data) +{ + struct filter_extension_data *d = filter_data; + + enum list_objects_filter_omit omit_it = LOFO_IGNORE; + + enum list_objects_filter_result ret = + d->extension->filter_object_fn( + r, + filter_situation, + obj, + pathname, + filename, + &omit_it, + d->context); + + if (omits) { + if (omit_it == LOFO_KEEP) + oidset_remove(omits, &obj->oid); + else if (omit_it == LOFO_OMIT) + oidset_insert(omits, &obj->oid); + } + return ret; +} + +static void filter_extension_free(void *filter_data) +{ + struct filter_extension_data *d = filter_data; + d->extension->free_fn(the_repository, d->context); + free(d); +} + +static void filter_extension__init( + struct list_objects_filter_options *filter_options, + struct filter *filter) +{ + struct filter_extension_data *d = xcalloc(1, sizeof(*d)); + int i, r; + + for (i = 0; filter_extensions[i] != NULL; i++) { + if (!strcmp( + filter_options->extension_name, + filter_extensions[i]->name)) + break; + } + if (filter_extensions[i] == NULL) { + die(_("No filter extension found with name %s"), + filter_options->extension_name); + } + d->extension = filter_extensions[i]; + + r = d->extension->init_fn( + the_repository, filter_options->extension_value, &d->context); + if (r) { + die(_("Error initialising filter extension %s: %d"), + filter_options->extension_name, r); + } + + filter->filter_data = d; + filter->filter_object_fn = &filter_extension_filter_object; + filter->free_fn = &filter_extension_free; +} + /* A filter which only shows objects shown by all sub-filters. */ struct combine_filter_data { struct subfilter *sub; @@ -767,6 +850,7 @@ static filter_init_fn s_filters[] = { filter_trees_depth__init, filter_sparse_oid__init, filter_object_type__init, + filter_extension__init, filter_combine__init, }; From patchwork Sun Sep 5 23:51:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Olsen X-Patchwork-Id: 12476177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3DF2C433F5 for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 919AF60E93 for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235194AbhIEXws (ORCPT ); Sun, 5 Sep 2021 19:52:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234077AbhIEXwr (ORCPT ); Sun, 5 Sep 2021 19:52:47 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B939C061575 for ; Sun, 5 Sep 2021 16:51:43 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id o39-20020a05600c512700b002e74638b567so3570107wms.2 for ; Sun, 05 Sep 2021 16:51:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yFSdDYd/lw6tWAOJj9Ojhzsn8yDq+UXCk8FF68k8NjY=; b=UMfO0RJ8NKn/Y2fJmgao6Tr0MR9cv5oEcn7G3lCs1qh7fBGHoGzYjjGgSn7obdQxyc ieQ/CcPZF9pWrEyInB7YUpgdl8ulPvhP92RX/A2TfVTRxMjw7zGZcFla7JDKljxc8nYk suTDPjO8LO/dAlKhJyz2ShwBuNPCBcS86L+heWxCI7ifYPXkIgEWvFvEnxoO7dhNaLnZ wtS0VtNOAxoZNp1r/IaNPmuM16QjW0PK6GurPfl5sGGFUK1oSdkOrjibojr71W14vkea d41dbgOHfX8kbTgvuvOk90ohRJa4bLpHFXScCYDkRFTTfgkowNs4pmtZbfNiACdV8lT1 OLbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yFSdDYd/lw6tWAOJj9Ojhzsn8yDq+UXCk8FF68k8NjY=; b=Nrc7eq9hm31lqaQG9bV5pndUuNQ7UHCbFB9hCj0zpuo6THne0Qzcc8I0kU1Wfq0abg hKrQY+DYWr5FlrsYErZkiOQYeFLTIQcXcWMxyk5uh9IrYAuPnZUNXNC73pSSYB2z3Bel eGS/u6RDN/TzKfmFXpYrb/Qse+cmNEEumJq4BffZ/JWfyOaajZ14bSsoMK8soOoQRL3I gAzq31JjZ5pKCDYjIKJCU81dywliZzSrQ1fxbZFBrUGLdgoX1wCiALga+qkP7GkOiOpr iotwV8QuhJmAYYN/Y8pc02mvJOJPilfJeHpds4+bewcR9wbgF4X3/60pFQuX8ZQ9tNj0 WzBA== X-Gm-Message-State: AOAM530+Fsw+ZX3APbo6Oq7HJO3DL6bKGjz5SJQqJtvT61FNtvZx8e71 wJPnur2IRdqyUCEQTqaNwnBks/YQ224= X-Google-Smtp-Source: ABdhPJwxAEe8aYtv+VN86vp6TcbkyMzUJOCcVgCeSIx5gXqmWWwNt/Cg2O4p7lUlHb72LXpLWhRhQQ== X-Received: by 2002:a7b:c18c:: with SMTP id y12mr8576624wmi.3.1630885902056; Sun, 05 Sep 2021 16:51:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q85sm5683300wme.23.2021.09.05.16.51.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Sep 2021 16:51:41 -0700 (PDT) Message-Id: <43415de761ac029b67645d9131b38e761ef6ae21.1630885899.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 05 Sep 2021 23:51:37 +0000 Subject: [PATCH 2/4] Makefile for list-object-filter extensions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Andrew Olsen , Andrew Olsen Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Andrew Olsen From: Andrew Olsen Custom list-object-filter extensions can be compiled into Git using the FILTER_EXTENSIONS Makefile argument. Signed-off-by: Andrew Olsen --- Makefile | 35 +++++++++++++++++++++++++++-- compat/vcbuild/README | 5 +++-- config.mak.uname | 6 ++--- contrib/buildsystems/CMakeLists.txt | 7 ++++++ 4 files changed, 46 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index 429c276058d..0b1d0be81a5 100644 --- a/Makefile +++ b/Makefile @@ -471,6 +471,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# Define FILTER_EXTENSIONS to a space-separated list of static library plugins +# that implement the filter-object-list extension API. Each of this filter +# extensions will then be available in addition to the builtin ones such as +# "blob:limit" and "object:type". See contrib/filter-extensions/README.txt +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -824,6 +829,7 @@ XDIFF_LIB = xdiff/lib.a GENERATED_H += command-list.h GENERATED_H += config-list.h +GENERATED_C += list-objects-filter-extensions.c LIB_H := $(sort $(patsubst ./%,%,$(shell git ls-files '*.h' ':!t/' ':!Documentation/' 2>/dev/null || \ $(FIND) . \ @@ -916,6 +922,7 @@ LIB_OBJS += levenshtein.o LIB_OBJS += line-log.o LIB_OBJS += line-range.o LIB_OBJS += linear-assignment.o +LIB_OBJS += list-objects-filter-extensions.o LIB_OBJS += list-objects-filter-options.o LIB_OBJS += list-objects-filter.o LIB_OBJS += list-objects.o @@ -2116,6 +2123,19 @@ ifdef DEFAULT_HELP_FORMAT BASIC_CFLAGS += -DDEFAULT_HELP_FORMAT='"$(DEFAULT_HELP_FORMAT)"' endif +ifneq ($(FILTER_EXTENSIONS),) +FILTER_EXT_PATHS = $(dir $(FILTER_EXTENSIONS)) + +$(FILTER_EXTENSIONS): $(FILTER_EXT_PATHS) + $(QUIET_SUBDIR0)$(@D) $(QUIET_SUBDIR1) \ + ALL_CFLAGS='$(subst ','\'',$(ALL_CFLAGS))' \ + ALL_LDFLAGS='$(subst ','\'',$(ALL_LDFLAGS))' \ + PROFILE_DIR='$(subst ','\'',$(PROFILE_DIR))' \ + $(@F) + +GITLIBS += $(FILTER_EXTENSIONS) +endif + PAGER_ENV_SQ = $(subst ','\'',$(PAGER_ENV)) PAGER_ENV_CQ = "$(subst ",\",$(subst \,\\,$(PAGER_ENV)))" PAGER_ENV_CQ_SQ = $(subst ','\'',$(PAGER_ENV_CQ)) @@ -2222,7 +2242,7 @@ git.sp git.s git.o: EXTRA_CPPFLAGS = \ '-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \ '-DGIT_INFO_PATH="$(infodir_relative_SQ)"' -git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS) +git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS) $(EXTENSION_LIBS) $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ $(filter %.o,$^) $(LIBS) @@ -2261,6 +2281,10 @@ command-list.h: $(wildcard Documentation/git*.txt) $(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \ command-list.txt >$@+ && mv $@+ $@ +list-objects-filter-extensions.c: generate-list-objects-filter-extensions.sh GIT-BUILD-OPTIONS + $(QUIET_GEN)$(SHELL_PATH) ./generate-list-objects-filter-extensions.sh \ + $(FILTER_EXTENSIONS) > $@+ && mv $@+ $@ + SCRIPT_DEFINES = $(SHELL_PATH_SQ):$(DIFF_SQ):$(GIT_VERSION):\ $(localedir_SQ):$(NO_CURL):$(USE_GETTEXT_SCHEME):$(SANE_TOOL_PATH_SQ):\ $(gitwebdir_SQ):$(PERL_PATH_SQ):$(SANE_TEXT_GREP):$(PAGER_ENV):\ @@ -2612,6 +2636,7 @@ $(LIB_FILE): $(LIB_OBJS) $(XDIFF_LIB): $(XDIFF_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE @@ -2857,6 +2882,9 @@ ifdef RUNTIME_PREFIX @echo RUNTIME_PREFIX=\'true\' >>$@+ else @echo RUNTIME_PREFIX=\'false\' >>$@+ +endif +ifdef FILTER_EXTENSIONS + @echo FILTER_EXTENSIONS=\''$(subst ','\'',$(subst ','\'',$(FILTER_EXTENSIONS)))'\' >>$@+ endif @if cmp $@+ $@ >/dev/null 2>&1; then $(RM) $@+; else mv $@+ $@; fi @@ -3241,7 +3269,7 @@ clean: profile-clean coverage-clean cocciclean $(RM) $(HCC) $(RM) -r bin-wrappers $(dep_dirs) $(compdb_dir) compile_commands.json $(RM) -r po/build/ - $(RM) *.pyc *.pyo */*.pyc */*.pyo $(GENERATED_H) $(ETAGS_TARGET) tags cscope* + $(RM) *.pyc *.pyo */*.pyc */*.pyo $(GENERATED_H) $(GENERATED_C) $(ETAGS_TARGET) tags cscope* $(RM) -r .dist-tmp-dir .doc-tmp-dir $(RM) $(GIT_TARNAME).tar.gz $(RM) $(htmldocs).tar.gz $(manpages).tar.gz @@ -3256,6 +3284,9 @@ endif ifndef NO_TCLTK $(MAKE) -C gitk-git clean $(MAKE) -C git-gui clean +endif +ifneq ($(FILTER_EXTENSIONS),) + $(foreach FP,$(FILTER_EXTENSIONS),$(MAKE) -C $(dir $(FP)) clean && ) true endif $(RM) GIT-VERSION-FILE GIT-CFLAGS GIT-LDFLAGS GIT-BUILD-OPTIONS $(RM) GIT-USER-AGENT GIT-PREFIX diff --git a/compat/vcbuild/README b/compat/vcbuild/README index 51fb083dbbe..5e39022eade 100644 --- a/compat/vcbuild/README +++ b/compat/vcbuild/README @@ -92,8 +92,9 @@ The Steps of Build Git with VS2008 the git operations. 3. Inside Git's directory run the command: - make command-list.h config-list.h - to generate the header file needed to compile git. + make command-list.h config-list.h list-objects-filter-extensions.c + to generate those source files that are not included in the repo, but + instead are automatically generated from other files. 4. Then either build Git with the GNU Make Makefile in the Git projects root diff --git a/config.mak.uname b/config.mak.uname index 76516aaa9a5..405e7d91e7a 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -735,9 +735,9 @@ vcxproj: echo '') >git-remote-http/LinkOrCopyRemoteHttp.targets git add -f git/LinkOrCopyBuiltins.targets git-remote-http/LinkOrCopyRemoteHttp.targets - # Add command-list.h and config-list.h - $(MAKE) MSVC=1 SKIP_VCPKG=1 prefix=/mingw64 config-list.h command-list.h - git add -f config-list.h command-list.h + # Add command-list.h, config-list.h list-objects-filter-extensions.c + $(MAKE) MSVC=1 SKIP_VCPKG=1 prefix=/mingw64 config-list.h command-list.h list-objects-filter-extensions.c + git add -f config-list.h command-list.h list-objects-filter-extensions.c # Add scripts rm -f perl/perl.mak diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 171b4124afe..60627a2892f 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -624,6 +624,13 @@ if(NOT EXISTS ${CMAKE_BINARY_DIR}/config-list.h) OUTPUT_FILE ${CMAKE_BINARY_DIR}/config-list.h) endif() +if(NOT EXISTS ${CMAKE_BINARY_DIR}/list-objects-filter-extensions.c) + message("Generating list-objects-filter-extensions.c") + execute_process(COMMAND ${SH_EXE} ${CMAKE_SOURCE_DIR}/generate-list-objects-filter-extensions.sh + WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} + OUTPUT_FILE ${CMAKE_BINARY_DIR}/list-objects-filter-extensions.c) +endif() + include_directories(${CMAKE_BINARY_DIR}) #build From patchwork Sun Sep 5 23:51:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Olsen X-Patchwork-Id: 12476175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC231C4332F for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A51ED60F90 for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236017AbhIEXws (ORCPT ); Sun, 5 Sep 2021 19:52:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232565AbhIEXwr (ORCPT ); Sun, 5 Sep 2021 19:52:47 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F2B5C061575 for ; Sun, 5 Sep 2021 16:51:44 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id n14-20020a05600c3b8e00b002f8bd2f8ab6so406899wms.5 for ; Sun, 05 Sep 2021 16:51:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=u7Sh4RLFgZpb/PEvnkzFvuVfowghuBsscb1kvOHawd8=; b=KZnTiyksqeYBggyy4iweZ+NqaJSIrOR1xjCuDQ4P98zTbY9TvByuapQTd8mDchIlCi S7uNuRIenHtrpF6OvCMDs9Skp53yYcCdMlG6GiOxnyxhFmeqgftBz7HIihCanOPm/lgf o5ppKcEEYYC9724Dh/teg7xSEKcyztOaZDapQA29DyOGukE1kYtQAf4Qi0lbZNPqq4gh Ys/DRKYIh2IKmXLDW1tJzseS2KY2POGgqPUQUVi0j53Y0ROp6x0pGvMyjVDkNgYCLxYz jDQvCfgECSomsxnlg2GVj8h7UAFKJXdgjO2st9t+qOTkHfthDZa/zKGjRkhJTDD8vhl9 c5Pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=u7Sh4RLFgZpb/PEvnkzFvuVfowghuBsscb1kvOHawd8=; b=UST3qaXsPacWbHkAPm5iSraqP5K3yMA8CYAnSWKpwMVWVQaqbgtyG1nE4duSxUpHlS HOqvQbpzYWpJz9GbNZpcUBPsO4L4QL0bqQ1cOX2rl3hVwDyz6B0PapzLCo1Ufzwt3kCm w2cnOJ8z72dEd2B1C1EOEQOebc61gntCWva1l2OyabyKs3UpXjuS/ImQFb5T3jOfHLio d4KKcZMU3kCEm7368L6dorO1fMrLr5mft2ldQzBgIVE14nbKIrB6tnxoGu2m9CMhzaGT GBIEI0vTJpwvsq0OYjQe8Dv0LVMl7jiF8I4/55btlWbsQdRn4K4Grm5uKU2Ll3O5XZF4 j3Ig== X-Gm-Message-State: AOAM530hT//lMHnW/QAgpOG9bxAqAmHhWsMbe32ZJkfHu1mSXpkPNvSX LCb7vtHlMBb4w+pjYr/xsYOBoNMb+c0= X-Google-Smtp-Source: ABdhPJy8PAOazhCmrx6lcNhmptPctLvdQiNv6fHhUpsYqKA/9EEJO+VU/T52Fb3ofzxE1HX051A3dg== X-Received: by 2002:a1c:730a:: with SMTP id d10mr8819642wmb.59.1630885902678; Sun, 05 Sep 2021 16:51:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d9sm7449785wrb.36.2021.09.05.16.51.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Sep 2021 16:51:42 -0700 (PDT) Message-Id: <8e0d11ea53a080e8212768f370fb8f05eaded312.1630885899.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 05 Sep 2021 23:51:38 +0000 Subject: [PATCH 3/4] Sample list-object-filter extensions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Andrew Olsen , Andrew Olsen Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Andrew Olsen From: Andrew Olsen Basic filter extension example which filters to a random subset of blobs, and another example which shows how to do the same in C++ and how to link in another library required by a filter extension. Documentation changes follow. Signed-off-by: Andrew Olsen --- contrib/filter-extensions/rand/.gitignore | 2 + contrib/filter-extensions/rand/Makefile | 28 +++++ contrib/filter-extensions/rand/rand.c | 103 ++++++++++++++++++ contrib/filter-extensions/rand_cpp/.gitignore | 2 + contrib/filter-extensions/rand_cpp/Makefile | 34 ++++++ .../rand_cpp/adapter_functions.c | 6 + .../rand_cpp/adapter_functions.h | 10 ++ contrib/filter-extensions/rand_cpp/rand.cpp | 103 ++++++++++++++++++ 8 files changed, 288 insertions(+) create mode 100644 contrib/filter-extensions/rand/.gitignore create mode 100644 contrib/filter-extensions/rand/Makefile create mode 100644 contrib/filter-extensions/rand/rand.c create mode 100644 contrib/filter-extensions/rand_cpp/.gitignore create mode 100644 contrib/filter-extensions/rand_cpp/Makefile create mode 100644 contrib/filter-extensions/rand_cpp/adapter_functions.c create mode 100644 contrib/filter-extensions/rand_cpp/adapter_functions.h create mode 100644 contrib/filter-extensions/rand_cpp/rand.cpp diff --git a/contrib/filter-extensions/rand/.gitignore b/contrib/filter-extensions/rand/.gitignore new file mode 100644 index 00000000000..9eca6c88cf2 --- /dev/null +++ b/contrib/filter-extensions/rand/.gitignore @@ -0,0 +1,2 @@ +*.a +*.o diff --git a/contrib/filter-extensions/rand/Makefile b/contrib/filter-extensions/rand/Makefile new file mode 100644 index 00000000000..267221ee952 --- /dev/null +++ b/contrib/filter-extensions/rand/Makefile @@ -0,0 +1,28 @@ +# Run this via `FILTER_EXTENSIONS=contrib/filter-extensions/rand/rand.a make` +# from the main git directory. That way we inherit useful variables. + +ifneq ($(findstring s,$(MAKEFLAGS)),s) +ifndef V + QUIET_CC = @echo ' ' CC $@; + QUIET_AR = @echo ' ' AR $@; +endif +endif + +FILTER_STATIC_LIB = rand.a + +all: $(FILTER_STATIC_LIB) +ifeq ($(MAKELEVEL),0) + $(error "Run via parent git make") +endif + @: + +$(FILTER_STATIC_LIB): rand.o + $(QUIET_AR)$(AR) $(ARFLAGS) $@ $^ + +rand.o: rand.c + $(QUIET_CC)$(CC) -c $(ALL_CFLAGS) $< + +clean: + $(RM) $(FILTER_STATIC_LIB) rand.o + +.PHONY: all clean diff --git a/contrib/filter-extensions/rand/rand.c b/contrib/filter-extensions/rand/rand.c new file mode 100644 index 00000000000..af153709345 --- /dev/null +++ b/contrib/filter-extensions/rand/rand.c @@ -0,0 +1,103 @@ +#include "../../../git-compat-util.h" +#include "../../../list-objects-filter-extensions.h" +#include "../../../object.h" +#include "../../../hash.h" +#include "../../../trace.h" + + +static struct trace_key trace_filter = TRACE_KEY_INIT(FILTER); + +struct rand_context { + int percentageMatch; + int matchCount; + int blobCount; + int treeCount; + uint64_t started_at; +}; + +static int rand_init( + const struct repository *r, + const char *filter_arg, + void **context) +{ + struct rand_context *ctx = calloc(1, sizeof(struct rand_context)); + + ctx->percentageMatch = atoi(filter_arg); + if (ctx->percentageMatch > 100 || ctx->percentageMatch < 0) { + fprintf(stderr, "filter-rand: warning: invalid match %%: %s\n", + filter_arg); + ctx->percentageMatch = 1; // default 1% + } + fprintf(stderr, "filter-rand: matching %d%%\n", ctx->percentageMatch); + ctx->started_at = getnanotime(); + (*context) = ctx; + + return 0; +} + +static enum list_objects_filter_result rand_filter_object( + const struct repository *r, + const enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + enum list_objects_filter_omit *omit, + void *context) +{ + struct rand_context *ctx = (struct rand_context*)(context); + + if ((ctx->blobCount + ctx->treeCount + 1) % 100000 == 0) { + fprintf(stderr, "filter-rand: %d...\n", + (ctx->blobCount + ctx->treeCount + 1)); + } + + switch (filter_situation) { + default: + die("filter-rand: unknown filter_situation: %d", filter_situation); + + case LOFS_BEGIN_TREE: + ctx->treeCount++; + /* always include all tree objects */ + return LOFR_MARK_SEEN | LOFR_DO_SHOW; + + case LOFS_END_TREE: + return LOFR_ZERO; + + case LOFS_BLOB: + ctx->blobCount++; + + if ((rand() % 100) < ctx->percentageMatch) { + ctx->matchCount++; + trace_printf_key(&trace_filter, + "match: %s %s\n", + oid_to_hex(&obj->oid), + pathname + ); + return LOFR_MARK_SEEN | LOFR_DO_SHOW; + } else { + *omit = LOFO_OMIT; + return LOFR_MARK_SEEN; /* hard omit */ + } + } +} + +static void rand_free(const struct repository *r, void *context) +{ + struct rand_context *ctx = (struct rand_context*)(context); + double elapsed = (getnanotime() - ctx->started_at)/1E9; + int count = ctx->blobCount + ctx->treeCount; + + fprintf(stderr, "filter-rand: done: count=%d (blob=%d tree=%d) " + "matched=%d elapsed=%fs rate=%0.1f/s average=%0.1fus\n", + count, ctx->blobCount, ctx->treeCount, ctx->matchCount, + elapsed, count/elapsed, elapsed/count*1E6); + + free(ctx); +} + +const struct filter_extension filter_extension_rand = { + "rand", + &rand_init, + &rand_filter_object, + &rand_free, +}; diff --git a/contrib/filter-extensions/rand_cpp/.gitignore b/contrib/filter-extensions/rand_cpp/.gitignore new file mode 100644 index 00000000000..9eca6c88cf2 --- /dev/null +++ b/contrib/filter-extensions/rand_cpp/.gitignore @@ -0,0 +1,2 @@ +*.a +*.o diff --git a/contrib/filter-extensions/rand_cpp/Makefile b/contrib/filter-extensions/rand_cpp/Makefile new file mode 100644 index 00000000000..278121e3d5a --- /dev/null +++ b/contrib/filter-extensions/rand_cpp/Makefile @@ -0,0 +1,34 @@ +# Run this via `FILTER_EXTENSIONS=contrib/filter-extensions/rand_cpp/rand_cpp.a make` +# from the main git directory. That way we inherit useful variables. + +ifneq ($(findstring s,$(MAKEFLAGS)),s) +ifndef V + QUIET_CC = @echo ' ' CC $@; + QUIET_CXX = @echo ' ' CXX $@; + QUIET_AR = @echo ' ' AR $@; +endif +endif + +FILTER_STATIC_LIB = rand_cpp.a + +ALL_CXXFLAGS += -std=c++11 + +all: $(FILTER_STATIC_LIB) +ifeq ($(MAKELEVEL),0) + $(error "Run via parent git make") +endif + @: + +$(FILTER_STATIC_LIB): rand.o adapter_functions.o + $(QUIET_AR)$(AR) $(ARFLAGS) $@ $^ + +rand.o: rand.cpp + $(QUIET_CXX)$(CXX) -c $(ALL_CFLAGS) $(ALL_CXXFLAGS) $< + +adapter_functions.o: adapter_functions.c + $(QUIET_CC)$(CC) -c $(ALL_CFLAGS) $< + +clean: + $(RM) $(FILTER_STATIC_LIB) rand.o + +.PHONY: all clean diff --git a/contrib/filter-extensions/rand_cpp/adapter_functions.c b/contrib/filter-extensions/rand_cpp/adapter_functions.c new file mode 100644 index 00000000000..0d9d2a2aa96 --- /dev/null +++ b/contrib/filter-extensions/rand_cpp/adapter_functions.c @@ -0,0 +1,6 @@ +#include "../../../git-compat-util.h" +#include "../../../object.h" + +char *obj_to_hex_oid(struct object *obj) { + return oid_to_hex(&obj->oid); +} diff --git a/contrib/filter-extensions/rand_cpp/adapter_functions.h b/contrib/filter-extensions/rand_cpp/adapter_functions.h new file mode 100644 index 00000000000..1150c21a258 --- /dev/null +++ b/contrib/filter-extensions/rand_cpp/adapter_functions.h @@ -0,0 +1,10 @@ +#ifndef RAND_CPP_ADAPTER_FUNCTIONS_H +#define RAND_CPP_ADAPTER_FUNCTIONS_H + +struct object; + +uint64_t getnanotime(void); + +char *obj_to_hex_oid(struct object *obj); + +#endif /* RAND_CPP_ADAPTER_FUNCTIONS_H */ diff --git a/contrib/filter-extensions/rand_cpp/rand.cpp b/contrib/filter-extensions/rand_cpp/rand.cpp new file mode 100644 index 00000000000..cb608d14ed9 --- /dev/null +++ b/contrib/filter-extensions/rand_cpp/rand.cpp @@ -0,0 +1,103 @@ +#include +#include +#include + +#include + +extern "C" { + #include "../../../list-objects-filter-extensions.h" + #include "adapter_functions.h" +} + +namespace { + +struct rand_context { + int percentageMatch = 0; + int matchCount = 0; + int blobCount = 0; + int treeCount = 0; + uint64_t started_at = 0; +}; + +static int rand_init( + const struct repository *r, + const char *filter_arg, + void **context) +{ + struct rand_context *ctx = new rand_context(); + + ctx->percentageMatch = atoi(filter_arg); + if (ctx->percentageMatch > 100 || ctx->percentageMatch < 0) { + std::cerr << "filter-rand-cpp: warning: invalid match %: " << filter_arg << "\n"; + ctx->percentageMatch = 1; // default 1% + } + std::cerr << "filter-rand-cpp: matching " << ctx->percentageMatch << "%\n"; + ctx->started_at = getnanotime(); + + return 0; +} + +enum list_objects_filter_result rand_filter_object( + const struct repository *r, + const enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + enum list_objects_filter_omit *omit, + void *context) +{ + struct rand_context *ctx = static_cast(context); + + if ((ctx->blobCount + ctx->treeCount + 1) % 100000 == 0) { + std::cerr << "filter-rand-cpp: " << (ctx->blobCount + ctx->treeCount + 1) << "...\n"; + } + switch (filter_situation) { + default: + std::cerr << "filter-rand-cpp: unknown filter_situation: " << filter_situation << "\n"; + abort(); + + case LOFS_BEGIN_TREE: + ctx->treeCount++; + /* always include all tree objects */ + return static_cast(LOFR_MARK_SEEN | LOFR_DO_SHOW); + + case LOFS_END_TREE: + return LOFR_ZERO; + + case LOFS_BLOB: + ctx->blobCount++; + + if ((rand() % 100) < ctx->percentageMatch) { + ctx->matchCount++; + std::cout << "match: " << obj_to_hex_oid(obj) << pathname << "\n"; + return static_cast(LOFR_MARK_SEEN | LOFR_DO_SHOW); + } else { + *omit = LOFO_OMIT; + return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ + } + } +} + +void rand_free(const struct repository *r, void *context) { + struct rand_context *ctx = static_cast(context); + double elapsed = (getnanotime() - ctx->started_at)/1E9; + int count = ctx->blobCount + ctx->treeCount; + + std::cerr << "filter-rand-cpp: done: count=" << count + << " (blob=" << ctx->blobCount << " tree=" << ctx->treeCount << ")" + << " matched=" << ctx->matchCount + << " elapsed=" << elapsed << "s" + << " rate=" << count/elapsed << "/s" + << " average=" << elapsed/count*1E6 << "us\n"; + + delete ctx; +} + +} // namespace + +extern const struct filter_extension filter_extension_rand_cpp = { + "rand_cpp", + &rand_init, + &rand_filter_object, + &rand_free, +}; From patchwork Sun Sep 5 23:51:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Olsen X-Patchwork-Id: 12476173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9B15C43217 for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BF4A260EBA for ; Sun, 5 Sep 2021 23:52:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236681AbhIEXwu (ORCPT ); Sun, 5 Sep 2021 19:52:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235390AbhIEXws (ORCPT ); Sun, 5 Sep 2021 19:52:48 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAE10C061575 for ; Sun, 5 Sep 2021 16:51:44 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id z9-20020a7bc149000000b002e8861aff59so3742272wmi.0 for ; Sun, 05 Sep 2021 16:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EKbwJxXY1lIIETVYLLHHtfH4gsX5lwzIvam74mdf87s=; b=YINCKMq3QoOiyWSKWzswSprau4lDDgH6CvYZiKNLne3kfws92iChPqOesee7/JzEC0 DK66x1A4q7tfHLhNFWXZji8phOREqSRg9ISyxs98GaooOBcPtK1jmMYwohudn63VEMJw xNdJHWoW2sesKTN8w2dpOk7l1U41of7PCZScWqldJbZhk5Oq2yW25/jWw+lfAMGGSFXU R1gHkbMCed+EAzIArZL/eUm+Yj4PkKjG/yUSUSyXSwR61e0BQVQkwtHZwerfe0egXIkV mYkX/exASMphcghZLXJhrUvCk7oDyyosySZ+zSnSZoT+YRb+pKgPSvFJ3kQpW/daWaC9 L75g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EKbwJxXY1lIIETVYLLHHtfH4gsX5lwzIvam74mdf87s=; b=rX7jXT0EwR0FeTCSl6zsoJ3Hiie4kIWwQRyaLE6VXErfM7cartDwmM932V2MwEe75G BmdQmIqFIt4rsziNIpIGnHgdzmqQdxnolTm+M1w6DCFr/bJP1/ZYlrwhhYGLL9t/2KRu 1ZAIlH5fxEthlCKeNjTk4OtwGnSV8YHtmt6ExXHoRi+aBa7pHBOU7MCGZz1QX1q4ijBT jmyEW8NIbLogpuvkiSMRu9qZw74uEk9GrzvCOdQncTR3guyszcLY1RbbW47Fvx7nJ+8Y 08vKdBF3Z1oBUGd8zf/a2elHN+we3pMWBsnARG+sO1Q5SJZQMbRRhpJH6+K1MvKMclSF aJ2g== X-Gm-Message-State: AOAM531nmvOQrpuWzooy5xcMz35Ho5EV0S1WpTLXEgEjvvInHbQcohpe xc0BENZJqmkE9axAUNPJYDovtXSk2HQ= X-Google-Smtp-Source: ABdhPJw0uWF18Dz2IlSuaN7Ylyt4rfdOXskLG1AbhdSkkmG6eWIuc9bBhyQeyEL9IjYGiW6aAPQOoA== X-Received: by 2002:a1c:1d84:: with SMTP id d126mr8540229wmd.160.1630885903221; Sun, 05 Sep 2021 16:51:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l124sm5705264wml.8.2021.09.05.16.51.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Sep 2021 16:51:42 -0700 (PDT) Message-Id: <22475475cae2d32dfa664709f283e4c6e8f60e83.1630885899.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sun, 05 Sep 2021 23:51:39 +0000 Subject: [PATCH 4/4] Documentation for list-object-filter extensions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Andrew Olsen , Andrew Olsen Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Andrew Olsen From: Andrew Olsen Explains how to develop a custom extension for list-objects-filter behavior, and how to compile it into a custom build of Git using the FILTER_EXTENSIONS Makefile argument. Signed-off-by: Andrew Olsen --- Documentation/config/uploadpack.txt | 7 +- Documentation/rev-list-options.txt | 4 + contrib/filter-extensions/README.txt | 153 +++++++++++++++++++++++++++ 3 files changed, 161 insertions(+), 3 deletions(-) create mode 100644 contrib/filter-extensions/README.txt diff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt index 32fad5bbe81..b2ef2421a6d 100644 --- a/Documentation/config/uploadpack.txt +++ b/Documentation/config/uploadpack.txt @@ -66,9 +66,10 @@ uploadpackfilter.allow:: uploadpackfilter..allow:: Explicitly allow or ban the object filter corresponding to ``, where `` may be one of: `blob:none`, - `blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`. - If using combined filters, both `combine` and all of the nested - filter kinds must be allowed. Defaults to `uploadpackfilter.allow`. + `blob:limit`, `tree`, `sparse:oid`, `combine`, or a named filter extension + `extension:`. If using combined filters, both `combine` and all of + the nested filter kinds must be allowed. Defaults to + `uploadpackfilter.allow`. uploadpackfilter.tree.maxDepth:: Only allow `--filter=tree:` when `` is no more than the value of diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index b7bd27e1713..d7a317f0aa1 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -914,6 +914,10 @@ Note that the form '--filter=sparse:path=' that wants to read from an arbitrary path on the filesystem has been dropped for security reasons. + +The form '--filter=extension:[=]' uses a compile-time extension +to implement a named filter. Filter extensions may take an argument string +which is passed via ``. ++ Multiple '--filter=' flags can be specified to combine filters. Only objects which are accepted by every filter are included. + diff --git a/contrib/filter-extensions/README.txt b/contrib/filter-extensions/README.txt new file mode 100644 index 00000000000..3d5921cda9b --- /dev/null +++ b/contrib/filter-extensions/README.txt @@ -0,0 +1,153 @@ += List-Objects-Filter Extensions API +:pp: {plus}{plus} + +This API can be used to develop filter extensions used for custom filtering +behaviour with `git-upload-pack` and `git-rev-list`. The API is defined in +link:../../list-objects-filter-extensions.h[list-objects-filter-extensions.h] +and defines three functions to implement a filter operation. + +NOTE: Each filter implementing this API must compiled into Git as a +static library. There is some plumbing in the Makefile to help with this +via `FILTER_EXTENSIONS`. + +== Overview + +. You write a filter and compile it into your custom build of git. +. A filter request is received that specifically names the filter extension +that you have written, ie: `--filter=extension:[=]` +. The `init_fn` function of your filter is called. +. The `filter_object_fn` function of your filter is called for each object +at least once. +. The `free_fn` function of your filter is called. + +== Examples + +*link:./rand/[`rand`]* is a filter that matches all trees and a random +percentage of blobs, where the percentage is parsed from the filter arg. It +imports and uses the `oid_to_hex()` and `trace_key_printf()` functions from the +Git API. + +Build via: + +[,console] +---- +$ make FILTER_EXTENSIONS=contrib/filter-extensions/rand/rand.a + ... + SUBDIR contrib/filter-extensions/rand + ... +---- + +We can run against git's own repo: + +[,console] +---- +$ ./git rev-list refs/heads/master --objects --max-count 1 --filter=extension:rand=3 --filter-print-omitted | grep -c '^~' +filter-rand: matching 3% +filter-rand: done: count=4068 (blob=3866 tree=202) matched=117 elapsed=0.005017s rate=810843.1/s average=1.2us +3749 # number of omitted blobs = 3866 - 117 +---- + +== Development + +See the examples for a basic implementation. The comments in +link:../../list-objects-filter.h[`list-objects-filter.h`] and the built-in +filter implementations in +link:../../list-objects-filter.c[`list-objects-filter.c`] are important to +understand how filters are implemented - `filter_blobs_limit()` provides a +simple example, and `filter_sparse()` is more complex. + +The API differences between the built-in filters and the filter extensions: + +. Filter extensions don't handle ``omitset``s directly, instead setting `omit`. +. Filter extensions receive a void pointer they can use for context. + +== Building + +There is some plumbing in the Git Makefile to help with this via +`FILTER_EXTENSIONS`, setting it to space-separated paths of the filter extension +static libraries indicates that these filters should be compiled into git. +For example: + +[,console] +---- +make FILTER_EXTENSIONS=contrib/filter-extensions/rand/rand.a +---- + +Filter extensions don't need to be within the Git source tree. A filter +extension static library should either exist at the given path - ie, `rand.a` +should exist - or there should be a Makefile in that directory which will create +it when `make rand.a` is run. (Such a Makefile should also have a `clean` target +which deletes all object files and brings the directory back to its initial +state). + +The static library should define a struct of type `filter_extension` called +`filter_extension_NAME` where `NAME` is the name of your extension (ie `rand` +for `rand.a`). See +link:../../list-objects-filter-extensions.h[list-objects-filter-extensions.h] + +This definition should follow the following pattern: + +[,C] +---- +#include "list-objects-filter-extensions.h" + +/* Definitions of rand_init, rand_filter_object, rand_free ... */ + +const struct filter_extension filter_extension_rand = { + "rand", + &rand_init, + &rand_filter_object, + &rand_free, +}; +---- + +(The names of your `init_fn`, `filter_object_fn` and `free_fn` are not +important, but the string literal should again be the the name of your extension +- `"rand"` for the filter extension in `rand.a`.) + +You may use library functions from Git if you include the relevant Git headers, +since the filter extensions and Git itself will be linked together into a single +binary. + +You may depend on other libraries if you indicate that they are to be linked +into the Git binary using `LDFLAGS`. See the C{pp} example below. + +== Developing in C{pp} (and other languages) + +You can develop filter extensions with C{pp}, but many Git header files are not +compatible with modern C{pp}, so you won't be able to directly use Git library +functions. However, you can use them if you create wrapper functions in C that +delegates to the Git library functions you need, but which are also C{pp} +compatible. See link:./rand_cpp/[`rand_cpp`] for a simple example. A similar +solution would be to implement the extension itself in C, and have the +extension do any operations that require Git library functions, but have it +delegate to a C wrapper API that you add to a C{pp} library that already +contains the domain-specific operations that you need. In either case, remember +to wrap any functions that must be C-compatible with `extern C` when declaring +or defining them from within C{pp}. + +To build the C{pp} example: + +[,console] +---- +make FILTER_EXTENSIONS=contrib/filter-extensions/rand_cpp/rand_cpp.a \ + LDFLAGS=-lstdc++ +---- + +For other languages you'll either need to port definitions of some internal Git +structs (at a minimum, `object`, `object_id`, `repository`, and `hash_algo`) - +or again, you could write the extension in C but have it delegate to a domain +specific library in the language of your choice that has a C-compatible API. +Extra libraries can be required using `LDFLAGS`. + +== Linking more than one filter extension + +To link in more than one extension, set `FILTER_EXTENSIONS` to the +space-separated paths of all the extensions you want linked. For example, to +link in both example filters at once: + +[,console] +---- +make FILTER_EXTENSIONS="contrib/filter-extensions/rand/rand.a contrib/filter-extensions/rand_cpp/rand_cpp.a" \ + LDFLAGS=-lstdc++ +----