From patchwork Sat Jul 31 17:27:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B01CC4320E for ; Sat, 31 Jul 2021 17:27:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4A30260FE7 for ; Sat, 31 Jul 2021 17:27:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230441AbhGaR1v (ORCPT ); Sat, 31 Jul 2021 13:27:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229840AbhGaR1u (ORCPT ); Sat, 31 Jul 2021 13:27:50 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A527AC0613CF for ; Sat, 31 Jul 2021 10:27:42 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id c16so15692613wrp.13 for ; Sat, 31 Jul 2021 10:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Yzkzbi/Slg3JVOXhChvUByogsyK3fYn9Uf+8FgTW9dE=; b=GnFRTSRunx2P9y7EslAKqBSLmIK+D2XC8Tb7MZKjWtNapjESVLsM91QO8JXOc2GN2j GPMWS7xDRXd3BxCwI7mmvFSMJFckLad6b6PVSOYNLXVB3H4+9ptXy+MNmbjqJMM/jqnU D86NP0TsoPwYV91CQR1V85K+7oDhbmdZKAGl82oTZ97J1qlVVw+5OzCh0e6mSUWW1ln4 s7QLfZLZyMf379U/5riUsGSjz9ZWF/uSBwDyYxB1hJHaaUqlVEuLrtcW+VBcDNXkMEWU nSX9eqK/ICA7CsjQ2n0lXiUAUe7eKM8cXKTXOmAYrgHWPlb4Q54jUWd5woMSVqMCQWDV e7Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Yzkzbi/Slg3JVOXhChvUByogsyK3fYn9Uf+8FgTW9dE=; b=DsHk1OcPgC4R561D6NYMpXOsqUh/ZgNEU1VrINo4sAJJi8sbah+PzoKNrXeUUtJZyY Kb8I7hA2ITvrp3J4AMDkI8zAz66+3cIMxri/PhnHDbhH7UJnNdHXjqUwnmOxvlJ6TuEx LMK8svRnERoRFshkEzF7Ysh9ZcKBJx4n+lZCWiFEtfMYnqgYuPMKEErCFZShFhKXcUr6 lZMxx98y/FmGoVNgBLuwEFrL//nr9DRLtOkaIRP1AR0VqW4CESSYURv5/VDF4FAlmM1r Ej4mCDPUZy102PRSQBUW4iJP/Phxku5GOFY4L4fUekQdL2WJHSc+qWKWAKRZl7OH/n1h TRgA== X-Gm-Message-State: AOAM531xqpln9VsdOcSYoav6IokswBo4S0bByo4y6qtx3bvcJY243CSC 0ju1mv8920AurlxsSY35/ofSLerhX4U= X-Google-Smtp-Source: ABdhPJwgMw/zvtQ73BL9nf4S0LXbF34xKpyJxxMLQ9qiGYhxu5TrpFVH/Ygoe+0dh8pjUjqDD4ew8Q== X-Received: by 2002:adf:cd92:: with SMTP id q18mr9153112wrj.18.1627752461306; Sat, 31 Jul 2021 10:27:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m4sm4989680wmq.7.2021.07.31.10.27.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:40 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:30 +0000 Subject: [PATCH v4 1/9] merge-ort: rename str{map,intmap,set}_func() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren In order to make it clearer that these three variables holding a function refer to functions that will clear the strmap/strintmap/strset, rename them to str{map,intmap,set}_clear_func(). Signed-off-by: Elijah Newren --- merge-ort.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index e75b524153e..401a40247a3 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -519,11 +519,11 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, { struct rename_info *renames = &opti->renames; int i; - void (*strmap_func)(struct strmap *, int) = + void (*strmap_clear_func)(struct strmap *, int) = reinitialize ? strmap_partial_clear : strmap_clear; - void (*strintmap_func)(struct strintmap *) = + void (*strintmap_clear_func)(struct strintmap *) = reinitialize ? strintmap_partial_clear : strintmap_clear; - void (*strset_func)(struct strset *) = + void (*strset_clear_func)(struct strset *) = reinitialize ? strset_partial_clear : strset_clear; /* @@ -534,14 +534,14 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, * to deallocate them. */ free_strmap_strings(&opti->paths); - strmap_func(&opti->paths, 1); + strmap_clear_func(&opti->paths, 1); /* * All keys and values in opti->conflicted are a subset of those in * opti->paths. We don't want to deallocate anything twice, so we * don't free the keys and we pass 0 for free_values. */ - strmap_func(&opti->conflicted, 0); + strmap_clear_func(&opti->conflicted, 0); /* * opti->paths_to_free is similar to opti->paths; we created it with @@ -559,24 +559,24 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, /* Free memory used by various renames maps */ for (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) { - strintmap_func(&renames->dirs_removed[i]); - strmap_func(&renames->dir_renames[i], 0); - strintmap_func(&renames->relevant_sources[i]); + strintmap_clear_func(&renames->dirs_removed[i]); + strmap_clear_func(&renames->dir_renames[i], 0); + strintmap_clear_func(&renames->relevant_sources[i]); if (!reinitialize) assert(renames->cached_pairs_valid_side == 0); if (i != renames->cached_pairs_valid_side && -1 != renames->cached_pairs_valid_side) { - strset_func(&renames->cached_target_names[i]); - strmap_func(&renames->cached_pairs[i], 1); - strset_func(&renames->cached_irrelevant[i]); + strset_clear_func(&renames->cached_target_names[i]); + strmap_clear_func(&renames->cached_pairs[i], 1); + strset_clear_func(&renames->cached_irrelevant[i]); partial_clear_dir_rename_count(&renames->dir_rename_count[i]); if (!reinitialize) strmap_clear(&renames->dir_rename_count[i], 1); } } for (i = MERGE_SIDE1; i <= MERGE_SIDE2; ++i) { - strintmap_func(&renames->deferred[i].possible_trivial_merges); - strset_func(&renames->deferred[i].target_dirs); + strintmap_clear_func(&renames->deferred[i].possible_trivial_merges); + strset_clear_func(&renames->deferred[i].target_dirs); renames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */ } renames->cached_pairs_valid_side = 0; From patchwork Sat Jul 31 17:27:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02B33C4338F for ; Sat, 31 Jul 2021 17:27:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CB90C61042 for ; Sat, 31 Jul 2021 17:27:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230196AbhGaR1u (ORCPT ); Sat, 31 Jul 2021 13:27:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229694AbhGaR1u (ORCPT ); Sat, 31 Jul 2021 13:27:50 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5223DC0613D3 for ; Sat, 31 Jul 2021 10:27:43 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id d131-20020a1c1d890000b02902516717f562so8426353wmd.3 for ; Sat, 31 Jul 2021 10:27:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=AXse4muAeaYC3gLbixhZ8tneVeIz+kqzyPqgyg820Pk=; b=nS0VXZ5ujYoNb5Thi+kNXBD+9FGjqSBYNfZTR/CKsGtfl3ySCYGOpXvBwcFNNoJtRP aTqCUnSQthZLbeakcyBa5zOW8BNoy/BUGcrE/EPUBCj4cEAyRK+fknZLQ6yqeLR55ZaY H5kICoedd3ne5UeSbVsKoTzvWNmmEKXANq/Q0JBrL80dL2yKGNZkR/0jlbp7ElC6RPzG PM4r+I5m/Zyp+2Y0JZFb/oSrewlmWk0l1AFF5A6w7Hau2I4llpouT75cLrXlqDw4fK4E sD9MMI2AxXkC/+GNUmWF00ppBqgOBSHs1FtyksXjWStrnUe1wYCbCJ45usS4Wkdpg6un +f8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=AXse4muAeaYC3gLbixhZ8tneVeIz+kqzyPqgyg820Pk=; b=MnkbgU2JuGdeYrUYszKboM60x2z55pwVpbb1CBqBvn7VVCmDJ6k0Fps0Kb+yG3Iv8J mU6NmC11aooNpt00o9/9n0e2XW4wb1Ia/AqadlAL3qqivEjUFQegil+TorWGrttx9YKw R/oB42+36332DCdcFQxzDacN45/SReEKYcuMYKOMngOLfEru8XMuXKPfeqw2naUTIboD ezWZUPcJlAh44D28tl8HzzAi/dQBi5sM1ASD1+2CPzlmoyHEhib8DFCcUfLIK3YxvqmC o0/WEFFo9Iin+wMNschM5kQ8b9jOrhFLPQ3A8sNBMYgsVISNyzp4RCrlVjwoWVEVHT8a UK+g== X-Gm-Message-State: AOAM533ZSGTIeNYauMN1f3ZFtPsZAQ1X/x4u129yRqlf7wAW1e1CkJ67 VTEKyJ3d8Uyx6xuHQ4A09SUzUuQvpNk= X-Google-Smtp-Source: ABdhPJxDJ2+9Ml5m11uyZs05Yd4DPWE4sQ656pFXUOnaFoo0P0CPmFPpwyd1OP8XrePAZGd1yBjCnA== X-Received: by 2002:a05:600c:154d:: with SMTP id f13mr8928188wmg.0.1627752461976; Sat, 31 Jul 2021 10:27:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v6sm5491361wru.50.2021.07.31.10.27.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:41 -0700 (PDT) Message-Id: <8416afa89fb9d1e638bf6f52403f113c82de7424.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:31 +0000 Subject: [PATCH v4 2/9] diffcore-rename: use a mem_pool for exact rename detection's hashmap MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Exact rename detection, via insert_file_table(), uses a hashmap to store files by oid. Use a mem_pool for the hashmap entries so these can all be allocated and deallocated together. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 204.2 ms ± 3.0 ms 202.5 ms ± 3.2 ms mega-renames: 1.076 s ± 0.015 s 1.072 s ± 0.012 s just-one-mega: 364.1 ms ± 7.0 ms 357.3 ms ± 3.9 ms Signed-off-by: Elijah Newren --- diffcore-rename.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/diffcore-rename.c b/diffcore-rename.c index 4ef0459cfb5..73d884099eb 100644 --- a/diffcore-rename.c +++ b/diffcore-rename.c @@ -317,10 +317,11 @@ static int find_identical_files(struct hashmap *srcs, } static void insert_file_table(struct repository *r, + struct mem_pool *pool, struct hashmap *table, int index, struct diff_filespec *filespec) { - struct file_similarity *entry = xmalloc(sizeof(*entry)); + struct file_similarity *entry = mem_pool_alloc(pool, sizeof(*entry)); entry->index = index; entry->filespec = filespec; @@ -336,7 +337,8 @@ static void insert_file_table(struct repository *r, * and then during the second round we try to match * cache-dirty entries as well. */ -static int find_exact_renames(struct diff_options *options) +static int find_exact_renames(struct diff_options *options, + struct mem_pool *pool) { int i, renames = 0; struct hashmap file_table; @@ -346,7 +348,7 @@ static int find_exact_renames(struct diff_options *options) */ hashmap_init(&file_table, NULL, NULL, rename_src_nr); for (i = rename_src_nr-1; i >= 0; i--) - insert_file_table(options->repo, + insert_file_table(options->repo, pool, &file_table, i, rename_src[i].p->one); @@ -354,8 +356,8 @@ static int find_exact_renames(struct diff_options *options) for (i = 0; i < rename_dst_nr; i++) renames += find_identical_files(&file_table, i, options); - /* Free the hash data structure and entries */ - hashmap_clear_and_free(&file_table, struct file_similarity, entry); + /* Free the hash data structure (entries will be freed with the pool) */ + hashmap_clear(&file_table); return renames; } @@ -1341,6 +1343,7 @@ void diffcore_rename_extended(struct diff_options *options, int num_destinations, dst_cnt; int num_sources, want_copies; struct progress *progress = NULL; + struct mem_pool local_pool; struct dir_rename_info info; struct diff_populate_filespec_options dpf_options = { .check_binary = 0, @@ -1409,11 +1412,18 @@ void diffcore_rename_extended(struct diff_options *options, goto cleanup; /* nothing to do */ trace2_region_enter("diff", "exact renames", options->repo); + mem_pool_init(&local_pool, 32*1024); /* * We really want to cull the candidates list early * with cheap tests in order to avoid doing deltas. */ - rename_count = find_exact_renames(options); + rename_count = find_exact_renames(options, &local_pool); + /* + * Discard local_pool immediately instead of at "cleanup:" in order + * to reduce maximum memory usage; inexact rename detection uses up + * a fair amount of memory, and mem_pools can too. + */ + mem_pool_discard(&local_pool, 0); trace2_region_leave("diff", "exact renames", options->repo); /* Did we only want exact renames? */ From patchwork Sat Jul 31 17:27:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03713C4338F for ; Sat, 31 Jul 2021 17:27:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA26961050 for ; Sat, 31 Jul 2021 17:27:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231209AbhGaR1z (ORCPT ); Sat, 31 Jul 2021 13:27:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230408AbhGaR1v (ORCPT ); Sat, 31 Jul 2021 13:27:51 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F734C06175F for ; Sat, 31 Jul 2021 10:27:44 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id l4-20020a05600c1d04b02902506f89ad2dso9482211wms.1 for ; Sat, 31 Jul 2021 10:27:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4W9WNGrc3ipNQPgdDdR8WMqa8vuANw1l/AK9OiUX2hw=; b=E+x2eig+Q4QHyQw33euTietB2TUouaN3oZNeHIuZ3C44nO6Dr+W5LOG2Tt+zQ2D9Ke bZbTiv2fMYsTbtqJp/rMBLB7LR3A8kF6R3D24hnP+hGwvx+50Ie3mTrc7IIxKH26/BCM rPdY9jn2FwqSeao7/dxEKD/CsWS5SedKyIE3zpxQjCK8B0A1Qs3CG9Ppq63Awp1GWQA9 YhBZINFCQVAWLScqKNLHOZXUOH5BHyv7hUPiBrwlLazBLhSmovuoadotpgRnXtQ4W0vo +zac5NxtQKEX5s6f1hFyS1FnvM0TBolXP5uj2oL8WOX5LknpdaGqQxzK6vvWUcP4ZsR4 5F7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4W9WNGrc3ipNQPgdDdR8WMqa8vuANw1l/AK9OiUX2hw=; b=gZPi14k7phQDX2E1bHu/Q/QqbJMMyvviiF9UF6naUvN4XiEPueQ6F8A65I2bxgw5ct 9GSGX3zu3EI40Lr1fWMkSMedftn/yKTqaY02EnLY8LMKx8dd8MB60fAxzB6NAcN4+dWJ OS5HiEpdgWRBfCBNCVxC0I4kXVfLqNXZ4ah+44EQZBl3zANcdfDLVmMhOwGSWFsTB4JQ JL0Uu0vyMRRxB32+r1yb0VMXLa94ihe8E53tF+qJDaD7i5fDFUsfqOz0uIBhUKuJoXQ4 3utdCc6B1QPJcyN4LnM1Dnim1lNG1Mq8U6MhibddZB9UYUxPbVfUiw5OLCTFA28qDKDp qSdA== X-Gm-Message-State: AOAM532vR/A/sL8sCcga0Y4EyMnp3Te30ZYHC/iTSB/igNTl3zaw/tjH boOPiJOZ8tGWN0bYn1f/ftJM4fnOwzc= X-Google-Smtp-Source: ABdhPJz1mAz5Fdu+35MZ/sIghEGi1U71x5qcYw6rM0AxqdfmYWtBt5nn0grZAQ2LZ74lHE2Gdbc6aQ== X-Received: by 2002:a7b:c7d1:: with SMTP id z17mr8964591wmk.50.1627752462795; Sat, 31 Jul 2021 10:27:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x21sm5394983wmi.46.2021.07.31.10.27.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:42 -0700 (PDT) Message-Id: <2c0b90eaba5bbf03553c8ac2487b57b66a6ed18e.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:32 +0000 Subject: [PATCH v4 3/9] merge-ort: add pool_alloc, pool_calloc, and pool_strndup wrappers Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Make the code more flexible so that it can handle both being run with or without a memory pool by adding utility functions which will either call xmalloc, xcalloc, xstrndup or mem_pool_alloc, mem_pool_calloc, mem_pool_strndup depending on whether we have a non-NULL memory pool. A subsequent commit will make use of these. (We will actually be dropping these functions soon and just assuming we always have a memory pool, but the flexibility was very useful during development of merge-ort so I want to be able to restore it if needed.) Signed-off-by: Elijah Newren --- merge-ort.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/merge-ort.c b/merge-ort.c index 401a40247a3..63f67246d3d 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -664,6 +664,30 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } +MAYBE_UNUSED +static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) +{ + if (!pool) + return xcalloc(count, size); + return mem_pool_calloc(pool, count, size); +} + +MAYBE_UNUSED +static void *pool_alloc(struct mem_pool *pool, size_t size) +{ + if (!pool) + return xmalloc(size); + return mem_pool_alloc(pool, size); +} + +MAYBE_UNUSED +static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len) +{ + if (!pool) + return xstrndup(str, len); + return mem_pool_strndup(pool, str, len); +} + /* add a string to a strbuf, but converting "/" to "_" */ static void add_flattened_path(struct strbuf *out, const char *s) { From patchwork Sat Jul 31 17:27:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DDC8C4338F for ; Sat, 31 Jul 2021 17:27:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 103ED61050 for ; Sat, 31 Jul 2021 17:27:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231293AbhGaR15 (ORCPT ); Sat, 31 Jul 2021 13:27:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229840AbhGaR1w (ORCPT ); Sat, 31 Jul 2021 13:27:52 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 065A9C0613CF for ; Sat, 31 Jul 2021 10:27:45 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id z4so15736340wrv.11 for ; Sat, 31 Jul 2021 10:27:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0G2Aq0f99MPciHnA4RscP5rIVPzkEOXnzeJ+kRWgLxI=; b=DY/+MM4FJcwZfhvMP5+Jm6m1VD0xYbSIVk5DFvcE4o/OJzGLnXyIIHBEbrSofzbKai ExrTwzcoOWiM6mEewZq/l18erBh0XDIuX3MJu8EhN+l3W2Ppc+hm7X7n38ckN96wS5Vm HZ7XQOmeUj416BK+HCmao9ARtL9onFv0OO9yB4U21ZBvqCyjtYfqJbRBcKp2v6OK3lFf YbP65Blq5SzAM1jRy7SMO8Svep8ZqBiQd0kowkQPJyM1HBkeyhHdgzbAZCzGbSGXmD+x 2NJuTLZ6ClXQDEgA3N9LD/bzC56O8jndsx5vJ18r8tDX+gvq53xOCabhvfW7Yr1zxDZz gG5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0G2Aq0f99MPciHnA4RscP5rIVPzkEOXnzeJ+kRWgLxI=; b=AeHtUTZWA2nFewLAEQfZXeTzEDyAE51crV6yHX+AADohzRSV1D3y+bjFGV7S5RrP/J E+QkGS3URnKMx6V5pFvz0H5g/AmxbhPEEiyjetLscDtXuJVFiQ3eXWn+COqP0vtCwOnT eebjudBEZ9Lze3CcEGOPhaH2bZcAyUO7rHYiByimqfwvFTpJ8mbAJYtaX/CAUuFzc4jg t0DbQQEYPLzrTIIRvzH/+Ohf3sfo/XjH3HtQKFElhPPzgoeBPz0GTs3djLi0sq++F6NV 1mb6nPCD62DuuC45Fjhazn0qDJNR3rzwNW5lIDieo3VbFKKiQsPrVog8BfcRY0ucR0zm ytIQ== X-Gm-Message-State: AOAM5323MqaZvQRiNp+XMgcaX5ldSqiosQItVkHcnYdAwtmtKC7/OpjH vHx/+jEaF1VHfotL9F6KU9pmgQz1Q3g= X-Google-Smtp-Source: ABdhPJyCWOPJMZUDkEDAFN/csozjtCq0kaKXleumjXOj7nmuYtELPMzD21EWFsOkGn/JrDKeQc0gPw== X-Received: by 2002:a5d:6d86:: with SMTP id l6mr9174314wrs.260.1627752463612; Sat, 31 Jul 2021 10:27:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b14sm5568032wrm.43.2021.07.31.10.27.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:43 -0700 (PDT) Message-Id: <6646f6fd1cae3b7ebe279317b95405f2b6d71e9e.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:33 +0000 Subject: [PATCH v4 4/9] merge-ort: set up a memory pool Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren merge-ort has a lot of data structures, and they all tend to be freed together in clear_or_reinit_internal_opts(). Set up a memory pool to allow us to make these allocations and deallocations faster. Future commits will adjust various callers to make use of this memory pool. Signed-off-by: Elijah Newren --- merge-ort.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/merge-ort.c b/merge-ort.c index 63f67246d3d..3f425436263 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -37,6 +37,8 @@ #include "unpack-trees.h" #include "xdiff-interface.h" +#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */ + /* * We have many arrays of size 3. Whenever we have such an array, the * indices refer to one of the sides of the three-way merge. This is so @@ -339,6 +341,17 @@ struct merge_options_internal { */ struct strmap conflicted; + /* + * pool: memory pool for fast allocation/deallocation + * + * We allocate room for lots of filenames and auxiliary data + * structures in merge_options_internal, and it tends to all be + * freed together too. Using a memory pool for these provides a + * nice speedup. + */ + struct mem_pool internal_pool; + struct mem_pool *pool; /* NULL, or pointer to internal_pool */ + /* * paths_to_free: additional list of strings to free * @@ -603,6 +616,12 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, strmap_clear(&opti->output, 0); } +#if USE_MEMORY_POOL + mem_pool_discard(&opti->internal_pool, 0); + if (!reinitialize) + opti->pool = NULL; +#endif + /* Clean out callback_data as well. */ FREE_AND_NULL(renames->callback_data); renames->callback_data_nr = renames->callback_data_alloc = 0; @@ -4381,6 +4400,12 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) /* Initialization of various renames fields */ renames = &opt->priv->renames; +#if USE_MEMORY_POOL + mem_pool_init(&opt->priv->internal_pool, 0); + opt->priv->pool = &opt->priv->internal_pool; +#else + opt->priv->pool = NULL; +#endif for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->dirs_removed[i], NOT_RELEVANT, NULL, 0); From patchwork Sat Jul 31 17:27:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11539C4320E for ; Sat, 31 Jul 2021 17:27:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EB29861050 for ; Sat, 31 Jul 2021 17:27:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231292AbhGaR1z (ORCPT ); Sat, 31 Jul 2021 13:27:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230477AbhGaR1w (ORCPT ); Sat, 31 Jul 2021 13:27:52 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC759C0613D3 for ; Sat, 31 Jul 2021 10:27:45 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id b11so10383128wrx.6 for ; Sat, 31 Jul 2021 10:27:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=+9eI3QJnaA9ojY4vp8WbYszLhiAHQHDYAfyuZDfzWVY=; b=MzPTZnm0TOIzJXT16OSfS1/D9dOyx1asDsBIiE8QIgCHrEFK9VegQqPDRLcKpwIbUX j0Kd5w6Ee4pAPgiV9mOaj0k3iOsYRkzl6Kei1ANqQQ4Tk6/gwZ3bwOZ89Mm2milEHlNs ZSoki02ONec0sluYjeHV4PKdwoi7KN2cZ460CtbQR2JD3kDoeASq/Av2EV4kVKdpnuJY ESJuMXPNC0N1wT9+KfFX/1a5LKldbrfOh2AsVijXWG2Md11q64G9lxAfk8aScg5hUDMo T64rL8SUG2O6tL8TjUtZoK0C35KiBpuuc9zhRUb50s5uD1K+q5R2HZquB2IV7n9tCw7T lU3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=+9eI3QJnaA9ojY4vp8WbYszLhiAHQHDYAfyuZDfzWVY=; b=K7IToAGyA2J66Aj2eDVkqsuSFOeO/RJY0y2k0b1ieakWx9VKRTXb+2KnOPxZn9n9AH 0nVFhji01wsAs9L7AmNfxM22rChLcuKaY+nUZJish0DTBhi+GsC/F3xpsMCd+v15XcKx 0xUVqizu+v0V39Xy+zQh3SLC3fs5z1XtgR9MWtP2spLIvtgWl4FOwkwS+MqwYxV7Odcx h+q0fCzaWegnzANMgoyS1AU+ECkfFKKJVu8DUCfjBlwpoOZ0buI1vogjlR/zqdYzibnj 5+OzNl7ZGGOetoQ9vtU74DevgMS8vbjAw292Di7UFzYszBgPrVpfxqp6cr+4j9PomEtN zlRw== X-Gm-Message-State: AOAM533NBHVdC7Oa4K8wTX5yqYd1ez04dUbAYcW90pLmLmg66R2E8UYA YfcuGAfCLBASzzlXUCBqnCoEi4mc4u0= X-Google-Smtp-Source: ABdhPJyhKq0PWgGEsD3Sl/AQwtPVnkTQz0Cl2qy1EjxwywfhogzI2Ur6PGkXDWLPfPXT1jQ+b/GO3A== X-Received: by 2002:a5d:4dc7:: with SMTP id f7mr9342360wru.118.1627752464345; Sat, 31 Jul 2021 10:27:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l2sm5324919wru.67.2021.07.31.10.27.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:43 -0700 (PDT) Message-Id: <7c49aa601d0b0880dde559579a336809d88a1c01.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:34 +0000 Subject: [PATCH v4 5/9] merge-ort: switch our strmaps over to using memory pools MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren For all the strmaps (including strintmaps and strsets) whose memory is unconditionally freed as part of clear_or_reinit_internal_opts(), switch them over to using our new memory pool. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 202.5 ms ± 3.2 ms 198.1 ms ± 2.6 ms mega-renames: 1.072 s ± 0.012 s 715.8 ms ± 4.0 ms just-one-mega: 357.3 ms ± 3.9 ms 276.8 ms ± 4.2 ms Signed-off-by: Elijah Newren --- merge-ort.c | 125 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 75 insertions(+), 50 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index 3f425436263..99c75690855 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -539,15 +539,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, void (*strset_clear_func)(struct strset *) = reinitialize ? strset_partial_clear : strset_clear; - /* - * We marked opti->paths with strdup_strings = 0, so that we - * wouldn't have to make another copy of the fullpath created by - * make_traverse_path from setup_path_info(). But, now that we've - * used it and have no other references to these strings, it is time - * to deallocate them. - */ - free_strmap_strings(&opti->paths); - strmap_clear_func(&opti->paths, 1); + if (opti->pool) + strmap_clear_func(&opti->paths, 0); + else { + /* + * We marked opti->paths with strdup_strings = 0, so that + * we wouldn't have to make another copy of the fullpath + * created by make_traverse_path from setup_path_info(). + * But, now that we've used it and have no other references + * to these strings, it is time to deallocate them. + */ + free_strmap_strings(&opti->paths); + strmap_clear_func(&opti->paths, 1); + } /* * All keys and values in opti->conflicted are a subset of those in @@ -556,16 +560,19 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, */ strmap_clear_func(&opti->conflicted, 0); - /* - * opti->paths_to_free is similar to opti->paths; we created it with - * strdup_strings = 0 to avoid making _another_ copy of the fullpath - * but now that we've used it and have no other references to these - * strings, it is time to deallocate them. We do so by temporarily - * setting strdup_strings to 1. - */ - opti->paths_to_free.strdup_strings = 1; - string_list_clear(&opti->paths_to_free, 0); - opti->paths_to_free.strdup_strings = 0; + if (!opti->pool) { + /* + * opti->paths_to_free is similar to opti->paths; we + * created it with strdup_strings = 0 to avoid making + * _another_ copy of the fullpath but now that we've used + * it and have no other references to these strings, it is + * time to deallocate them. We do so by temporarily + * setting strdup_strings to 1. + */ + opti->paths_to_free.strdup_strings = 1; + string_list_clear(&opti->paths_to_free, 0); + opti->paths_to_free.strdup_strings = 0; + } if (opti->attr_index.cache_nr) /* true iff opt->renormalize */ discard_index(&opti->attr_index); @@ -683,7 +690,6 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } -MAYBE_UNUSED static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) { if (!pool) @@ -691,7 +697,6 @@ static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) return mem_pool_calloc(pool, count, size); } -MAYBE_UNUSED static void *pool_alloc(struct mem_pool *pool, size_t size) { if (!pool) @@ -699,7 +704,6 @@ static void *pool_alloc(struct mem_pool *pool, size_t size) return mem_pool_alloc(pool, size); } -MAYBE_UNUSED static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len) { if (!pool) @@ -835,8 +839,9 @@ static void setup_path_info(struct merge_options *opt, assert(!df_conflict || !resolved); /* df_conflict implies !resolved */ assert(resolved == (merged_version != NULL)); - mi = xcalloc(1, resolved ? sizeof(struct merged_info) : - sizeof(struct conflict_info)); + mi = pool_calloc(opt->priv->pool, 1, + resolved ? sizeof(struct merged_info) : + sizeof(struct conflict_info)); mi->directory_name = current_dir_name; mi->basename_offset = current_dir_name_len; mi->clean = !!resolved; @@ -1128,7 +1133,7 @@ static int collect_merge_info_callback(int n, len = traverse_path_len(info, p->pathlen); /* +1 in both of the following lines to include the NUL byte */ - fullpath = xmalloc(len + 1); + fullpath = pool_alloc(opt->priv->pool, len + 1); make_traverse_path(fullpath, len + 1, info, p->path, p->pathlen); /* @@ -1383,7 +1388,7 @@ static int handle_deferred_entries(struct merge_options *opt, copy = renames->deferred[side].possible_trivial_merges; strintmap_init_with_options(&renames->deferred[side].possible_trivial_merges, 0, - NULL, + opt->priv->pool, 0); strintmap_for_each_entry(©, &iter, entry) { const char *path = entry->key; @@ -2335,12 +2340,21 @@ static void apply_directory_rename_modifications(struct merge_options *opt, VERIFY_CI(ci); /* Find parent directories missing from opt->priv->paths */ - cur_path = new_path; + if (opt->priv->pool) { + cur_path = mem_pool_strdup(opt->priv->pool, new_path); + free((char*)new_path); + new_path = (char *)cur_path; + } else { + cur_path = new_path; + } + while (1) { /* Find the parent directory of cur_path */ char *last_slash = strrchr(cur_path, '/'); if (last_slash) { - parent_name = xstrndup(cur_path, last_slash - cur_path); + parent_name = pool_strndup(opt->priv->pool, + cur_path, + last_slash - cur_path); } else { parent_name = opt->priv->toplevel_dir; break; @@ -2349,7 +2363,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt, /* Look it up in opt->priv->paths */ entry = strmap_get_entry(&opt->priv->paths, parent_name); if (entry) { - free((char*)parent_name); + if (!opt->priv->pool) + free((char*)parent_name); parent_name = entry->key; /* reuse known pointer */ break; } @@ -2376,12 +2391,15 @@ static void apply_directory_rename_modifications(struct merge_options *opt, parent_name = cur_dir; } - /* - * We are removing old_path from opt->priv->paths. old_path also will - * eventually need to be freed, but it may still be used by e.g. - * ci->pathnames. So, store it in another string-list for now. - */ - string_list_append(&opt->priv->paths_to_free, old_path); + if (!opt->priv->pool) { + /* + * We are removing old_path from opt->priv->paths. + * old_path also will eventually need to be freed, but it + * may still be used by e.g. ci->pathnames. So, store it + * in another string-list for now. + */ + string_list_append(&opt->priv->paths_to_free, old_path); + } assert(ci->filemask == 2 || ci->filemask == 4); assert(ci->dirmask == 0); @@ -2416,7 +2434,8 @@ static void apply_directory_rename_modifications(struct merge_options *opt, new_ci->stages[index].mode = ci->stages[index].mode; oidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid); - free(ci); + if (!opt->priv->pool) + free(ci); ci = new_ci; } @@ -3623,7 +3642,8 @@ static void process_entry(struct merge_options *opt, * the directory to remain here, so we need to move this * path to some new location. */ - CALLOC_ARRAY(new_ci, 1); + new_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci)); + /* We don't really want new_ci->merged.result copied, but it'll * be overwritten below so it doesn't matter. We also don't * want any directory mode/oid values copied, but we'll zero @@ -3715,7 +3735,7 @@ static void process_entry(struct merge_options *opt, const char *a_path = NULL, *b_path = NULL; int rename_a = 0, rename_b = 0; - new_ci = xmalloc(sizeof(*new_ci)); + new_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci)); if (S_ISREG(a_mode)) rename_a = 1; @@ -3788,12 +3808,14 @@ static void process_entry(struct merge_options *opt, strmap_remove(&opt->priv->paths, path, 0); /* * We removed path from opt->priv->paths. path - * will also eventually need to be freed, but - * it may still be used by e.g. ci->pathnames. - * So, store it in another string-list for now. + * will also eventually need to be freed if not + * part of a memory pool...but it may still be + * used by e.g. ci->pathnames. So, store it in + * another string-list for now in that case. */ - string_list_append(&opt->priv->paths_to_free, - path); + if (!opt->priv->pool) + string_list_append(&opt->priv->paths_to_free, + path); } /* @@ -4335,6 +4357,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) { struct rename_info *renames; int i; + struct mem_pool *pool = NULL; /* Sanity checks on opt */ trace2_region_enter("merge", "sanity checks", opt->repo); @@ -4406,9 +4429,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) #else opt->priv->pool = NULL; #endif + pool = opt->priv->pool; for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->dirs_removed[i], - NOT_RELEVANT, NULL, 0); + NOT_RELEVANT, pool, 0); strmap_init_with_options(&renames->dir_rename_count[i], NULL, 1); strmap_init_with_options(&renames->dir_renames[i], @@ -4422,7 +4446,7 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) */ strintmap_init_with_options(&renames->relevant_sources[i], -1 /* explicitly invalid */, - NULL, 0); + pool, 0); strmap_init_with_options(&renames->cached_pairs[i], NULL, 1); strset_init_with_options(&renames->cached_irrelevant[i], @@ -4432,9 +4456,9 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) } for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->deferred[i].possible_trivial_merges, - 0, NULL, 0); + 0, pool, 0); strset_init_with_options(&renames->deferred[i].target_dirs, - NULL, 1); + pool, 1); renames->deferred[i].trivial_merges_okay = 1; /* 1 == maybe */ } @@ -4447,9 +4471,10 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) * In contrast, conflicted just has a subset of keys from paths, so * we don't want to free those (it'd be a duplicate free). */ - strmap_init_with_options(&opt->priv->paths, NULL, 0); - strmap_init_with_options(&opt->priv->conflicted, NULL, 0); - string_list_init_nodup(&opt->priv->paths_to_free); + strmap_init_with_options(&opt->priv->paths, pool, 0); + strmap_init_with_options(&opt->priv->conflicted, pool, 0); + if (!opt->priv->pool) + string_list_init_nodup(&opt->priv->paths_to_free); /* * keys & strbufs in output will sometimes need to outlive "paths", From patchwork Sat Jul 31 17:27:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B280AC432BE for ; Sat, 31 Jul 2021 17:27:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 96ACF61050 for ; Sat, 31 Jul 2021 17:27:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231404AbhGaR16 (ORCPT ); Sat, 31 Jul 2021 13:27:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230479AbhGaR1x (ORCPT ); Sat, 31 Jul 2021 13:27:53 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83984C0613D5 for ; Sat, 31 Jul 2021 10:27:46 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id h13so2238564wrp.1 for ; Sat, 31 Jul 2021 10:27:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mxs78hw82mLcuiyukDBRKAlZGx7o8hUECUOhrCJvi0E=; b=QSmsZVabMUiI72ttU9COIwDKxURZGYEpkcIzSaE0GhQsWSMvPAkmSmUYTRQMu7q4x1 YIEmetzxvyj1OGRIUGx1c+Oa9kOd39qUvDohABQwbV5TI1hdcyoGATdkaMOOccNxVLtj s4wOoDcHCgCiMJMdQU9qGXl5dmHTu62uVK2b6/x4Mpm1E9lKeXiiqNxxGfa5ZNDlbCTf wsqqCVyK+vDI4eOz4mjfVEGsD3Xd9ultGj25BESZAocK9yBsozY3gn8BBaOLDQ0uzHTr 0EklhvK0NL/w4OAC2dEIDVwl6EdNXot0L+hHr6+OTyTPFdDyZ0ykCXwZLsfwDSp6ipLr G/eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mxs78hw82mLcuiyukDBRKAlZGx7o8hUECUOhrCJvi0E=; b=sqqwvrqS5YPFv1dFCXAnSJUXZ5yfTtm8XXB2ZIbA+D3QEaxs4AULpQCk72faup1tOP HJshJDJZ2NIUt82dOVW9WEHjLVPMO/teknR0LWUvV629VL49OfZmVGgVKcOWWoJ0eoKH FmqyQ6rFYlswOwY3j31R2YhsTHppTbH9ORMJXXFzjp56xMFih2TcEFYZfR/FxigOMFK4 Lj4BUD56gnvKZGKHTQd3vOoyk+bx9qAW2guz9u2CBvAfuhGxz0GzNe45v/JDHZyuH3e2 BSFdDrBtMJwsmnXoEnpNkPSwBXxQAa6iPg7mtF7Z1G1r8FUOTnNk2ORdPYC/Jdz0V8eS yltA== X-Gm-Message-State: AOAM533oAyXZEcivlsJmaCaPg1/CIiuJttyfbCHRFtBiVDKzn0g2VJXe I0dn4dhlPZV4UK7UsrtflayMzwxyed4= X-Google-Smtp-Source: ABdhPJw/wYDkrnDZqrQ9REzNSYMQkmw00guEUJoNpIet2x49/fpAvfrLSGbuT1QNmO75p5Lau5MYkg== X-Received: by 2002:adf:f1c6:: with SMTP id z6mr9193613wro.207.1627752465185; Sat, 31 Jul 2021 10:27:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t8sm5974864wmj.5.2021.07.31.10.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:44 -0700 (PDT) Message-Id: <08cf2498f96759a94543d61b9a05e02280ab19d2.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:35 +0000 Subject: [PATCH v4 6/9] diffcore-rename, merge-ort: add wrapper functions for filepair alloc/dealloc Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren We want to be able to allocate filespecs and filepairs using a mem_pool. However, filespec data will still remain outside the pool (perhaps in the future we could plumb the pool through the various diff APIs to allocate the filespec data too, but for now we are limiting the scope). Add some extra functions to allocate these appropriately based on the non-NULL-ness of opt->priv->pool, as well as some extra functions to handle correctly deallocating the relevant parts of them. A future commit will make use of these new functions. Signed-off-by: Elijah Newren --- diffcore-rename.c | 41 +++++++++++++++++++++++++++++++++++++++++ diffcore.h | 2 ++ merge-ort.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+) diff --git a/diffcore-rename.c b/diffcore-rename.c index 73d884099eb..5bc559f79e9 100644 --- a/diffcore-rename.c +++ b/diffcore-rename.c @@ -1328,6 +1328,47 @@ static void handle_early_known_dir_renames(struct dir_rename_info *info, rename_src_nr = new_num_src; } +static void free_filespec_data(struct diff_filespec *spec) +{ + if (!--spec->count) + diff_free_filespec_data(spec); +} + +MAYBE_UNUSED +static void pool_free_filespec(struct mem_pool *pool, + struct diff_filespec *spec) +{ + if (!pool) { + free_filespec(spec); + return; + } + + /* + * Similar to free_filespec(), but only frees the data. The spec + * itself was allocated in the pool and should not be individually + * freed. + */ + free_filespec_data(spec); +} + +MAYBE_UNUSED +void pool_diff_free_filepair(struct mem_pool *pool, + struct diff_filepair *p) +{ + if (!pool) { + diff_free_filepair(p); + return; + } + + /* + * Similar to diff_free_filepair() but only frees the data from the + * filespecs; not the filespecs or the filepair which were + * allocated from the pool. + */ + free_filespec_data(p->one); + free_filespec_data(p->two); +} + void diffcore_rename_extended(struct diff_options *options, struct strintmap *relevant_sources, struct strintmap *dirs_removed, diff --git a/diffcore.h b/diffcore.h index 533b30e21e7..b58ee6b1934 100644 --- a/diffcore.h +++ b/diffcore.h @@ -127,6 +127,8 @@ struct diff_filepair { #define DIFF_PAIR_MODE_CHANGED(p) ((p)->one->mode != (p)->two->mode) void diff_free_filepair(struct diff_filepair *); +void pool_diff_free_filepair(struct mem_pool *pool, + struct diff_filepair *p); int diff_unmodified_pair(struct diff_filepair *); diff --git a/merge-ort.c b/merge-ort.c index 99c75690855..e79830f9181 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -690,6 +690,48 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } +MAYBE_UNUSED +static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, + const char *path) +{ + struct diff_filespec *spec; + size_t len; + + if (!pool) + return alloc_filespec(path); + + /* Same code as alloc_filespec, except allocate from pool */ + len = strlen(path); + + spec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1)); + memcpy(spec+1, path, len); + spec->path = (void*)(spec+1); + + spec->count = 1; + spec->is_binary = -1; + return spec; +} + +MAYBE_UNUSED +static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, + struct diff_queue_struct *queue, + struct diff_filespec *one, + struct diff_filespec *two) +{ + struct diff_filepair *dp; + + if (!pool) + return diff_queue(queue, one, two); + + /* Same code as diff_queue, except allocate from pool */ + dp = mem_pool_calloc(pool, 1, sizeof(*dp)); + dp->one = one; + dp->two = two; + if (queue) + diff_q(queue, dp); + return dp; +} + static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) { if (!pool) From patchwork Sat Jul 31 17:27:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0136BC432BE for ; Sat, 31 Jul 2021 17:27:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E2D9161042 for ; Sat, 31 Jul 2021 17:27:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231347AbhGaR2B (ORCPT ); Sat, 31 Jul 2021 13:28:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231215AbhGaR1z (ORCPT ); Sat, 31 Jul 2021 13:27:55 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CF44C061798 for ; Sat, 31 Jul 2021 10:27:48 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id h13so2238583wrp.1 for ; Sat, 31 Jul 2021 10:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=XNtC/p8BeIl4g0JeAYc6QF8lRXLJ0YJUip1x0g2U0sA=; b=q+haH9aOwiVWhSlZAjQ5fU0tkEdTSxjbBdaZz3zcpR3D02BFN7Z8dXT3iYrwGBV8Ff CH5J1q92Lr+93Hyz6snLSgzhavnXT6pN6E9prGcpUaqtPqHaYHLLNQ8Co0nlR16iJWGf nJeDq/xXYzK6RKqV9FyWldKb48a+zhIgDr4LUosYHxUdPVHD5sL9EgfKDrC9T5lRPE72 sRfUW9v5t7kgVwo1deMyjSnc2y5NdsWD7lcUaWKd/gVcHLaffJCwzSaxDai76DmwpGuA cT4Lsfc9Ef3Vc8iGZp+XSXyAdG7aIZrEhKHFc1TGysUtkn6VmDUXM2/ohEPVs4i+4Jvs Ff7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=XNtC/p8BeIl4g0JeAYc6QF8lRXLJ0YJUip1x0g2U0sA=; b=Eris57wSFfrT/oDmBMfOI/IOn5ruSEj1CPgOU0OdkO144fq2Jq3w0DQJuGKolccwlf fTYEVScCFefNn2mbg+mgViZgTDNOzJFsPWXgzWzsGFozjKJG5tXzsbBXrKd/QhVi4Jg1 1ty3tiOVJYxnvmMlM6JAfvNhD0/eA97zMkNrHR4eKwsZhxa8sqrKnTggioa0RSWbj5Rj qj5MFruf8yul9f2cq62Rl1hYMm0MYcgoOmt0u3QchBkMOicJ2Woi8oFXqB4H1ZylNwgo 4PY7pkE482QRyr8cP1NtbVtjbfAWQqnPkOShC+sVGo1OXUqYpqLkfRiysYs3I1elVWuW pjBg== X-Gm-Message-State: AOAM530Et5/iRs1EVhhzvMvIQbKekWHZ7S17hLiVHMiPqGoLL/h/h7mC uB6ofXJB4PwjTRyEGqYlIuhLgoajTkU= X-Google-Smtp-Source: ABdhPJxSwViigFYfPzxmoyMQPeNTlTmA8y2r16+IOOX6Y3GMuPO+SN666UctJKJwM69nkfvf8v100w== X-Received: by 2002:a5d:4e87:: with SMTP id e7mr8760325wru.188.1627752465881; Sat, 31 Jul 2021 10:27:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e5sm6664198wrr.36.2021.07.31.10.27.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:45 -0700 (PDT) Message-Id: <4ffa5af8b570772bc4ccc859ce05c04c289646b0.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:36 +0000 Subject: [PATCH v4 7/9] merge-ort: store filepairs and filespecs in our mem_pool MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 198.1 ms ± 2.6 ms 198.5 ms ± 3.4 ms mega-renames: 715.8 ms ± 4.0 ms 679.1 ms ± 5.6 ms just-one-mega: 276.8 ms ± 4.2 ms 271.9 ms ± 2.8 ms Signed-off-by: Elijah Newren --- diffcore-rename.c | 9 ++++----- diffcore.h | 1 + merge-ort.c | 26 ++++++++++++++------------ 3 files changed, 19 insertions(+), 17 deletions(-) diff --git a/diffcore-rename.c b/diffcore-rename.c index 5bc559f79e9..7e6b3e1b143 100644 --- a/diffcore-rename.c +++ b/diffcore-rename.c @@ -1334,7 +1334,6 @@ static void free_filespec_data(struct diff_filespec *spec) diff_free_filespec_data(spec); } -MAYBE_UNUSED static void pool_free_filespec(struct mem_pool *pool, struct diff_filespec *spec) { @@ -1351,7 +1350,6 @@ static void pool_free_filespec(struct mem_pool *pool, free_filespec_data(spec); } -MAYBE_UNUSED void pool_diff_free_filepair(struct mem_pool *pool, struct diff_filepair *p) { @@ -1370,6 +1368,7 @@ void pool_diff_free_filepair(struct mem_pool *pool, } void diffcore_rename_extended(struct diff_options *options, + struct mem_pool *pool, struct strintmap *relevant_sources, struct strintmap *dirs_removed, struct strmap *dir_rename_count, @@ -1683,7 +1682,7 @@ void diffcore_rename_extended(struct diff_options *options, pair_to_free = p; if (pair_to_free) - diff_free_filepair(pair_to_free); + pool_diff_free_filepair(pool, pair_to_free); } diff_debug_queue("done copying original", &outq); @@ -1693,7 +1692,7 @@ void diffcore_rename_extended(struct diff_options *options, for (i = 0; i < rename_dst_nr; i++) if (rename_dst[i].filespec_to_free) - free_filespec(rename_dst[i].filespec_to_free); + pool_free_filespec(pool, rename_dst[i].filespec_to_free); cleanup_dir_rename_info(&info, dirs_removed, dir_rename_count != NULL); FREE_AND_NULL(rename_dst); @@ -1710,5 +1709,5 @@ void diffcore_rename_extended(struct diff_options *options, void diffcore_rename(struct diff_options *options) { - diffcore_rename_extended(options, NULL, NULL, NULL, NULL); + diffcore_rename_extended(options, NULL, NULL, NULL, NULL, NULL); } diff --git a/diffcore.h b/diffcore.h index b58ee6b1934..badc2261c20 100644 --- a/diffcore.h +++ b/diffcore.h @@ -181,6 +181,7 @@ void partial_clear_dir_rename_count(struct strmap *dir_rename_count); void diffcore_break(struct repository *, int); void diffcore_rename(struct diff_options *); void diffcore_rename_extended(struct diff_options *options, + struct mem_pool *pool, struct strintmap *relevant_sources, struct strintmap *dirs_removed, struct strmap *dir_rename_count, diff --git a/merge-ort.c b/merge-ort.c index e79830f9181..f4f0a3d57f0 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -690,7 +690,6 @@ static void path_msg(struct merge_options *opt, strbuf_addch(sb, '\n'); } -MAYBE_UNUSED static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, const char *path) { @@ -712,7 +711,6 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, return spec; } -MAYBE_UNUSED static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, struct diff_queue_struct *queue, struct diff_filespec *one, @@ -930,6 +928,7 @@ static void add_pair(struct merge_options *opt, unsigned dir_rename_mask) { struct diff_filespec *one, *two; + struct mem_pool *pool = opt->priv->pool; struct rename_info *renames = &opt->priv->renames; int names_idx = is_add ? side : 0; @@ -980,11 +979,11 @@ static void add_pair(struct merge_options *opt, return; } - one = alloc_filespec(pathname); - two = alloc_filespec(pathname); + one = pool_alloc_filespec(pool, pathname); + two = pool_alloc_filespec(pool, pathname); fill_filespec(is_add ? two : one, &names[names_idx].oid, 1, names[names_idx].mode); - diff_queue(&renames->pairs[side], one, two); + pool_diff_queue(pool, &renames->pairs[side], one, two); } static void collect_rename_info(struct merge_options *opt, @@ -2893,6 +2892,7 @@ static void use_cached_pairs(struct merge_options *opt, { struct hashmap_iter iter; struct strmap_entry *entry; + struct mem_pool *pool = opt->priv->pool; /* * Add to side_pairs all entries from renames->cached_pairs[side_index]. @@ -2906,9 +2906,9 @@ static void use_cached_pairs(struct merge_options *opt, new_name = old_name; /* We don't care about oid/mode, only filenames and status */ - one = alloc_filespec(old_name); - two = alloc_filespec(new_name); - diff_queue(pairs, one, two); + one = pool_alloc_filespec(pool, old_name); + two = pool_alloc_filespec(pool, new_name); + pool_diff_queue(pool, pairs, one, two); pairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D'; } } @@ -3016,6 +3016,7 @@ static int detect_regular_renames(struct merge_options *opt, diff_queued_diff = renames->pairs[side_index]; trace2_region_enter("diff", "diffcore_rename", opt->repo); diffcore_rename_extended(&diff_opts, + opt->priv->pool, &renames->relevant_sources[side_index], &renames->dirs_removed[side_index], &renames->dir_rename_count[side_index], @@ -3066,7 +3067,7 @@ static int collect_renames(struct merge_options *opt, if (p->status != 'A' && p->status != 'R') { possibly_cache_new_pair(renames, p, side_index, NULL); - diff_free_filepair(p); + pool_diff_free_filepair(opt->priv->pool, p); continue; } @@ -3079,7 +3080,7 @@ static int collect_renames(struct merge_options *opt, possibly_cache_new_pair(renames, p, side_index, new_path); if (p->status != 'R' && !new_path) { - diff_free_filepair(p); + pool_diff_free_filepair(opt->priv->pool, p); continue; } @@ -3197,7 +3198,7 @@ cleanup: side_pairs = &renames->pairs[s]; for (i = 0; i < side_pairs->nr; ++i) { struct diff_filepair *p = side_pairs->queue[i]; - diff_free_filepair(p); + pool_diff_free_filepair(opt->priv->pool, p); } } @@ -3210,7 +3211,8 @@ simple_cleanup: if (combined.nr) { int i; for (i = 0; i < combined.nr; i++) - diff_free_filepair(combined.queue[i]); + pool_diff_free_filepair(opt->priv->pool, + combined.queue[i]); free(combined.queue); } From patchwork Sat Jul 31 17:27:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89B8CC4338F for ; Sat, 31 Jul 2021 17:27:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6F2AA61042 for ; Sat, 31 Jul 2021 17:27:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231435AbhGaR2A (ORCPT ); Sat, 31 Jul 2021 13:28:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230515AbhGaR1y (ORCPT ); Sat, 31 Jul 2021 13:27:54 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C6ABC061799 for ; Sat, 31 Jul 2021 10:27:48 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id m20-20020a05600c4f54b029024e75a15716so8424048wmq.2 for ; Sat, 31 Jul 2021 10:27:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=I+3F03G+KYmxjI0L6JdxmvBn1HFd+nJBLZJ0YJyORGI=; b=X+D9uocKCZk7WZZdwmRg/10hUJpsx6nBMP6ULqjEr86oc+D1Bq0Ep+iQGMFwcMJzkO +wHwmjk6shLW7PoBvG4G3p3z8eK11sOSzwmtWZYWTtFMlqldyWN80ozI9w3Q2RlvlugS 0mhAfY7S+K+L7nwg1h8r1g/NtEQhpQT2y6fQRGqSJL6KmHQ/LZftm1b72tlcAYsbTMzl CNTwFgZ2J73tQatjaoHvKaoQe1LWav1/ShR6Q2QvdTTCjtl/t9zoa0UaghWV81e62LfI ZDd5GGGZak2txBxXN+IGCvCZYVT5lKdc3WCzXT9La2nRtCdnCxMw+Daqgx6zACmLjkY/ gRcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=I+3F03G+KYmxjI0L6JdxmvBn1HFd+nJBLZJ0YJyORGI=; b=qea0dzDnwO68gA6CJ4m+fJswStR3DQwhC9g3bs058egWEigO+cPAvlyHE1WnamzmRH kh8ZNS25CS4OMm11t6UwZab/AawtIqFb4TkFRyaX5+LKtbWvteCbW0h52esBMjDah1kT EzrVoKRd6ufSXAJk+eiUX8e2k94E2irrQtrhZYHWu99ZjBKNwreNB0hHADXxKdtGFxHh FBid4aBiu2HW37unF31FmfcGCzw3RY0pEGa5aUhIXkjjsItdPhpOciL7euKHaNoflVQG RRl+d+m7GCIgMTIlq5gV40SWmDtYIRhJ6PLBl3MsCoOWfw/oRUv9jzbjg7Kx0bWu05XP pGug== X-Gm-Message-State: AOAM531jml8R8NzRtjMuIv4rj00rlz8v5ap5JXRiNVQ5e8X+lJVHzqB9 V7LQm0YBjhvmLthT0c5HJEODThvuuoA= X-Google-Smtp-Source: ABdhPJwjuMyCkiIpblv0hWEKCsmb10UPpsnS+8uwv0QvMweTD2uG3gLLO0k/bRKI7eL7t0pEDDWHPg== X-Received: by 2002:a1c:7907:: with SMTP id l7mr8988313wme.87.1627752466872; Sat, 31 Jul 2021 10:27:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z12sm5114009wml.18.2021.07.31.10.27.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:46 -0700 (PDT) Message-Id: <1556f0443c35a95fbdbc1cb1841f81f9c4615f4c.1627752458.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:37 +0000 Subject: [PATCH v4 8/9] merge-ort: reuse path strings in pool_alloc_filespec MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren pool_alloc_filespec() was written so that the code when pool != NULL mimicked the code from alloc_filespec(), which including allocating enough extra space for the path and then copying it. However, the path passed to pool_alloc_filespec() is always going to already be in the same memory pool, so we may as well reuse it instead of copying it. For the testcases mentioned in commit 557ac0350d ("merge-ort: begin performance work; instrument with trace2_region_* calls", 2020-10-28), this change improves the performance as follows: Before After no-renames: 198.5 ms ± 3.4 ms 198.3 ms ± 2.9 ms mega-renames: 679.1 ms ± 5.6 ms 661.8 ms ± 5.9 ms just-one-mega: 271.9 ms ± 2.8 ms 264.6 ms ± 2.5 ms Signed-off-by: Elijah Newren --- merge-ort.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index f4f0a3d57f0..86ab8f60121 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -694,17 +694,13 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, const char *path) { struct diff_filespec *spec; - size_t len; if (!pool) return alloc_filespec(path); - /* Same code as alloc_filespec, except allocate from pool */ - len = strlen(path); - - spec = mem_pool_calloc(pool, 1, st_add3(sizeof(*spec), len, 1)); - memcpy(spec+1, path, len); - spec->path = (void*)(spec+1); + /* Similar to alloc_filespec, but allocate from pool and reuse path */ + spec = mem_pool_calloc(pool, 1, sizeof(*spec)); + spec->path = (char*)path; /* spec won't modify it */ spec->count = 1; spec->is_binary = -1; @@ -2904,6 +2900,25 @@ static void use_cached_pairs(struct merge_options *opt, const char *new_name = entry->value; if (!new_name) new_name = old_name; + if (pool) { + /* + * cached_pairs has _copies* of old_name and new_name, + * because it has to persist across merges. When + * pool != NULL + * pool_alloc_filespec() will just re-use the existing + * filenames, which will also get re-used by + * opt->priv->paths if they become renames, and then + * get freed at the end of the merge, leaving the copy + * in cached_pairs dangling. Avoid this by making a + * copy here. + * + * When pool == NULL, pool_alloc_filespec() calls + * alloc_filespec(), which makes a copy; we don't want + * to add another. + */ + old_name = mem_pool_strdup(pool, old_name); + new_name = mem_pool_strdup(pool, new_name); + } /* We don't care about oid/mode, only filenames and status */ one = pool_alloc_filespec(pool, old_name); From patchwork Sat Jul 31 17:27:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12412551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA26C4338F for ; Sat, 31 Jul 2021 17:28:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 364FA60FE7 for ; Sat, 31 Jul 2021 17:28:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229840AbhGaR2F (ORCPT ); Sat, 31 Jul 2021 13:28:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231253AbhGaR1z (ORCPT ); Sat, 31 Jul 2021 13:27:55 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AC4DC0613CF for ; Sat, 31 Jul 2021 10:27:49 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id h13so2238658wrp.1 for ; Sat, 31 Jul 2021 10:27:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+mAd0JUlMqph96PBZ0W+GdwMnFsO1xwtoZY87818VeI=; b=G6sVwsxqQlD6EfGUncJc5/T/kzHa/zGlBAs3qZjjGnhHDtHYqzFB9WJhkud7cKhibS mJJDfzZGpZCYw98gR5WKCq7JbcL9UiYblK2OtefYSvmoWXDdi8umydIZFcPLJY8cNYds Dazcd1lt7jvOxM5TFeFa1Tw32czUHj7qjYovWuLgqrpHPP8rUJALoTrGWjhdRxTWns6u EnOXRDbacyPCOjLuPQAemUvt1DGO+dpgcA2gj+tGSazidcl0ephFMzIMzRO5c7ngERRT OhyhlwAraZ/0U3XDgIGmKQgtgtnQxzNGdbugnpPU27QNODEbMSzBX+RCWN95gPlB+s5a uh0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+mAd0JUlMqph96PBZ0W+GdwMnFsO1xwtoZY87818VeI=; b=h+4dEPhKmu7+5jcMKGRAf2W28Wb2RqEF3fcOR7GKZMWmCuuYeUwcimmkfsl0++RaR0 M4Iqxaii4Xg+cdJps9u2jWQB5z1dQn2SDEjHPJC/fulTCFPMBFT8W+s49bSnyynpRkcy 5f2nR9HclzF/ZX884IB0G0K37X09qikaIGTWYVaK/2jaHTUyafd1pXhJdDr///lDl3FF xk2WS+gbGhUQ0FRabW57vrnlr7kn/LghKoDVXGXn4E1+1X2BE0sQPHrfwHXS8JC0ZZea iHvhtapCXSdfCd24no6xDN8HrLXgs5sVNIYkZb0I2B0y8ZiBbP6prYbXD6r46YrR5i63 saUw== X-Gm-Message-State: AOAM532CWAQ6KTSl6DJz7RvM6e/VzMA+Iq6uFbxnAPZ7x/GYALcT++me V7g6pso9CJw6ZKw9i9RDLhSHPmeov6Y= X-Google-Smtp-Source: ABdhPJxS+CDrwVvKZ7t5oWHvpL4MtLaLflzbAYmkPKsfYrHiFS0MCYP2aKKgGz6YDlSnlHKGjccbSw== X-Received: by 2002:adf:ef0d:: with SMTP id e13mr9302987wro.390.1627752467666; Sat, 31 Jul 2021 10:27:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o34sm5211974wms.10.2021.07.31.10.27.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jul 2021 10:27:47 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 31 Jul 2021 17:27:38 +0000 Subject: [PATCH v4 9/9] merge-ort: remove compile-time ability to turn off usage of memory pools Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff King , Eric Sunshine , Elijah Newren , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Simplify code maintenance by removing the ability to toggle between usage of memory pools and direct allocations. This allows us to also remove paths_to_free since it was solely about bookkeeping to make sure we freed the necessary paths, and allows us to remove some auxiliary functions. Suggested-by: Jeff King Signed-off-by: Elijah Newren --- merge-ort.c | 209 ++++++++++++---------------------------------------- 1 file changed, 47 insertions(+), 162 deletions(-) diff --git a/merge-ort.c b/merge-ort.c index 86ab8f60121..88ade50f4ed 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -37,8 +37,6 @@ #include "unpack-trees.h" #include "xdiff-interface.h" -#define USE_MEMORY_POOL 1 /* faster, but obscures memory leak hunting */ - /* * We have many arrays of size 3. Whenever we have such an array, the * indices refer to one of the sides of the three-way merge. This is so @@ -305,8 +303,6 @@ struct merge_options_internal { * * these keys serve to intern all the path strings, which allows * us to do pointer comparison on directory names instead of * strcmp; we just have to be careful to use the interned strings. - * (Technically paths_to_free may track some strings that were - * removed from froms paths.) * * The values of paths: * * either a pointer to a merged_info, or a conflict_info struct @@ -349,18 +345,7 @@ struct merge_options_internal { * freed together too. Using a memory pool for these provides a * nice speedup. */ - struct mem_pool internal_pool; - struct mem_pool *pool; /* NULL, or pointer to internal_pool */ - - /* - * paths_to_free: additional list of strings to free - * - * If keys are removed from "paths", they are added to paths_to_free - * to ensure they are later freed. We avoid free'ing immediately since - * other places (e.g. conflict_info.pathnames[]) may still be - * referencing these paths. - */ - struct string_list paths_to_free; + struct mem_pool pool; /* * output: special messages and conflict notices for various paths @@ -539,19 +524,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, void (*strset_clear_func)(struct strset *) = reinitialize ? strset_partial_clear : strset_clear; - if (opti->pool) - strmap_clear_func(&opti->paths, 0); - else { - /* - * We marked opti->paths with strdup_strings = 0, so that - * we wouldn't have to make another copy of the fullpath - * created by make_traverse_path from setup_path_info(). - * But, now that we've used it and have no other references - * to these strings, it is time to deallocate them. - */ - free_strmap_strings(&opti->paths); - strmap_clear_func(&opti->paths, 1); - } + strmap_clear_func(&opti->paths, 0); /* * All keys and values in opti->conflicted are a subset of those in @@ -560,20 +533,6 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, */ strmap_clear_func(&opti->conflicted, 0); - if (!opti->pool) { - /* - * opti->paths_to_free is similar to opti->paths; we - * created it with strdup_strings = 0 to avoid making - * _another_ copy of the fullpath but now that we've used - * it and have no other references to these strings, it is - * time to deallocate them. We do so by temporarily - * setting strdup_strings to 1. - */ - opti->paths_to_free.strdup_strings = 1; - string_list_clear(&opti->paths_to_free, 0); - opti->paths_to_free.strdup_strings = 0; - } - if (opti->attr_index.cache_nr) /* true iff opt->renormalize */ discard_index(&opti->attr_index); @@ -623,11 +582,7 @@ static void clear_or_reinit_internal_opts(struct merge_options_internal *opti, strmap_clear(&opti->output, 0); } -#if USE_MEMORY_POOL - mem_pool_discard(&opti->internal_pool, 0); - if (!reinitialize) - opti->pool = NULL; -#endif + mem_pool_discard(&opti->pool, 0); /* Clean out callback_data as well. */ FREE_AND_NULL(renames->callback_data); @@ -693,12 +648,9 @@ static void path_msg(struct merge_options *opt, static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, const char *path) { + /* Similar to alloc_filespec(), but allocate from pool and reuse path */ struct diff_filespec *spec; - if (!pool) - return alloc_filespec(path); - - /* Similar to alloc_filespec, but allocate from pool and reuse path */ spec = mem_pool_calloc(pool, 1, sizeof(*spec)); spec->path = (char*)path; /* spec won't modify it */ @@ -712,12 +664,9 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, struct diff_filespec *one, struct diff_filespec *two) { + /* Same code as diff_queue(), except allocate from pool */ struct diff_filepair *dp; - if (!pool) - return diff_queue(queue, one, two); - - /* Same code as diff_queue, except allocate from pool */ dp = mem_pool_calloc(pool, 1, sizeof(*dp)); dp->one = one; dp->two = two; @@ -726,27 +675,6 @@ static struct diff_filepair *pool_diff_queue(struct mem_pool *pool, return dp; } -static void *pool_calloc(struct mem_pool *pool, size_t count, size_t size) -{ - if (!pool) - return xcalloc(count, size); - return mem_pool_calloc(pool, count, size); -} - -static void *pool_alloc(struct mem_pool *pool, size_t size) -{ - if (!pool) - return xmalloc(size); - return mem_pool_alloc(pool, size); -} - -static void *pool_strndup(struct mem_pool *pool, const char *str, size_t len) -{ - if (!pool) - return xstrndup(str, len); - return mem_pool_strndup(pool, str, len); -} - /* add a string to a strbuf, but converting "/" to "_" */ static void add_flattened_path(struct strbuf *out, const char *s) { @@ -875,9 +803,9 @@ static void setup_path_info(struct merge_options *opt, assert(!df_conflict || !resolved); /* df_conflict implies !resolved */ assert(resolved == (merged_version != NULL)); - mi = pool_calloc(opt->priv->pool, 1, - resolved ? sizeof(struct merged_info) : - sizeof(struct conflict_info)); + mi = mem_pool_calloc(&opt->priv->pool, 1, + resolved ? sizeof(struct merged_info) : + sizeof(struct conflict_info)); mi->directory_name = current_dir_name; mi->basename_offset = current_dir_name_len; mi->clean = !!resolved; @@ -924,7 +852,6 @@ static void add_pair(struct merge_options *opt, unsigned dir_rename_mask) { struct diff_filespec *one, *two; - struct mem_pool *pool = opt->priv->pool; struct rename_info *renames = &opt->priv->renames; int names_idx = is_add ? side : 0; @@ -975,11 +902,11 @@ static void add_pair(struct merge_options *opt, return; } - one = pool_alloc_filespec(pool, pathname); - two = pool_alloc_filespec(pool, pathname); + one = pool_alloc_filespec(&opt->priv->pool, pathname); + two = pool_alloc_filespec(&opt->priv->pool, pathname); fill_filespec(is_add ? two : one, &names[names_idx].oid, 1, names[names_idx].mode); - pool_diff_queue(pool, &renames->pairs[side], one, two); + pool_diff_queue(&opt->priv->pool, &renames->pairs[side], one, two); } static void collect_rename_info(struct merge_options *opt, @@ -1170,7 +1097,7 @@ static int collect_merge_info_callback(int n, len = traverse_path_len(info, p->pathlen); /* +1 in both of the following lines to include the NUL byte */ - fullpath = pool_alloc(opt->priv->pool, len + 1); + fullpath = mem_pool_alloc(&opt->priv->pool, len + 1); make_traverse_path(fullpath, len + 1, info, p->path, p->pathlen); /* @@ -1425,7 +1352,7 @@ static int handle_deferred_entries(struct merge_options *opt, copy = renames->deferred[side].possible_trivial_merges; strintmap_init_with_options(&renames->deferred[side].possible_trivial_merges, 0, - opt->priv->pool, + &opt->priv->pool, 0); strintmap_for_each_entry(©, &iter, entry) { const char *path = entry->key; @@ -2377,21 +2304,17 @@ static void apply_directory_rename_modifications(struct merge_options *opt, VERIFY_CI(ci); /* Find parent directories missing from opt->priv->paths */ - if (opt->priv->pool) { - cur_path = mem_pool_strdup(opt->priv->pool, new_path); - free((char*)new_path); - new_path = (char *)cur_path; - } else { - cur_path = new_path; - } + cur_path = mem_pool_strdup(&opt->priv->pool, new_path); + free((char*)new_path); + new_path = (char *)cur_path; while (1) { /* Find the parent directory of cur_path */ char *last_slash = strrchr(cur_path, '/'); if (last_slash) { - parent_name = pool_strndup(opt->priv->pool, - cur_path, - last_slash - cur_path); + parent_name = mem_pool_strndup(&opt->priv->pool, + cur_path, + last_slash - cur_path); } else { parent_name = opt->priv->toplevel_dir; break; @@ -2400,8 +2323,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt, /* Look it up in opt->priv->paths */ entry = strmap_get_entry(&opt->priv->paths, parent_name); if (entry) { - if (!opt->priv->pool) - free((char*)parent_name); parent_name = entry->key; /* reuse known pointer */ break; } @@ -2428,16 +2349,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt, parent_name = cur_dir; } - if (!opt->priv->pool) { - /* - * We are removing old_path from opt->priv->paths. - * old_path also will eventually need to be freed, but it - * may still be used by e.g. ci->pathnames. So, store it - * in another string-list for now. - */ - string_list_append(&opt->priv->paths_to_free, old_path); - } - assert(ci->filemask == 2 || ci->filemask == 4); assert(ci->dirmask == 0); strmap_remove(&opt->priv->paths, old_path, 0); @@ -2471,8 +2382,6 @@ static void apply_directory_rename_modifications(struct merge_options *opt, new_ci->stages[index].mode = ci->stages[index].mode; oidcpy(&new_ci->stages[index].oid, &ci->stages[index].oid); - if (!opt->priv->pool) - free(ci); ci = new_ci; } @@ -2888,7 +2797,6 @@ static void use_cached_pairs(struct merge_options *opt, { struct hashmap_iter iter; struct strmap_entry *entry; - struct mem_pool *pool = opt->priv->pool; /* * Add to side_pairs all entries from renames->cached_pairs[side_index]. @@ -2900,30 +2808,24 @@ static void use_cached_pairs(struct merge_options *opt, const char *new_name = entry->value; if (!new_name) new_name = old_name; - if (pool) { - /* - * cached_pairs has _copies* of old_name and new_name, - * because it has to persist across merges. When - * pool != NULL - * pool_alloc_filespec() will just re-use the existing - * filenames, which will also get re-used by - * opt->priv->paths if they become renames, and then - * get freed at the end of the merge, leaving the copy - * in cached_pairs dangling. Avoid this by making a - * copy here. - * - * When pool == NULL, pool_alloc_filespec() calls - * alloc_filespec(), which makes a copy; we don't want - * to add another. - */ - old_name = mem_pool_strdup(pool, old_name); - new_name = mem_pool_strdup(pool, new_name); - } + + /* + * cached_pairs has *copies* of old_name and new_name, + * because it has to persist across merges. Since + * pool_alloc_filespec() will just re-use the existing + * filenames, which will also get re-used by + * opt->priv->paths if they become renames, and then + * get freed at the end of the merge, that would leave + * the copy in cached_pairs dangling. Avoid this by + * making a copy here. + */ + old_name = mem_pool_strdup(&opt->priv->pool, old_name); + new_name = mem_pool_strdup(&opt->priv->pool, new_name); /* We don't care about oid/mode, only filenames and status */ - one = pool_alloc_filespec(pool, old_name); - two = pool_alloc_filespec(pool, new_name); - pool_diff_queue(pool, pairs, one, two); + one = pool_alloc_filespec(&opt->priv->pool, old_name); + two = pool_alloc_filespec(&opt->priv->pool, new_name); + pool_diff_queue(&opt->priv->pool, pairs, one, two); pairs->queue[pairs->nr-1]->status = entry->value ? 'R' : 'D'; } } @@ -3031,7 +2933,7 @@ static int detect_regular_renames(struct merge_options *opt, diff_queued_diff = renames->pairs[side_index]; trace2_region_enter("diff", "diffcore_rename", opt->repo); diffcore_rename_extended(&diff_opts, - opt->priv->pool, + &opt->priv->pool, &renames->relevant_sources[side_index], &renames->dirs_removed[side_index], &renames->dir_rename_count[side_index], @@ -3082,7 +2984,7 @@ static int collect_renames(struct merge_options *opt, if (p->status != 'A' && p->status != 'R') { possibly_cache_new_pair(renames, p, side_index, NULL); - pool_diff_free_filepair(opt->priv->pool, p); + pool_diff_free_filepair(&opt->priv->pool, p); continue; } @@ -3095,7 +2997,7 @@ static int collect_renames(struct merge_options *opt, possibly_cache_new_pair(renames, p, side_index, new_path); if (p->status != 'R' && !new_path) { - pool_diff_free_filepair(opt->priv->pool, p); + pool_diff_free_filepair(&opt->priv->pool, p); continue; } @@ -3213,7 +3115,7 @@ cleanup: side_pairs = &renames->pairs[s]; for (i = 0; i < side_pairs->nr; ++i) { struct diff_filepair *p = side_pairs->queue[i]; - pool_diff_free_filepair(opt->priv->pool, p); + pool_diff_free_filepair(&opt->priv->pool, p); } } @@ -3226,7 +3128,7 @@ simple_cleanup: if (combined.nr) { int i; for (i = 0; i < combined.nr; i++) - pool_diff_free_filepair(opt->priv->pool, + pool_diff_free_filepair(&opt->priv->pool, combined.queue[i]); free(combined.queue); } @@ -3701,7 +3603,7 @@ static void process_entry(struct merge_options *opt, * the directory to remain here, so we need to move this * path to some new location. */ - new_ci = pool_calloc(opt->priv->pool, 1, sizeof(*new_ci)); + new_ci = mem_pool_calloc(&opt->priv->pool, 1, sizeof(*new_ci)); /* We don't really want new_ci->merged.result copied, but it'll * be overwritten below so it doesn't matter. We also don't @@ -3794,7 +3696,8 @@ static void process_entry(struct merge_options *opt, const char *a_path = NULL, *b_path = NULL; int rename_a = 0, rename_b = 0; - new_ci = pool_alloc(opt->priv->pool, sizeof(*new_ci)); + new_ci = mem_pool_alloc(&opt->priv->pool, + sizeof(*new_ci)); if (S_ISREG(a_mode)) rename_a = 1; @@ -3863,19 +3766,8 @@ static void process_entry(struct merge_options *opt, b_path = path; strmap_put(&opt->priv->paths, b_path, new_ci); - if (rename_a && rename_b) { + if (rename_a && rename_b) strmap_remove(&opt->priv->paths, path, 0); - /* - * We removed path from opt->priv->paths. path - * will also eventually need to be freed if not - * part of a memory pool...but it may still be - * used by e.g. ci->pathnames. So, store it in - * another string-list for now in that case. - */ - if (!opt->priv->pool) - string_list_append(&opt->priv->paths_to_free, - path); - } /* * Do special handling for b_path since process_entry() @@ -4482,13 +4374,8 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) /* Initialization of various renames fields */ renames = &opt->priv->renames; -#if USE_MEMORY_POOL - mem_pool_init(&opt->priv->internal_pool, 0); - opt->priv->pool = &opt->priv->internal_pool; -#else - opt->priv->pool = NULL; -#endif - pool = opt->priv->pool; + mem_pool_init(&opt->priv->pool, 0); + pool = &opt->priv->pool; for (i = MERGE_SIDE1; i <= MERGE_SIDE2; i++) { strintmap_init_with_options(&renames->dirs_removed[i], NOT_RELEVANT, pool, 0); @@ -4525,15 +4412,13 @@ static void merge_start(struct merge_options *opt, struct merge_result *result) * Although we initialize opt->priv->paths with strdup_strings=0, * that's just to avoid making yet another copy of an allocated * string. Putting the entry into paths means we are taking - * ownership, so we will later free it. paths_to_free is similar. + * ownership, so we will later free it. * * In contrast, conflicted just has a subset of keys from paths, so * we don't want to free those (it'd be a duplicate free). */ strmap_init_with_options(&opt->priv->paths, pool, 0); strmap_init_with_options(&opt->priv->conflicted, pool, 0); - if (!opt->priv->pool) - string_list_init_nodup(&opt->priv->paths_to_free); /* * keys & strbufs in output will sometimes need to outlive "paths",