From patchwork Wed Nov 2 22:01:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?SZEDER_G=C3=A1bor?= X-Patchwork-Id: 13029048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BCE2C4332F for ; Wed, 2 Nov 2022 22:01:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230211AbiKBWBw (ORCPT ); Wed, 2 Nov 2022 18:01:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230153AbiKBWBu (ORCPT ); Wed, 2 Nov 2022 18:01:50 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 414E59FDC for ; Wed, 2 Nov 2022 15:01:49 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id v7so44173wmn.0 for ; Wed, 02 Nov 2022 15:01:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mkXnShDT+bgE/eNK3xMrOsSJBFOcZA1BKzgE6P8twfY=; b=f/dvavFbiXoyBJ9IgE3HarwT1X2lWCWykVNWraQkMkp9Y/jI9UL00nKqbtN47nGSA3 jVt07VihXxWNGoTtVxo9U5gvgJn/Rj4cX9qGWWOowzddtlDadyTJa5vJ50V9r6WIYIB5 0h5tmbXGkCWlswz1UOlVtakFM6NU/oHLpO2Xz9oyAg1dti5rX2I6XU9vJqm6OzOOrbFW FqKoT10ypvwFnfIG7P+br7LRAs05fUEL2mk7L4OdiDJvJ2u+5NMzAehcUU7MACAXg9SX E9kF7Y9D4SuVK86WPVY5Mu2ccJvos63wQFJGRhL1pdTkcSu/UX3GY0LcyYa+WxMytsgC yGQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mkXnShDT+bgE/eNK3xMrOsSJBFOcZA1BKzgE6P8twfY=; b=ZiiW37on/nhfECHoz2hC7zS2qEhYikRwA/j4ac9EuSuiNjrhzJkhH7xcEgMpMJO/52 XQin7eFSU6dVyrVdLzQz0qqfVqnyQh7zK3z4wS2teoEoBeLq+N8IUXOhtvuZTA5hovkb OwI4Wf0ZofDalTwuQ4hSCD9Ca1kMtG5qtnKK6bREJoZrJGyLPrYoe0TV7G9lg9w5ae8n a+THrfi/XXd0EqhHrjyDulBkODhNBBKjFEYK8JN35AShJmkj/cw4YqbxjB7/xHPpNF5H laE4NQen8gSwhhxHUBB9F3X+8hkr1JLHuq+DMc2EEv634GjsPfuv+SyMdggXTLh335jz O3pQ== X-Gm-Message-State: ACrzQf1ByH7iBocVwACNcMGmvk3OMcGSpQ+pqDEV1HaZveMlXXKt5wdP O+4Gfn+N4cdyPqAt1AIvO2pjy+m1/zM= X-Google-Smtp-Source: AMsMyM7QPSWTwe9d7N/uTE0tMIIi5gEMTdkp902zvhxj9Wr+Wk8Pbl5yfOBmJ/S4jFgvZ523sCV2xw== X-Received: by 2002:a7b:cd99:0:b0:3cf:7556:a52c with SMTP id y25-20020a7bcd99000000b003cf7556a52cmr10807816wmj.53.1667426507729; Wed, 02 Nov 2022 15:01:47 -0700 (PDT) Received: from localhost (84-236-78-97.pool.digikabel.hu. [84.236.78.97]) by smtp.gmail.com with ESMTPSA id c4-20020a05600c0a4400b003cf894c05e4sm1613080wmq.22.2022.11.02.15.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 15:01:47 -0700 (PDT) From: =?utf-8?q?SZEDER_G=C3=A1bor?= To: git@vger.kernel.org Cc: Taylor Blau , man dog , =?utf-8?q?SZEDER_G=C3=A1bor?= Subject: [PATCH 1/3] line-log: free diff queue when processing non-merge commits Date: Wed, 2 Nov 2022 23:01:40 +0100 Message-Id: <20221102220142.574890-2-szeder.dev@gmail.com> X-Mailer: git-send-email 2.38.1.564.g99c012faba In-Reply-To: <20221102220142.574890-1-szeder.dev@gmail.com> References: <20221102220142.574890-1-szeder.dev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When processing a non-merge commit, the line-level log first asks the tree-diff machinery whether any of the files in the given line ranges were modified between the current commit and its parent, and if some of them were, then it loads the contents of those files from both commits to see whether their line ranges were modified and/or need to be adjusted. Alas, it doesn't free() the diff queue holding the results of that query and the contents of those files once its done. This can add up to a substantial amount of leaked memory, especially when the file in question is big and is frequently modified: a user reported "Out of memory, malloc failed" errors with a 2MB text file that was modified ~2800 times [1] (I estimate the leak would use up almost 11GB memory in that case). Free that diff queue to plug this memory leak. However, instead of simply open-coding the necessary three lines, add them as a helper function to the diff API, because it will be useful elsewhere as well. [1] https://public-inbox.org/git/CAFOPqVXz2XwzX8vGU7wLuqb2ZuwTuOFAzBLRM_QPk+NJa=eC-g@mail.gmail.com/ Signed-off-by: SZEDER Gábor --- diff.c | 7 +++++++ diffcore.h | 1 + line-log.c | 1 + 3 files changed, 9 insertions(+) diff --git a/diff.c b/diff.c index 35e46dd968..ef94175163 100644 --- a/diff.c +++ b/diff.c @@ -5773,6 +5773,13 @@ void diff_free_filepair(struct diff_filepair *p) free(p); } +void diff_free_queue(struct diff_queue_struct *q) +{ + for (int i = 0; i < q->nr; i++) + diff_free_filepair(q->queue[i]); + free(q->queue); +} + const char *diff_aligned_abbrev(const struct object_id *oid, int len) { int abblen; diff --git a/diffcore.h b/diffcore.h index badc2261c2..9b588a1ee1 100644 --- a/diffcore.h +++ b/diffcore.h @@ -162,6 +162,7 @@ struct diff_filepair *diff_queue(struct diff_queue_struct *, struct diff_filespec *, struct diff_filespec *); void diff_q(struct diff_queue_struct *, struct diff_filepair *); +void diff_free_queue(struct diff_queue_struct *q); /* dir_rename_relevance: the reason we want rename information for a dir */ enum dir_rename_relevance { diff --git a/line-log.c b/line-log.c index 51d93310a4..7a74daf2e8 100644 --- a/line-log.c +++ b/line-log.c @@ -1195,6 +1195,7 @@ static int process_ranges_ordinary_commit(struct rev_info *rev, struct commit *c if (parent) add_line_range(rev, parent, parent_range); free_line_log_data(parent_range); + diff_free_queue(&queue); return changed; } From patchwork Wed Nov 2 22:01:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?SZEDER_G=C3=A1bor?= X-Patchwork-Id: 13029049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E92FC4332F for ; Wed, 2 Nov 2022 22:01:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230254AbiKBWBz (ORCPT ); Wed, 2 Nov 2022 18:01:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230208AbiKBWBw (ORCPT ); Wed, 2 Nov 2022 18:01:52 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93CA69FED for ; Wed, 2 Nov 2022 15:01:50 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id l14so219003wrw.2 for ; Wed, 02 Nov 2022 15:01:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=meNx4qMBZCbA9MsohxwdX8UH6wW0Nf1nJgUCnZAl2fA=; b=HPHSHUAHy/Qg6BWAObPq3Pwa+kXNT/nzV3h7nII/owryfQQzjejFfugocusjfVsEmP BYjuihnTZTa0nwfBQALR9zopCUZHI+3maix94/8+BzrjJGAaVo8XchMKG5RjfM15R0Oq BITJ/b7wbPIyyV3c0nXr2PUB/q/aUbS6GwrIxZi5ipsDG0dEemPQcugzTEJrFfMA21Ei ysVmWRVsfpdeFoHkehykVijHdQP12faqq6W6X/S4K0nbmknxjCjb4jQrG9DX8+yx5EDH IN/1iKSpDAE4/9y/OXRlEnjhZUeR1n6NEB2fdwWMxkrOgg8MSzdWNq6s8b+PN1ELBtPA 8XGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=meNx4qMBZCbA9MsohxwdX8UH6wW0Nf1nJgUCnZAl2fA=; b=WiYZ7Yps/acorEyFTJH+khyoUpyG2qXB95hJN5sfyoS2uRvWGV1TAuZVTJHiXgl4Dl wpDnHXXWBX6BE6yvSQbLFVYLY9wdomnBh3TKy2WxC6hVpcqhL+k5RmicO/CxHKM4ptkp vzzXgY5FcdLi+emB5PFm1KfvrSqzW79X2nqfOepzrdbt0HKXxZW0e3fjPs8f5/MziIDE o3E32+JNdxsfv7SnaFiGLu9pg8tjcVRm8DejtDkqf7e6ZS3Y+SkgZG5IULiwRJFmjYRX JJ3otR5EnHn/ckFN9Rddk9wygRIbFEbwji7fFDS7xNNaQQGBl56raWviUdAFwoVuglcd oetw== X-Gm-Message-State: ACrzQf0vCaNcV2rjvsd2K2D80JQ5YnsSxMdKn8sRxNkrVw04YaDzH/Zu hz3hHaLIKOhYMMzwdZgfCCQy6MZ6Qz4= X-Google-Smtp-Source: AMsMyM5iV5mrMJDbG2ugHHOGMhmg+dKREhUWeJuQxok/nqpFVeOWJRWywwkH/DHDwpEa+ldEtcsiDw== X-Received: by 2002:adf:f5c4:0:b0:236:c419:710f with SMTP id k4-20020adff5c4000000b00236c419710fmr13079823wrp.56.1667426509142; Wed, 02 Nov 2022 15:01:49 -0700 (PDT) Received: from localhost (84-236-78-97.pool.digikabel.hu. [84.236.78.97]) by smtp.gmail.com with ESMTPSA id n42-20020a05600c502a00b003cf5ec79bf9sm3399291wmr.40.2022.11.02.15.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 15:01:48 -0700 (PDT) From: =?utf-8?q?SZEDER_G=C3=A1bor?= To: git@vger.kernel.org Cc: Taylor Blau , man dog , =?utf-8?q?SZEDER_G=C3=A1bor?= Subject: [PATCH 2/3] line-log: free the diff queues' arrays when processing merge commits Date: Wed, 2 Nov 2022 23:01:41 +0100 Message-Id: <20221102220142.574890-3-szeder.dev@gmail.com> X-Mailer: git-send-email 2.38.1.564.g99c012faba In-Reply-To: <20221102220142.574890-1-szeder.dev@gmail.com> References: <20221102220142.574890-1-szeder.dev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When processing merge commits, the line-level log first creates an array of diff queues, each comparing the merge commit with one of its parents, to check whether any of the files in the given line ranges were modified. Alas, when freeing these queues it only frees the filepairs in the queues, but not the queues' internal arrays holding pointers to those filepairs. Use the diff_free_queue() helper function introduced in the previous commit to free the diff queues' internal arrays as well. Signed-off-by: SZEDER Gábor --- line-log.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/line-log.c b/line-log.c index 7a74daf2e8..a7f3e7f6ce 100644 --- a/line-log.c +++ b/line-log.c @@ -1089,10 +1089,8 @@ static struct diff_filepair *diff_filepair_dup(struct diff_filepair *pair) static void free_diffqueues(int n, struct diff_queue_struct *dq) { - int i, j; - for (i = 0; i < n; i++) - for (j = 0; j < dq[i].nr; j++) - diff_free_filepair(dq[i].queue[j]); + for (int i = 0; i < n; i++) + diff_free_queue(&dq[i]); free(dq); } From patchwork Wed Nov 2 22:01:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?SZEDER_G=C3=A1bor?= X-Patchwork-Id: 13029050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41211C433FE for ; Wed, 2 Nov 2022 22:01:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230320AbiKBWB5 (ORCPT ); Wed, 2 Nov 2022 18:01:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230224AbiKBWBx (ORCPT ); Wed, 2 Nov 2022 18:01:53 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32AB0F02A for ; Wed, 2 Nov 2022 15:01:52 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id j15so211481wrq.3 for ; Wed, 02 Nov 2022 15:01:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BFyqAM3oQTFGg6melyvXp4NoiSYsmOxbkbVCMdc4IXA=; b=kQVfA88PM0sAHWugZMkR60K+faj1a128z12RpbfJrXP8NTGmKggl0D6T9qXljgO4hm yfp8xysdEWYXRJBCZuxgjxGr436HLcjz9ven9Xj2zh+vANSCKbfacNuMAUCmYGw75h1e CCJZ5SpU9iFzTy9iMGnXvat9F7mhuqnpKPqGK/HpO5xbarRPfvumMeLnCiA/UnKO8AWT v0et+39ZGSUuCmvSAaqE3J63TlqWEBKfQ65BgnS71e/7AjADpGUXl1bt3y46bsj6RyK6 71CeZjObWsm90h9tzf+zRR+hkfyng70cS0rKbjNwNxkr3RKpgEBZ+HJ098BSce3ePe7S rEng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BFyqAM3oQTFGg6melyvXp4NoiSYsmOxbkbVCMdc4IXA=; b=gI/029n+JY61yxm4VT9CmaP5yAONqgUMcf1eAC8wU8XFTgINdSZ1pb5mRi+JlVokMT dsw5REzPjbWSTuvkkuMc62AhpfoWfGkMNsjrHFqfxKeCQiYPtKI1be2gx6C2Ls5ODrnt a1PdJ00tJUiRpt70PGZmSZ+fJosxnLnACRZFdxCXADHTU3e7GHklEg9LH/XLGKoK04iU tomoohjwWPD3DTuY6Z8OCj5I+7x98LezhSdl3UShR/Gq536Esz62Nv9kotzzNSNDd4Yf AJAfg2zWjYX2T8y2cRcBhpXMrqQU4WQ4AoPUY1hg8A3AyJbzsQ5QUBuWlDlkFl2rmGqE 3fqQ== X-Gm-Message-State: ACrzQf1y/jc90iI1sNW9nCrClimYwJJ+wiWWrP1KPLO9uLv1SLPpQ18S IUFQdi5vM2pkEPckibI9jxlPZSp2Fow= X-Google-Smtp-Source: AMsMyM4KLTsPYL/orwcsG5JiazlLZqlP421I62x8tCZpdUSeiWbLAXzE8ver9B1zzGZCp2Rs32pbzA== X-Received: by 2002:adf:f14e:0:b0:236:d073:dad with SMTP id y14-20020adff14e000000b00236d0730dadmr11012080wro.15.1667426510683; Wed, 02 Nov 2022 15:01:50 -0700 (PDT) Received: from localhost (84-236-78-97.pool.digikabel.hu. [84.236.78.97]) by smtp.gmail.com with ESMTPSA id y3-20020a05600c17c300b003b4cba4ef71sm3517969wmo.41.2022.11.02.15.01.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 15:01:50 -0700 (PDT) From: =?utf-8?q?SZEDER_G=C3=A1bor?= To: git@vger.kernel.org Cc: Taylor Blau , man dog , =?utf-8?q?SZEDER_G=C3=A1bor?= Subject: [PATCH 3/3] diff.c: use diff_free_queue() Date: Wed, 2 Nov 2022 23:01:42 +0100 Message-Id: <20221102220142.574890-4-szeder.dev@gmail.com> X-Mailer: git-send-email 2.38.1.564.g99c012faba In-Reply-To: <20221102220142.574890-1-szeder.dev@gmail.com> References: <20221102220142.574890-1-szeder.dev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Use diff_free_queue() instead of open-coding it. This shortens the code and make it less repetitive. Note that the second hunk in diff_flush() is interesting, because the 'free_queue' label separates the loop freeing the queue's filepairs from free()-ing the queue's internal array. This is somewhat suspicious, but it was not an issue before: there is only one place from where we jump to this label with a goto, and that is protected by an 'if (!q->nr && ...)' condition, i.e. we only skipped the loop freeing the filepairs when there were no filepairs in the queue to begin with. Signed-off-by: SZEDER Gábor --- diff.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/diff.c b/diff.c index ef94175163..03e6ffb5e4 100644 --- a/diff.c +++ b/diff.c @@ -6337,13 +6337,9 @@ static int diff_get_patch_id(struct diff_options *options, struct object_id *oid int diff_flush_patch_id(struct diff_options *options, struct object_id *oid, int diff_header_only) { struct diff_queue_struct *q = &diff_queued_diff; - int i; int result = diff_get_patch_id(options, oid, diff_header_only); - for (i = 0; i < q->nr; i++) - diff_free_filepair(q->queue[i]); - - free(q->queue); + diff_free_queue(q); DIFF_QUEUE_CLEAR(q); return result; @@ -6612,10 +6608,8 @@ void diff_flush(struct diff_options *options) if (output_format & DIFF_FORMAT_CALLBACK) options->format_callback(q, options, options->format_callback_data); - for (i = 0; i < q->nr; i++) - diff_free_filepair(q->queue[i]); free_queue: - free(q->queue); + diff_free_queue(q); DIFF_QUEUE_CLEAR(q); diff_free(options);