From patchwork Tue Oct 16 21:02:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 10644185 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC071181D for ; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF3F52A9C6 for ; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C39632A9C4; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68EC62A9C3 for ; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727086AbeJQEzE (ORCPT ); Wed, 17 Oct 2018 00:55:04 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:35968 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725960AbeJQEzE (ORCPT ); Wed, 17 Oct 2018 00:55:04 -0400 Received: by mail-pf1-f196.google.com with SMTP id l81-v6so12054011pfg.3 for ; Tue, 16 Oct 2018 14:02:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+0hudgD5U21xaWUm5jU/uM+up9mp8pBozRZkrLKlBd8=; b=Nr73Cbm8HH/ZIzsesULAKhGKBGFVymHiX+/ETx5s6INZaQCnz6A2SjoRiiJRCGJVr6 v7LdKtPWZmyUD7NALYcJuuYncyL/xo3h8aDyjW76s2sjWXU2lj6GAjXNN5vujtC789/m skkoUKVFqbbz5HGj5j+Pqcl78vSLAmZwanGuw+fJx96o2UcBp0lCRq9tVeHYRJOs4O/P eKffZ0z0/xSZX3snCMwP2iUCPE9XvWGcQZtME5CCU1o6VrtTBF44SvyOB8JZyXHc9UDQ 6vmi6AE1nQ+iuY5QdN4PJdH3VoHzfGkIAxFqoqzHGNc/9+xZOFG4V0cEAXMQsly3z/lz xUIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+0hudgD5U21xaWUm5jU/uM+up9mp8pBozRZkrLKlBd8=; b=QcE+mDljtFReZOzIm8eB19uYTiXYsoTVyO3qjTblPIyvYAxd4QrqGt/Ez5B5tsfmJ6 jYuinkD9C/ig6SUyrIwmM2uehRCn14Sa+Q30i0lTGMyCrNYDPhY+I4dY1VVvofJ8eodl KYahw/q1h7zZK2ijb8QEQUOcopp7Z1z7iOmuns0jjM5ri6haBlq28WmFGMV+Q1KFMd09 ryZH+hNbGP4pMm1zBesqXc7Y1x8sjlyZTcMYkFEYAJ+Avt/zOOQk4bN0p+3Qx1/OlXw6 8vALM1RRmVhNii/4xb7gsuSI0uU6JTz6r4f24wR+hl6361EW700zlhd/Dk9f1tI6odRv 7ToQ== X-Gm-Message-State: ABuFfoiWI8NZMXPfPcedgOzA1mPcYgdWmQgLqNLD/QnmP/lffqogXgQx sB4TNMaNzUBK1J0+7ZdceLo2IyMH X-Google-Smtp-Source: ACcGV63bSPXN1TXFq+UwnqMJ2eCfOVq5cvTWJUAJhmvop/T6oIPTDj+9VVC9WnmZDDOwhQqHRqsljQ== X-Received: by 2002:a63:c54a:: with SMTP id g10-v6mr21277789pgd.201.1539723770224; Tue, 16 Oct 2018 14:02:50 -0700 (PDT) Received: from [127.0.0.1] ([40.112.137.127]) by smtp.gmail.com with ESMTPSA id h32-v6sm6877429pgb.94.2018.10.16.14.02.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 14:02:49 -0700 (PDT) Date: Tue, 16 Oct 2018 14:02:49 -0700 (PDT) X-Google-Original-Date: Tue, 16 Oct 2018 21:02:43 GMT Message-Id: <4b752704bd1dd676823d735757e8ed5e48eeba07.1539723766.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH 1/3] Fix typo 'detla' -> 'delta' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin Signed-off-by: Johannes Schindelin --- pack-objects.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pack-objects.h b/pack-objects.h index 2ca39cfcfe..86ee93feb4 100644 --- a/pack-objects.h +++ b/pack-objects.h @@ -377,7 +377,7 @@ static inline unsigned long oe_delta_size(struct packing_data *pack, return e->delta_size_; /* - * pack->detla_size[] can't be NULL because oe_set_delta_size() + * pack->delta_size[] can't be NULL because oe_set_delta_size() * must have been called when a new delta is saved with * oe_set_delta(). * If oe_delta() returns NULL (i.e. default state, which means From patchwork Tue Oct 16 21:02:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 10644189 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21258109C for ; Tue, 16 Oct 2018 21:03:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0DC842A9BF for ; Tue, 16 Oct 2018 21:03:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0251F2A9C3; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8AEA82A9C1 for ; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727097AbeJQEzG (ORCPT ); Wed, 17 Oct 2018 00:55:06 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:32988 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725960AbeJQEzF (ORCPT ); Wed, 17 Oct 2018 00:55:05 -0400 Received: by mail-pg1-f193.google.com with SMTP id y18-v6so11441392pge.0 for ; Tue, 16 Oct 2018 14:02:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eyWBzAhEsfNbITJS8ySsJ06Hy3HEuv8Cn05uRslF1dk=; b=d/oHDxcR7Cm9+i9tmKvo1RTbBZBkASEfSnYLcqihEx1UDljI1EYWeHfehs6LqpNlOv 6cLIJSHqRQDkmY0Syoz3bqrPCl534E6MWiCtWJlazUSWFuelvNL35J8KS2VsA6Ew2Z+P 5brykwL03HbK8dGvylJuVRPNvPVpR4V3ndxwt4GUDUs0rblEDZ60t5B4cKcKgcgb/FWZ MNPKSlqS38/NZzZqMT18sIGbHD+RJZfQVi6Ys+9eJXiQo1rNmqQcg8kiEH37Np+0BCbP 9Opy9Jdws0ogh5sz2sjH2EOEUOz7GrZc7e3b4d0tWFMVyP8xh4LMGraMuC+Iy5buGjJi TVWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eyWBzAhEsfNbITJS8ySsJ06Hy3HEuv8Cn05uRslF1dk=; b=AY1izuEBSIZ1+cHFE6EPbKZVq98WZCQz0ohsuHLUu6d26daeYL3vLQsvPChfWK7DKy IIpoVI6nFWx06ujuEyqp6gHx19o747GjzDX/KPe+0eYPDTMUy2afD3M0AAh2cNB/Ns3/ hYCQKgc/cKW7U+qjv8zOJtoplTsHuDPNaWHXuqBLiPeRq4Eu8hj10h58xM9ot8puAjjg eEXTrfJLuVxXwWAtYNWMZR6zwdjXqEYHXQKDoat1WUkolYPfl/SwOp50zOBeFYB7ol9r zyZc24EFTSBxSF0I9kzsVzKkWn3cztrW+gLtWi1hD0owzVc77gosFlV8JW/JnPds2q6s AKRQ== X-Gm-Message-State: ABuFfojshHw1oE6uzX9gBe40w1Yjh5+ZVpDk2tGKCdc+8se6wLOSU5J3 RsqDNIgidCsvbFD2Zj1r+vbKGQzc X-Google-Smtp-Source: ACcGV61fiXmHgHIHHdSkrX9YhSgl0W2dNF//5rAlIZGlXkCRMMuV/Jq+bx7CvX+VLFhH8LTeHnM84Q== X-Received: by 2002:a62:de05:: with SMTP id h5-v6mr23819807pfg.258.1539723771690; Tue, 16 Oct 2018 14:02:51 -0700 (PDT) Received: from [127.0.0.1] ([40.112.137.127]) by smtp.gmail.com with ESMTPSA id q8-v6sm16996016pgv.65.2018.10.16.14.02.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 14:02:50 -0700 (PDT) Date: Tue, 16 Oct 2018 14:02:50 -0700 (PDT) X-Google-Original-Date: Tue, 16 Oct 2018 21:02:44 GMT Message-Id: In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH 2/3] pack-objects (mingw): demonstrate a segmentation fault with large deltas Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin There is a problem in the way 9ac3f0e5b3e4 (pack-objects: fix performance issues on packing large deltas, 2018-07-22) initializes that mutex in the `packing_data` struct. The problem manifests in a segmentation fault on Windows, when a mutex (AKA critical section) is accessed without being initialized. (With pthreads, you apparently do not really have to initialize them?) This was reported in https://github.com/git-for-windows/git/issues/1839. Signed-off-by: Johannes Schindelin --- t/t5321-pack-large-objects.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 t/t5321-pack-large-objects.sh diff --git a/t/t5321-pack-large-objects.sh b/t/t5321-pack-large-objects.sh new file mode 100755 index 0000000000..c36c66fbb4 --- /dev/null +++ b/t/t5321-pack-large-objects.sh @@ -0,0 +1,32 @@ +#!/bin/sh +# +# Copyright (c) 2018 Johannes Schindelin +# + +test_description='git pack-object with "large" deltas + +' +. ./test-lib.sh +. "$TEST_DIRECTORY"/lib-pack.sh + +# Two similar-ish objects that we have computed deltas between. +A=01d7713666f4de822776c7622c10f1b07de280dc +B=e68fe8129b546b101aee9510c5328e7f21ca1d18 + +test_expect_success 'setup' ' + clear_packs && + { + pack_header 2 && + pack_obj $A $B && + pack_obj $B + } >ab.pack && + pack_trailer ab.pack && + git index-pack --stdin X-Patchwork-Id: 10644191 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 451A83C13 for ; Tue, 16 Oct 2018 21:03:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 35D1F2A9BF for ; Tue, 16 Oct 2018 21:03:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A0712A9C1; Tue, 16 Oct 2018 21:03:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB9A02A9C5 for ; Tue, 16 Oct 2018 21:02:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727112AbeJQEzI (ORCPT ); Wed, 17 Oct 2018 00:55:08 -0400 Received: from mail-pl1-f193.google.com ([209.85.214.193]:36763 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725960AbeJQEzH (ORCPT ); Wed, 17 Oct 2018 00:55:07 -0400 Received: by mail-pl1-f193.google.com with SMTP id y11-v6so11585136plt.3 for ; Tue, 16 Oct 2018 14:02:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IgwtKBxE28CYX6zEdezl6YM+t3YaWTQTEqV7cWdAc6M=; b=JmQKJjgCCyQRbuJ8Q45rVUgIeGmc0IMGxtAPTp9iPuBTnzCZQ4awX3sRcj/5kgR2NZ UGyEAeeYvi9mJE9kQlW4sW1V4td2Zb8VM0qaGSO9bpuMLoqhLFUBNlWYqYy2HvCug31h 3p60unVyd+R9RFy8uyZpuvxUozm7pCfIjlLH/3hlG0n3jeLsA8sNEHGGUBEqErkOZL7L iOHn+ogodpKfQMpULGdPsNgzOLQbrarbV8rQtkUoqXwF23HIHnuuabJTIdHt2386u38e Eu+xKV6NdBvikCs1OY6irM92+RF1nW9QB7lp2J2+bl06eWDfzpvA++nU8xxHU1tYuVsl oTTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IgwtKBxE28CYX6zEdezl6YM+t3YaWTQTEqV7cWdAc6M=; b=Vsd4qivz/3t4r+fwvGH/Lcuwe9o1Acgbt/7mrRPZvKZjeuf4Q1fF+8k/2Ny/c5q5QE ZYqGZR+cyEMGUW4OG3rEY2FJwzsGuOWeApLovzgdKiQoq5L4uUVL0A33Qo+ABYJ4XmIA nKlSiF0ht6GZqp7fAb0RNwFO0MhR2QXI3sArl7KqHdnxdSJJAa3kWQNmDy23KhfG/9CD aG3qP55HHXq6BgaJrN+dZEQGH+mdCe/s57p5RA4XijBwUWBiq7vWao7ioutNnza+15m/ iwS6q/vTDrPUtoleDHpsRqYbiCbM/ayKGsbSq4aiRwG73f8pBaNkyA/9BPm5lWUWaJi4 9Ubg== X-Gm-Message-State: ABuFfogDDssZvExiSdHFNM23GD8t8QF79fSY353+luEshzDrqVbmxk3H JcFu0bJvZCZtR5zZyx9moOMNS6sD X-Google-Smtp-Source: ACcGV63Ha+Jc+B/wCuwxzx7Ah9KiWRmCIIoTu4FuxBeeMWCGEvYk/UrsEUqBiB/GUyXA02DFVxogkg== X-Received: by 2002:a17:902:66e6:: with SMTP id e93-v6mr23332998plk.128.1539723773296; Tue, 16 Oct 2018 14:02:53 -0700 (PDT) Received: from [127.0.0.1] ([40.112.137.127]) by smtp.gmail.com with ESMTPSA id k185-v6sm29145458pgd.2.2018.10.16.14.02.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 14:02:52 -0700 (PDT) Date: Tue, 16 Oct 2018 14:02:52 -0700 (PDT) X-Google-Original-Date: Tue, 16 Oct 2018 21:02:45 GMT Message-Id: <8979693922364ef2e4a84a29298b081e86fa9ffa.1539723766.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Johannes Schindelin via GitGitGadget" Subject: [PATCH 3/3] pack-objects (mingw): initialize `packing_data` mutex in the correct spot Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , Johannes Schindelin Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Schindelin In 9ac3f0e5b3e4 (pack-objects: fix performance issues on packing large deltas, 2018-07-22), a mutex was introduced that is used to guard the call to set the delta size. This commit even added code to initialize it, but at an incorrect spot: in `init_threaded_search()`, while the call to `oe_set_delta_size()` (and hence to `packing_data_lock()`) can happen in the call chain `check_object()` <- `get_object_details()` <- `prepare_pack()` <- `cmd_pack_objects()`, which is long before the `prepare_pack()` function calls `ll_find_deltas()` (which initializes the threaded search). Another tell-tale that the mutex was initialized in an incorrect spot is that the function to initialize it lives in builtin/, while the code that uses the mutex is defined in a libgit.a header file. Let's use a more appropriate function: `prepare_packing_data()`, which not only lives in libgit.a, but *has* to be called before the `packing_data` struct is used that contains that mutex. This fixes https://github.com/git-for-windows/git/issues/1839. Signed-off-by: Johannes Schindelin --- builtin/pack-objects.c | 1 - pack-objects.c | 3 +++ t/t5321-pack-large-objects.sh | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index e6316d294d..e752cf9c7a 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -2363,7 +2363,6 @@ static void init_threaded_search(void) pthread_mutex_init(&cache_mutex, NULL); pthread_mutex_init(&progress_mutex, NULL); pthread_cond_init(&progress_cond, NULL); - pthread_mutex_init(&to_pack.lock, NULL); old_try_to_free_routine = set_try_to_free_routine(try_to_free_from_threads); } diff --git a/pack-objects.c b/pack-objects.c index 7e624c30eb..b6cdbb0166 100644 --- a/pack-objects.c +++ b/pack-objects.c @@ -148,6 +148,9 @@ void prepare_packing_data(struct packing_data *pdata) 1U << OE_SIZE_BITS); pdata->oe_delta_size_limit = git_env_ulong("GIT_TEST_OE_DELTA_SIZE", 1UL << OE_DELTA_SIZE_BITS); +#ifndef NO_PTHREADS + pthread_mutex_init(&pdata->lock, NULL); +#endif } struct object_entry *packlist_alloc(struct packing_data *pdata, diff --git a/t/t5321-pack-large-objects.sh b/t/t5321-pack-large-objects.sh index c36c66fbb4..a75eab87d3 100755 --- a/t/t5321-pack-large-objects.sh +++ b/t/t5321-pack-large-objects.sh @@ -24,7 +24,7 @@ test_expect_success 'setup' ' git index-pack --stdin