From patchwork Sat Aug 10 20:27:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11088673 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01DE414D5 for ; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E5D4A22A65 for ; Sat, 10 Aug 2019 20:28:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9E9A26E74; Sat, 10 Aug 2019 20:28:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B7B1522ADC for ; Sat, 10 Aug 2019 20:28:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726373AbfHJU1y (ORCPT ); Sat, 10 Aug 2019 16:27:54 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:43979 "EHLO mail-qk1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725863AbfHJU1y (ORCPT ); Sat, 10 Aug 2019 16:27:54 -0400 Received: by mail-qk1-f193.google.com with SMTP id m2so10030962qkd.10 for ; Sat, 10 Aug 2019 13:27:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NLBOZqqXta6iLuZNW8agpwsuwtaAjq+P13p4+vwbQxQ=; b=S5hrTYSv0UEliwxcHkCpKfh2fgyf3ZTTUb+PQVSebPmUhKXPYiEMhfJlNgwQGciyYx uPdnQJoHt4K1yjlohpS//5YX+jewwXQU4uYhlFCiE/d04R5LNBWvgL0c1cZGZ1Wfz97P 1O8fQi1P1v5iNBQQX4bSjH82mvTgj4y0bQtJmIf0NSkMiYIEx9ItQcuJe9E2zaq38SQx 4xvhWxRMP5zz8ZZKGTyPystcnjxNWEk0uolOxxHDAZ3IioG1WKsGN71Fia9YIDVqzS0u UV0V0JQYqY4wJXMH178x/zdenhO3FNkMk+Li5b2radKthOvD0apbbD1igd5xPSXt+1dC cq7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NLBOZqqXta6iLuZNW8agpwsuwtaAjq+P13p4+vwbQxQ=; b=kzAHCtyHgrSvnekCBFLHcrgGt+lozdmleltYoYV7bf3i0355IOP/pbWwYrHFsTz8HF 0ErCa4zyN0pEHlzCfF5yd1xfCkl5eNtzdBkS9/EFYIWtnJ2/Z4Vapoq0ndGloHePI+Bc W9edpK6v4G19+8howJJHuhh5Xx/kmVylciVc37+f0pncMzNKZTC15/G9peSaRQ1VkGKY 9CNazkWmvrOGFwgOq1dLVXbVg9kPARRDci7kyWrlaDbzsUed/cLb+fpZdxfWxPrB2+44 blS6cSeV65KFluzvEPsKWGmmME2uo/XpOUhNA/lNdcwmDNLs8WnOHqx7vbnWcg7TPHJX itdA== X-Gm-Message-State: APjAAAXi/JriKPyAC9PN7IADHN82gF0mrKp3tb3/kXSvGeOPYEvKmIfn U1Jg4zKB75wNivlq3Gz1w71s1tW6TNE= X-Google-Smtp-Source: APXvYqzv4aYoRQDT0KIoYw00R/iwLpuqyPbc9XX49+F9P2z2jtHkrYcv8PSMFpSHH17AuS8ih/XS/A== X-Received: by 2002:a37:4e4c:: with SMTP id c73mr23791348qkb.309.1565468872837; Sat, 10 Aug 2019 13:27:52 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id u1sm54384295qth.21.2019.08.10.13.27.50 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 10 Aug 2019 13:27:52 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: Christian Couder , Olga Telezhnaya , =?utf-8?b?Tmd1eeG7hW4gVGjDoWkg?= =?utf-8?b?Tmfhu41jIER1eQ==?= , Stefan Beller , Denton Liu , Jeff King , Junio C Hamano Subject: [GSoC][PATCH 1/4] object-store: add lock to read_object_file_extended() Date: Sat, 10 Aug 2019 17:27:27 -0300 Message-Id: <052de4c139bf4962182e6cb8f4aa315aa6130124.1565468806.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow read_object_file_extended() to be called by multiple threads protecting it with a lock. The lock usage can be toggled with enable_obj_read_lock() and disable_obj_read_lock(). Probably there are many spots in read_object_file_extended()'s call chain that could be executed unlocked (and thus, in parallel). But, for now, we are only interested in allowing parallel access to zlib inflation. This is one of the sections where object reading spends most of the time and it's already thread-safe. So, to take advantage of that, the lock is released when entering it and re-acquired right after. We may refine the lock to also exploit other possible parallel spots in the future, but threaded zlib inflation should already give great speedups. Note that add_delta_base_cache() was also modified to skip adding already present entries to the cache. This wasn't possible before, but now it is since phase I and phase III of unpack_entry() may execute concurrently. Signed-off-by: Matheus Tavares --- object-store.h | 4 ++++ packfile.c | 7 ++++++ sha1-file.c | 61 +++++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 67 insertions(+), 5 deletions(-) diff --git a/object-store.h b/object-store.h index 7f7b3cdd80..cfc9484995 100644 --- a/object-store.h +++ b/object-store.h @@ -159,6 +159,10 @@ const char *loose_object_path(struct repository *r, struct strbuf *buf, void *map_loose_object(struct repository *r, const struct object_id *oid, unsigned long *size); +void enable_obj_read_lock(void); +void disable_obj_read_lock(void); +void obj_read_lock(void); +void obj_read_unlock(void); void *read_object_file_extended(struct repository *r, const struct object_id *oid, enum object_type *type, diff --git a/packfile.c b/packfile.c index fc43a6c52c..de93dc50e2 100644 --- a/packfile.c +++ b/packfile.c @@ -1115,7 +1115,9 @@ unsigned long get_size_from_delta(struct packed_git *p, do { in = use_pack(p, w_curs, curpos, &stream.avail_in); stream.next_in = in; + obj_read_unlock(); st = git_inflate(&stream, Z_FINISH); + obj_read_lock(); curpos += stream.next_in - in; } while ((st == Z_OK || st == Z_BUF_ERROR) && stream.total_out < sizeof(delta_head)); @@ -1468,6 +1470,9 @@ static void add_delta_base_cache(struct packed_git *p, off_t base_offset, struct delta_base_cache_entry *ent = xmalloc(sizeof(*ent)); struct list_head *lru, *tmp; + if (get_delta_base_cache_entry(p, base_offset)) + return; + delta_base_cached += base_size; list_for_each_safe(lru, tmp, &delta_base_cache_lru) { @@ -1597,7 +1602,9 @@ static void *unpack_compressed_entry(struct packed_git *p, do { in = use_pack(p, w_curs, curpos, &stream.avail_in); stream.next_in = in; + obj_read_unlock(); st = git_inflate(&stream, Z_FINISH); + obj_read_lock(); if (!stream.avail_out) break; /* the payload is larger than it should be */ curpos += stream.next_in - in; diff --git a/sha1-file.c b/sha1-file.c index 84fd02f107..f5ff51aedb 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -1560,16 +1560,54 @@ int pretend_object_file(void *buf, unsigned long len, enum object_type type, return 0; } +static pthread_mutex_t obj_read_mutex; +static int obj_read_use_lock = 0; + +/* + * Enabling the object read lock allows multiple threads to safely call the + * following functions in parallel: repo_read_object_file(), read_object_file() + * and read_object_file_extended(). + */ +void enable_obj_read_lock(void) +{ + if (obj_read_use_lock) + return; + + obj_read_use_lock = 1; + pthread_mutex_init(&obj_read_mutex, NULL); +} + +void disable_obj_read_lock(void) +{ + if (!obj_read_use_lock) + return; + + obj_read_use_lock = 0; + pthread_mutex_destroy(&obj_read_mutex); +} + +void obj_read_lock(void) +{ + if(obj_read_use_lock) + pthread_mutex_lock(&obj_read_mutex); +} + +void obj_read_unlock(void) +{ + if(obj_read_use_lock) + pthread_mutex_unlock(&obj_read_mutex); +} + /* * This function dies on corrupt objects; the callers who want to * deal with them should arrange to call read_object() and give error * messages themselves. */ -void *read_object_file_extended(struct repository *r, - const struct object_id *oid, - enum object_type *type, - unsigned long *size, - int lookup_replace) +static void *do_read_object_file_extended(struct repository *r, + const struct object_id *oid, + enum object_type *type, + unsigned long *size, + int lookup_replace) { void *data; const struct packed_git *p; @@ -1602,6 +1640,19 @@ void *read_object_file_extended(struct repository *r, return NULL; } +void *read_object_file_extended(struct repository *r, + const struct object_id *oid, + enum object_type *type, + unsigned long *size, + int lookup_replace) +{ + void *data; + obj_read_lock(); + data = do_read_object_file_extended(r, oid, type, size, lookup_replace); + obj_read_unlock(); + return data; +} + void *read_object_with_reference(struct repository *r, const struct object_id *oid, const char *required_type_name, From patchwork Sat Aug 10 20:27:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11088679 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89000746 for ; Sat, 10 Aug 2019 20:28:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79AE022A65 for ; Sat, 10 Aug 2019 20:28:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6DF1F22ADC; Sat, 10 Aug 2019 20:28:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D72C026E55 for ; Sat, 10 Aug 2019 20:28:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726406AbfHJU17 (ORCPT ); Sat, 10 Aug 2019 16:27:59 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:38026 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725863AbfHJU16 (ORCPT ); Sat, 10 Aug 2019 16:27:58 -0400 Received: by mail-qk1-f196.google.com with SMTP id u190so10664454qkh.5 for ; Sat, 10 Aug 2019 13:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tNmCBAGkWtu3W2MOdb6EhCFlaFqoCcepeubQxKBrXtM=; b=dcq92GDpKYYeQAsdz2PTt6ojSOIdRz3wqqIKi1ZPvNPzuGs9VGNrG7A5ezSjjXIclJ tFgfyxQhWEqx+zZ+dmZHPgIMoYa33BhRnKViYzumU76FG9LB07cyYlbVQa4XnurWp7Eu SnA6Pzv6aTBP3mxENC4n1qjk17EdbcOpvQOz1y3+u/GHc1ahiqqoL7MTu+LHjclfktBV xvYnlxOBKF1iAC6dzjH4OQPqMXTq14DI+e0EtxPHrDaZgXjT6W88NVIofTXg4/HC0qHl jj1ZZghx6CxtrZespVbkst5l3LOwefnyHhKvRbaW87/hApFVQRVY0Vl5Xc5cVD0Mvij1 dVOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tNmCBAGkWtu3W2MOdb6EhCFlaFqoCcepeubQxKBrXtM=; b=RGFiS4b3+GWmXD0RNMa806DIorYAXhAMgkw+q58G8W/zcpPegavZh+m5N5uz/cruVY YShQGWPbdcxVwzTI6MMqkIQvgHzA2/ZVJhKN62Qk1fwqLhYo39R/YNKJDZwpo6t01gd6 90kXjIEmwlas8ueXY24iel6I9+WHSX5sa8B6ah/Bl8N4yNtGPhUDQk7NEw5CcI8ABCPj 4NEL+sJwIQhp2ucU3oJrw1TG9X/FGevI3JrpYgu2jxtk7K9FoMZ6CJAdB5I07EAFcyjx zsu4ltqTAs72WO2HEO7E4U1q1rcqa7zxcUgR2ZBymVM3szmq1qZDCF8eHdpBH0K/nADb Civw== X-Gm-Message-State: APjAAAXTld8vNq8inPTla/sjsJY8zDlEKTt738cAxqdKYP2kz/Vchnt5 q+n0sEBQqhYtrHIssg5M3E6KVKYZpDk= X-Google-Smtp-Source: APXvYqwahuu5tEJoCG26jOOX1zd6iD3WLJHiruygWapB8FiGMiWiK4quzcVltp+zzk0AJS7atOENLw== X-Received: by 2002:a37:aa88:: with SMTP id t130mr24861590qke.12.1565468877814; Sat, 10 Aug 2019 13:27:57 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id u1sm54384295qth.21.2019.08.10.13.27.55 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 10 Aug 2019 13:27:57 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: Christian Couder , Olga Telezhnaya , =?utf-8?b?Tmd1eeG7hW4gVGjDoWkg?= =?utf-8?b?Tmfhu41jIER1eQ==?= , Junio C Hamano , Denton Liu , "brian m. carlson" Subject: [GSoC][PATCH 2/4] grep: allow locks to be enabled individually Date: Sat, 10 Aug 2019 17:27:28 -0300 Message-Id: <235de7de2874bd089b106be75121e1616308ed55.1565468806.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP git-grep has some internal locks to protect thread-unsafe operations when running with threads. The usage of these locks can be toggled through the variable 'grep_use_locks'. However, it's not currently possible to enable each lock individually. And since object reading has its own locks now, it is desirable to disable the respective grep lock (and only that) in cases where we can do so. To do that, transform 'grep_use_locks' from a binary variable to a bitmask, which controls each lock individually. The actual disabling of grep_read_lock, when possible, will be done in the following patch. Signed-off-by: Matheus Tavares --- builtin/grep.c | 2 +- grep.c | 4 ++-- grep.h | 8 ++++++-- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/builtin/grep.c b/builtin/grep.c index 560051784e..a871bad8ad 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -205,7 +205,7 @@ static void start_threads(struct grep_opt *opt) pthread_cond_init(&cond_add, NULL); pthread_cond_init(&cond_write, NULL); pthread_cond_init(&cond_result, NULL); - grep_use_locks = 1; + grep_use_locks = GREP_USE_ALL_LOCKS; for (i = 0; i < ARRAY_SIZE(todo); i++) { strbuf_init(&todo[i].out, 0); diff --git a/grep.c b/grep.c index cd952ef5d3..3aca0db435 100644 --- a/grep.c +++ b/grep.c @@ -1523,13 +1523,13 @@ pthread_mutex_t grep_attr_mutex; static inline void grep_attr_lock(void) { - if (grep_use_locks) + if (grep_use_locks & GREP_USE_ATTR_LOCK) pthread_mutex_lock(&grep_attr_mutex); } static inline void grep_attr_unlock(void) { - if (grep_use_locks) + if (grep_use_locks & GREP_USE_ATTR_LOCK) pthread_mutex_unlock(&grep_attr_mutex); } diff --git a/grep.h b/grep.h index 1875880f37..02bffacfa2 100644 --- a/grep.h +++ b/grep.h @@ -229,6 +229,10 @@ int grep_source(struct grep_opt *opt, struct grep_source *gs); struct grep_opt *grep_opt_dup(const struct grep_opt *opt); int grep_threads_ok(const struct grep_opt *opt); +#define GREP_USE_READ_LOCK (1 << 0) +#define GREP_USE_ATTR_LOCK (1 << 1) +#define GREP_USE_ALL_LOCKS (~0) + /* * Mutex used around access to the attributes machinery if * opt->use_threads. Must be initialized/destroyed by callers! @@ -239,13 +243,13 @@ extern pthread_mutex_t grep_read_mutex; static inline void grep_read_lock(void) { - if (grep_use_locks) + if (grep_use_locks & GREP_USE_READ_LOCK) pthread_mutex_lock(&grep_read_mutex); } static inline void grep_read_unlock(void) { - if (grep_use_locks) + if (grep_use_locks & GREP_USE_READ_LOCK) pthread_mutex_unlock(&grep_read_mutex); } From patchwork Sat Aug 10 20:27:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11088675 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 316D514F7 for ; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F99822A65 for ; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13A5D22ADC; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03A9B26E69 for ; Sat, 10 Aug 2019 20:28:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726464AbfHJU2D (ORCPT ); Sat, 10 Aug 2019 16:28:03 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:46725 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725863AbfHJU2D (ORCPT ); Sat, 10 Aug 2019 16:28:03 -0400 Received: by mail-qt1-f196.google.com with SMTP id j15so5646230qtl.13 for ; Sat, 10 Aug 2019 13:28:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WLQblxTsIs6isPvJ42iTZ/nKzLW0BcZi1MiNG6b1XWE=; b=R7Bqbbfs8CpQ91DAgqMKeAcrUb6d5smHz6OChrTfNsFahnSDQlK5zlEwNTr17VcLXC smpJ8RbmM7AjBzUJXDUF3S57l9tFwqwqSBb7qUE15k+A/CBqOdOLMcS1qUwpj+KPi76w lpjyzYV/3y3H8h1mE7ldMWeNmPSgUCqGQw6voJPjMugtePDd4O/wG/ZgyOi2oWb7Vutb t2aKhKM+ImI0YzRFvO2kWnoltAbFpWv75kqm5Ozp2tnUNw8tJhyl4TyEb7QakCGj2V7B EdMvFGevkkdDJjmg8irVLGmyvdTPQyJG0bWyA+i1qcCwRspV5gH3mX/L4/x+FfquffgN /49A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WLQblxTsIs6isPvJ42iTZ/nKzLW0BcZi1MiNG6b1XWE=; b=umS8ksQM3CO83ZWBR4GMJpF+atE6yC3oyqMGXxhzDEbdE/tF8QV4WzUpiLf6yrq2Hh VRqVstBmi612Qjnl/dh36HrQZr86NPFv1cs/Qb7VmufTb4qIoGeOLsE4HYYOkRpLjkwi BBww51SJfc6HyenZr+ymuOdcNFMtvhBfo2NJckMeGbTJ3CUUZoJm5RrHd5lkYFHj0gfC szmCYuJ6s2YKzHu2atIwUrqUsgRNpQ4BZKzEyUtl9BZFfT+Uz7X+UuScz+x/H7tZ4oV/ RLFYajJtG5Y0EmhimW8YAK5yiN4DJJFI48xVeGaoGH4PqeHQqU7J7jWJXRG7chJsH/eF XsKA== X-Gm-Message-State: APjAAAWJNvOs4EcU/jGxySWMoi2Efuac1BbANWfN0C1wLgXBxygAku5X TwAjvdyBFSdjLJigH5DLIqzH6Nkzw4k= X-Google-Smtp-Source: APXvYqx+mafa60YuCY+TaU9bG16CIy/VgKaGzpd7ZvLxlNpV3igPP114DTSu1tHyWevvXffATwXHug== X-Received: by 2002:a0c:ac98:: with SMTP id m24mr24489109qvc.9.1565468881643; Sat, 10 Aug 2019 13:28:01 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id u1sm54384295qth.21.2019.08.10.13.27.59 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 10 Aug 2019 13:28:01 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: Christian Couder , Olga Telezhnaya , =?utf-8?b?Tmd1eeG7hW4gVGjDoWkg?= =?utf-8?b?Tmfhu41jIER1eQ==?= , Brandon Williams , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= , Junio C Hamano Subject: [GSoC][PATCH 3/4] grep: disable grep_read_mutex when possible Date: Sat, 10 Aug 2019 17:27:29 -0300 Message-Id: X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP git-grep uses 'grep_read_mutex' to protect some object reading operations. But these have their own internal lock now, which ensure a better performance (with more parallel regions). So, disable the former when it's possible to use the latter, with enable_obj_read_lock(). Signed-off-by: Matheus Tavares --- builtin/grep.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/builtin/grep.c b/builtin/grep.c index a871bad8ad..fa51392222 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -205,7 +205,17 @@ static void start_threads(struct grep_opt *opt) pthread_cond_init(&cond_add, NULL); pthread_cond_init(&cond_write, NULL); pthread_cond_init(&cond_result, NULL); - grep_use_locks = GREP_USE_ALL_LOCKS; + if (recurse_submodules || opt->allow_textconv) { + /* + * textconv and submodules' operations are not thread-safe yet + * so we must use grep_read_lock when grepping multithreaded + * with these options. + */ + grep_use_locks = GREP_USE_ALL_LOCKS; + } else { + grep_use_locks = GREP_USE_ATTR_LOCK; + enable_obj_read_lock(); + } for (i = 0; i < ARRAY_SIZE(todo); i++) { strbuf_init(&todo[i].out, 0); @@ -227,7 +237,7 @@ static void start_threads(struct grep_opt *opt) } } -static int wait_all(void) +static int wait_all(struct grep_opt *opt) { int hit = 0; int i; @@ -263,6 +273,9 @@ static int wait_all(void) pthread_cond_destroy(&cond_write); pthread_cond_destroy(&cond_result); grep_use_locks = 0; + if (!recurse_submodules && !opt->allow_textconv) { + disable_obj_read_lock(); + } return hit; } @@ -1140,7 +1153,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix) } if (num_threads > 1) - hit |= wait_all(); + hit |= wait_all(&opt); if (hit && show_in_pager) run_pager(&opt, prefix); clear_pathspec(&pathspec); From patchwork Sat Aug 10 20:27:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 11088677 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFE771850 for ; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FF2B22A65 for ; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9448622ADC; Sat, 10 Aug 2019 20:28:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 237CE26E73 for ; Sat, 10 Aug 2019 20:28:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726465AbfHJU2I (ORCPT ); Sat, 10 Aug 2019 16:28:08 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:35409 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725863AbfHJU2I (ORCPT ); Sat, 10 Aug 2019 16:28:08 -0400 Received: by mail-qt1-f196.google.com with SMTP id d23so99286395qto.2 for ; Sat, 10 Aug 2019 13:28:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp-br.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r7BVam1vzjvKeDq7MRpzhR+6JLEl5IbPg6CHrc5oOZs=; b=qmmnQoD4UsiMxo94kZvsiwGKqxbt37dc8V+9D7azetLcm/IuhDWwhDjUx8GWI75u9f aWXv8Er7qTUOzWbmnsxo9pB2WvqjFL5uP+b6o/QkxdbFl2T0/jibtfZhsFR2H9lz/Ykd H7bmeW7pSt/+udRDEt0ADf4gQPWB0Nrdt0I7ysGeMzPEeO4iL3r5n8YF1pESTViY1c/i Cz2gRHsjjceqhXMSGucB2/IEs7F6AhLssdIP7Dj+jBIZl9/39jbt4DckyL1DS/BmTPyq TOCLQK5bu0GMwpVb3pPviYtF+ATKGbYTSXMRpGO8nSFXu1DIKHhSYf7gRjohFfxuzJzx GLYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r7BVam1vzjvKeDq7MRpzhR+6JLEl5IbPg6CHrc5oOZs=; b=cTjn2LEeaXc7RSUWa6dXofgEXKzaGPeOba1R8McitOlqQNXiP03TLMkhe4lNzd21dI M6FMG4PkrTPo5mW5FIwyrjgGQNJZGayT7azuh5G8uFTdup7cj3Xt0NnUU+7q37w0gJqT E0gUB71SgTeRpkfOBcbPYGDI5dpjsboxU3BlQa77FaxMBfckK/aRQf1X1+ufp15Ytepw p1AhbkCgj0VOIiuaSZ4BHW8/x54wA3fYMuBgjOohU/9a0EstXmU+wyLubj3/gRAbO9f+ I3WbFf9ZAfnUQrO3ISYtEZAotVvFwbCH7m0IiEQsW6VzYbHhDR8/Sw2YPuLcP+dZBB7J ytzg== X-Gm-Message-State: APjAAAW9eKfRu5owoGcBcI4LU4M0A8bMlxKXw5GtSBSAk7fPYjUz69j+ y2efK7Q7O+3vqLvLgfXyX/lK2Cp8ZwU= X-Google-Smtp-Source: APXvYqzeTiQLR+MpUY1W+QfQXQeqydEQ6tnXJ496Z92dRqzs0q6siDVgjiDsz+97xThQzsTkkcGQwg== X-Received: by 2002:ac8:60a:: with SMTP id d10mr23593025qth.31.1565468887210; Sat, 10 Aug 2019 13:28:07 -0700 (PDT) Received: from mango.spo.virtua.com.br ([2804:14c:81:942d::1]) by smtp.gmail.com with ESMTPSA id u1sm54384295qth.21.2019.08.10.13.28.04 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 10 Aug 2019 13:28:06 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: Christian Couder , Olga Telezhnaya , =?utf-8?b?Tmd1eeG7hW4gVGjDoWkg?= =?utf-8?b?Tmfhu41jIER1eQ==?= , Junio C Hamano , Jeff King , Brandon Williams , Manav Rathi Subject: [GSoC][PATCH 4/4] grep: re-enable threads in some non-worktree cases Date: Sat, 10 Aug 2019 17:27:30 -0300 Message-Id: <8c26abe9156e069ad4d19e9f0ce131cd1453f030.1565468806.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.22.0 In-Reply-To: References: MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP They were disabled at 53b8d93 ("grep: disable threading in non-worktree case", 12-12-2011), due to observable performance drops. But now that zlib inflation can be performed in parallel, for some of git-grep's options, we can regain the speedup. Grepping 'abcd[02]' ("Regex 1") and '(static|extern) (int|double) \*' ("Regex 2") at chromium's repository[1] I got: Threads | Regex 1 | Regex 2 ---------|------------|----------- 1 | 17.3557s | 20.8410s 2 | 9.7170s | 11.2415s 8 | 6.1723s | 6.9378s These are all means of 30 executions after 2 warmup runs. All tests were executed on an i7-7700HQ with 16GB of RAM and SSD. But to make sure the optimization also performs well on HDD, the tests were repeated on an AMD Turion 64 X2 TL-62 (dual-core) with 4GB of RAM and HDD (SATA-150, 5400 rpm): Threads | Regex 1 | Regex 2 ---------|------------|----------- 1 | 40.3347s | 47.6173s 2 | 27.6547s | 35.1797s Unfortunately, textconv and submodules' operations remain thread-unsafe, needing locks to be safely executed when threaded. Because of that, it's not currently worthy to grep in parallel with them. So, when --textconv or --recurse-submodules are given for a non-worktree case, threads are kept disabled. In order to clarify this behavior, let's also add a "NOTES" section to Documentation/git-grep.txt explaining the thread usage details. [1]: chromium’s repo at commit 03ae96f (“Add filters testing at DSF=2”, 04-06-2019), after a 'git gc' execution. Signed-off-by: Matheus Tavares --- Documentation/git-grep.txt | 12 ++++++++++++ builtin/grep.c | 3 ++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt index 2d27969057..9686875fbc 100644 --- a/Documentation/git-grep.txt +++ b/Documentation/git-grep.txt @@ -330,6 +330,18 @@ EXAMPLES `git grep solution -- :^Documentation`:: Looks for `solution`, excluding files in `Documentation`. +NOTES +----- + +The --threads option (and grep.threads configuration) will be ignored when +--open-files-in-pager is used, forcing a single-threaded execution. + +When grepping the index file (with --cached or giving tree objects), the +following options will also suppress thread creation: + + --recurse_submodules + --textconv + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/grep.c b/builtin/grep.c index fa51392222..e5a9da471a 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -1073,7 +1073,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix) pathspec.recursive = 1; pathspec.recurse_submodules = !!recurse_submodules; - if (list.nr || cached || show_in_pager) { + if (show_in_pager || + ((list.nr || cached) && (recurse_submodules || opt.allow_textconv))) { if (num_threads > 1) warning(_("invalid option combination, ignoring --threads")); num_threads = 1;