From patchwork Tue Jun 29 08:11:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Wong X-Patchwork-Id: 12349455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17DCDC11F66 for ; Tue, 29 Jun 2021 08:11:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E026161DC4 for ; Tue, 29 Jun 2021 08:11:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232405AbhF2INj (ORCPT ); Tue, 29 Jun 2021 04:13:39 -0400 Received: from dcvr.yhbt.net ([64.71.152.64]:47958 "EHLO dcvr.yhbt.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232349AbhF2INi (ORCPT ); Tue, 29 Jun 2021 04:13:38 -0400 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 2C89F1F8C7 for ; Tue, 29 Jun 2021 08:11:09 +0000 (UTC) From: Eric Wong To: git@vger.kernel.org Subject: [PATCH 1/4] use_pack: attempt to handle ENOMEM from mmap Date: Tue, 29 Jun 2021 08:11:05 +0000 Message-Id: <20210629081108.28657-2-e@80x24.org> In-Reply-To: <20210629081108.28657-1-e@80x24.org> References: <20210629081108.28657-1-e@80x24.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Since use_pack() can already safely munmap packs to respect core.packedGitLimit, attempt to gracefully handle ENOMEM errors the same way by unmapping a window and retrying. This benefits unprivileged users who lack permissions to raise the `sys.vm.max_map_count' sysctl and/or RLIMIT_DATA resource limit. I've also verified it is safe to release a pack here by unconditionally calling unuse_one_window() before xmmap_gently(): --- a/packfile.c +++ b/packfile.c @@ -649,6 +649,7 @@ unsigned char *use_pack(struct packed_git *p, && unuse_one_window(p)) ; /* nothing */ do { + unuse_one_window(p); win->base = xmmap_gently(NULL, win->len, PROT_READ, MAP_PRIVATE, p->pack_fd, win->offset); Signed-off-by: Eric Wong --- packfile.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/packfile.c b/packfile.c index 755aa7aec5..a0da790fb4 100644 --- a/packfile.c +++ b/packfile.c @@ -648,9 +648,12 @@ unsigned char *use_pack(struct packed_git *p, while (packed_git_limit < pack_mapped && unuse_one_window(p)) ; /* nothing */ - win->base = xmmap_gently(NULL, win->len, - PROT_READ, MAP_PRIVATE, - p->pack_fd, win->offset); + do { + win->base = xmmap_gently(NULL, win->len, + PROT_READ, MAP_PRIVATE, + p->pack_fd, win->offset); + } while (win->base == MAP_FAILED && errno == ENOMEM + && unuse_one_window(p)); if (win->base == MAP_FAILED) die_errno("packfile %s cannot be mapped", p->pack_name); From patchwork Tue Jun 29 08:11:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Wong X-Patchwork-Id: 12349453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A70BC11F67 for ; Tue, 29 Jun 2021 08:11:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ED82061DD6 for ; Tue, 29 Jun 2021 08:11:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232410AbhF2INn (ORCPT ); Tue, 29 Jun 2021 04:13:43 -0400 Received: from dcvr.yhbt.net ([64.71.152.64]:48022 "EHLO dcvr.yhbt.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232349AbhF2INk (ORCPT ); Tue, 29 Jun 2021 04:13:40 -0400 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 4CD081F8C8 for ; Tue, 29 Jun 2021 08:11:09 +0000 (UTC) From: Eric Wong To: git@vger.kernel.org Subject: [PATCH 2/4] map_loose_object_1: attempt to handle ENOMEM from mmap Date: Tue, 29 Jun 2021 08:11:06 +0000 Message-Id: <20210629081108.28657-3-e@80x24.org> In-Reply-To: <20210629081108.28657-1-e@80x24.org> References: <20210629081108.28657-1-e@80x24.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This benefits unprivileged users who lack permissions to raise the `sys.vm.max_map_count' sysctl and/or RLIMIT_DATA resource limit. Multi-threaded callers to map_loose_object_1 (e.g. "git grep") appear to guard against thread-safety problems. Other multi-core aware pieces of code (e.g. parallel-checkout) uses child processes Forcing a call to unuse_one_window() once before xmmap_gently() revealed no test suite failures: --- a/object-file.c +++ b/object-file.c @@ -1197,6 +1197,7 @@ static void *map_loose_object_1(struct repository *r, const char *path, return NULL; } do { + unuse_one_window(NULL); map = xmmap_gently(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0); } while (map == MAP_FAILED && errno == ENOMEM Signed-off-by: Eric Wong --- object-file.c | 8 +++++++- packfile.c | 2 +- packfile.h | 2 ++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/object-file.c b/object-file.c index f233b440b2..4c043f1f3c 100644 --- a/object-file.c +++ b/object-file.c @@ -1196,7 +1196,13 @@ static void *map_loose_object_1(struct repository *r, const char *path, close(fd); return NULL; } - map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0); + do { + map = xmmap_gently(NULL, *size, PROT_READ, + MAP_PRIVATE, fd, 0); + } while (map == MAP_FAILED && errno == ENOMEM + && unuse_one_window(NULL)); + if (map == MAP_FAILED) + die_errno("%s cannot be mapped", path); } close(fd); } diff --git a/packfile.c b/packfile.c index a0da790fb4..4bc7fa0f36 100644 --- a/packfile.c +++ b/packfile.c @@ -265,7 +265,7 @@ static void scan_windows(struct packed_git *p, } } -static int unuse_one_window(struct packed_git *current) +int unuse_one_window(struct packed_git *current) { struct packed_git *p, *lru_p = NULL; struct pack_window *lru_w = NULL, *lru_l = NULL; diff --git a/packfile.h b/packfile.h index 3ae117a8ae..da9a061e30 100644 --- a/packfile.h +++ b/packfile.h @@ -196,4 +196,6 @@ int is_promisor_object(const struct object_id *oid); int load_idx(const char *path, const unsigned int hashsz, void *idx_map, size_t idx_size, struct packed_git *p); +int unuse_one_window(struct packed_git *); + #endif From patchwork Tue Jun 29 08:11:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Wong X-Patchwork-Id: 12349457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CE79C11F68 for ; Tue, 29 Jun 2021 08:11:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3FC8961DF2 for ; Tue, 29 Jun 2021 08:11:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232414AbhF2INo (ORCPT ); Tue, 29 Jun 2021 04:13:44 -0400 Received: from dcvr.yhbt.net ([64.71.152.64]:48078 "EHLO dcvr.yhbt.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232408AbhF2INo (ORCPT ); Tue, 29 Jun 2021 04:13:44 -0400 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 5F3A91F8C9 for ; Tue, 29 Jun 2021 08:11:09 +0000 (UTC) From: Eric Wong To: git@vger.kernel.org Subject: [PATCH 3/4] check_packed_git_idx: attempt to handle ENOMEM from mmap Date: Tue, 29 Jun 2021 08:11:07 +0000 Message-Id: <20210629081108.28657-4-e@80x24.org> In-Reply-To: <20210629081108.28657-1-e@80x24.org> References: <20210629081108.28657-1-e@80x24.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This benefits unprivileged users who lack permissions to raise the `sys.vm.max_map_count' sysctl and/or RLIMIT_DATA resource limit. It appears none of our multithreaded code is calling this, or is doing so while holding appropropriate locks to serialize access to the object database. Unconditionally calling unuse_one_window() before xmmap_gently() revealed no test suite failures: --- a/packfile.c +++ b/packfile.c @@ -98,6 +98,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) return error("index file %s is too small", path); } do { + unuse_one_window(p); idx_map = xmmap_gently(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0); } while (idx_map == MAP_FAILED && errno == ENOMEM Signed-off-by: Eric Wong --- packfile.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packfile.c b/packfile.c index 4bc7fa0f36..2904560f52 100644 --- a/packfile.c +++ b/packfile.c @@ -97,7 +97,11 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) close(fd); return error("index file %s is too small", path); } - idx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0); + do { + idx_map = xmmap_gently(NULL, idx_size, PROT_READ, MAP_PRIVATE, + fd, 0); + } while (idx_map == MAP_FAILED && errno == ENOMEM + && unuse_one_window(p)); close(fd); ret = load_idx(path, hashsz, idx_map, idx_size, p); From patchwork Tue Jun 29 08:11:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Wong X-Patchwork-Id: 12349459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 789A5C11F6A for ; Tue, 29 Jun 2021 08:11:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5BC3D61DC2 for ; Tue, 29 Jun 2021 08:11:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232416AbhF2INs (ORCPT ); Tue, 29 Jun 2021 04:13:48 -0400 Received: from dcvr.yhbt.net ([64.71.152.64]:48148 "EHLO dcvr.yhbt.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232422AbhF2INq (ORCPT ); Tue, 29 Jun 2021 04:13:46 -0400 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 716181F9F3 for ; Tue, 29 Jun 2021 08:11:09 +0000 (UTC) From: Eric Wong To: git@vger.kernel.org Subject: [PATCH 4/4] xmmap: inform Linux users of tuning knobs on ENOMEM Date: Tue, 29 Jun 2021 08:11:08 +0000 Message-Id: <20210629081108.28657-5-e@80x24.org> In-Reply-To: <20210629081108.28657-1-e@80x24.org> References: <20210629081108.28657-1-e@80x24.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In cases where we can't (or haven't gotten around to) attempting graceful handling of mmap failures, Linux users may benefit from additional information on how to avoid the problem. Signed-off-by: Eric Wong --- object-file.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/object-file.c b/object-file.c index 4c043f1f3c..4139a5ebe6 100644 --- a/object-file.c +++ b/object-file.c @@ -1027,8 +1027,14 @@ void *xmmap(void *start, size_t length, int prot, int flags, int fd, off_t offset) { void *ret = xmmap_gently(start, length, prot, flags, fd, offset); - if (ret == MAP_FAILED) + if (ret == MAP_FAILED) { +#if defined(__linux__) + if (errno == ENOMEM) + die_errno(_( +"mmap failed, check sys.vm.max_map_count and/or RLIMIT_DATA")); +#endif /* OS-specific bits */ die_errno(_("mmap failed")); + } return ret; }