From patchwork Tue Mar 22 18:22:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2169EC433EF for ; Tue, 22 Mar 2022 18:23:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238491AbiCVSYj (ORCPT ); Tue, 22 Mar 2022 14:24:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238572AbiCVSYh (ORCPT ); Tue, 22 Mar 2022 14:24:37 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D6A4BFA for ; Tue, 22 Mar 2022 11:23:08 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id bi13-20020a05600c3d8d00b0038c2c33d8f3so3820403wmb.4 for ; Tue, 22 Mar 2022 11:23:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EIf3mbzht4BPeo7LeJtEkOmyAKN0GqJr9U0P3kl38NI=; b=el5daZ4QdU+96ATAsS50s9xpg5bWUwxv+8xaQ/uUbbqNwY2uNWkO+LvlvAp6dHhqqt ZNZmpRuIrNBHyydcOO4yzyhoYIzMD1Z67bHN1IoLWpuYvk6vnqg6FLdodCf1rzVlgykt ZwCVAxuBC2S5kdS8Y2ZHmZ8jekfD1eDTgGn144rBOkXkhAqQ8tiZH3DQykF4H48bZ7ta NwC5DHVZguiF+DkNzPIUVG1ZTmWVEoeFpbR9CARIIbHMIJVHrRyOJuFvGydXSUgjJKgr C+mATlNi+qUmugU+++SQJ/R7npZ5Al7xM4fylp4whJ0sahf3SPo4r6TIn5j0j4E4+u0J /hRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EIf3mbzht4BPeo7LeJtEkOmyAKN0GqJr9U0P3kl38NI=; b=xd58ZDiHpNbChJ4Pp8T8Oot3WLVS846kPnvUZp3nVrpVzTv2lgaQ4xGqY57E5Ht70f QsPAcLn/HSpC9knrqxSAb305SHQsV7uQIBj/t99cODiRHfFkB8jbkcZ5Q3aYraHTmEwd sCF1/0STJB32I75nZ+MsHIBHNcHE4DAJhOVd4SWYHHknA+rXtG8A/u0+PXh7eUtBw1iY Xrf2tWt1SMSMT5hzrRuqh4e24H+mI1npVefY5JKLjDzgqNzePgRyw8HmPTyIvgkixLoa sYe5i7iJWVRvmi3oJBn1u5B6zLGIl/e+CkEl65AQEo58xnU4atZVCoII2aBo0qKq2AiE ImSA== X-Gm-Message-State: AOAM531SmLU6eJHORm8HHFg+n6PWAnppX+rmKcj2NXoRfo16R5+UZpbj fXInBiPbhYeRoU+vkl9AGhEtRlXuV/8= X-Google-Smtp-Source: ABdhPJz9FMN8r1WPxV5ey/XXHoDOyh0rKvjqkJQ7pK4UVGd6A6GjqgLpwpYTqpPk4LQSDjD9YUUoSg== X-Received: by 2002:a1c:29c3:0:b0:350:9797:b38f with SMTP id p186-20020a1c29c3000000b003509797b38fmr5184318wmp.22.1647973386143; Tue, 22 Mar 2022 11:23:06 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e12-20020a5d6d0c000000b001a65e479d20sm8168737wrq.83.2022.03.22.11.23.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:04 -0700 (PDT) Message-Id: <779a15b38e8d48f13e7ba9bc460faa8cf6ea26f4.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:34 +0000 Subject: [PATCH v3 01/27] fsm-listen-win32: handle shortnames Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach FSMonitor daemon on Windows to recognize shortname paths as aliases of normal longname paths. FSMonitor clients, such as `git status`, should receive the longname spelling of changed files (when possible). Sometimes we receive FS events using the shortname, such as when a CMD shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1". The FS notification arrives using whatever combination of long and shortnames were used by the other process. (Shortnames do seem to be case normalized, however.) Use Windows GetLongPathNameW() to try to map the pathname spelling in the notification event into the normalized longname spelling. (This can fail if the file/directory is deleted, moved, or renamed, because we are asking the FS for the mapping in response to the event and after it has already happened, but we try.) Special case the shortname spelling of ".git" to avoid under-reporting these events. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++----- t/t7527-builtin-fsmonitor.sh | 65 +++++ 2 files changed, 374 insertions(+), 54 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 5b928ab66e5..3f1b68267bd 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -25,6 +25,9 @@ struct one_watch DWORD count; struct strbuf path; + wchar_t wpath_longname[MAX_PATH + 1]; + DWORD wpath_longname_len; + HANDLE hDir; HANDLE hEvent; OVERLAPPED overlapped; @@ -34,6 +37,21 @@ struct one_watch * need to later call GetOverlappedResult() and possibly CancelIoEx(). */ BOOL is_active; + + /* + * Are shortnames enabled on the containing drive? This is + * always true for "C:/" drives and usually never true for + * other drives. + * + * We only set this for the worktree because we only need to + * convert shortname paths to longname paths for items we send + * to clients. (We don't care about shortname expansion for + * paths inside a GITDIR because we never send them to + * clients.) + */ + BOOL has_shortnames; + BOOL has_tilda; + wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; struct fsmonitor_daemon_backend_data @@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data }; /* - * Convert the WCHAR path from the notification into UTF8 and - * then normalize it. + * Convert the WCHAR path from the event into UTF8 and normalize it. + * + * `wpath_len` is in WCHARS not bytes. */ -static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, +static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len, struct strbuf *normalized_path) { int reserve; int len = 0; strbuf_reset(normalized_path); - if (!info->FileNameLength) + if (!wpath_len) goto normalize; /* @@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, * sequence of 2 UTF8 characters. That should let us * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. */ - reserve = info->FileNameLength + 1; + reserve = 2 * wpath_len + 1; strbuf_grow(normalized_path, reserve); for (;;) { - len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, - info->FileNameLength / sizeof(WCHAR), + len = WideCharToMultiByte(CP_UTF8, 0, + wpath, wpath_len, normalized_path->buf, strbuf_avail(normalized_path) - 1, NULL, NULL); @@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, goto normalize; if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"), - GetLastError(), - (int)(info->FileNameLength / sizeof(WCHAR)), - info->FileName); + GetLastError(), (int)wpath_len, wpath); return -1; } @@ -98,6 +115,152 @@ normalize: return strbuf_normalize_path(normalized_path); } +/* + * See if the worktree root directory has shortnames enabled. + * This will help us decide if we need to do an expensive shortname + * to longname conversion on every notification event. + * + * We do not want to create a file to test this, so we assume that the + * root directory contains a ".git" file or directory. (Our caller + * only calls us for the worktree root, so this should be fine.) + * + * Remember the spelling of the shortname for ".git" if it exists. + */ +static void check_for_shortnames(struct one_watch *watch) +{ + wchar_t buf_in[MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + wchar_t *last_slash = NULL; + wchar_t *last_bslash = NULL; + wchar_t *last; + + /* build L"/.git" */ + wcscpy(buf_in, watch->wpath_longname); + wcscpy(buf_in + watch->wpath_longname_len, L".git"); + + if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH)) + return; + + last_slash = wcsrchr(buf_out, L'/'); + last_bslash = wcsrchr(buf_out, L'\\'); + if (last_slash > last_bslash) + last = last_slash + 1; + else if (last_bslash) + last = last_bslash + 1; + else + last = buf_out; + + if (!wcscmp(last, L".git")) + return; + + watch->has_shortnames = 1; + wcsncpy(watch->dotgit_shortname, last, + ARRAY_SIZE(watch->dotgit_shortname)); + + /* + * The shortname for ".git" is usually of the form "GIT~1", so + * we should be able to avoid shortname to longname mapping on + * every notification event if the source string does not + * contain a "~". + * + * However, the documentation for GetLongPathNameW() says + * that there are filesystems that don't follow that pattern + * and warns against this optimization. + * + * Lets test this. + */ + if (wcschr(watch->dotgit_shortname, L'~')) + watch->has_tilda = 1; +} + +enum get_relative_result { + GRR_NO_CONVERSION_NEEDED, + GRR_HAVE_CONVERSION, + GRR_SHUTDOWN, +}; + +/* + * Info notification paths are relative to the root of the watch. + * If our CWD is still at the root, then we can use relative paths + * to convert from shortnames to longnames. If our process has a + * different CWD, then we need to construct an absolute path, do + * the conversion, and then return the root-relative portion. + * + * We use the longname form of the root as our basis and assume that + * it already has a trailing slash. + * + * `wpath_len` is in WCHARS not bytes. + */ +static enum get_relative_result get_relative_longname( + struct one_watch *watch, + const wchar_t *wpath, DWORD wpath_len, + wchar_t *wpath_longname) +{ + wchar_t buf_in[2 * MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + DWORD root_len; + + /* Build L"/" */ + root_len = watch->wpath_longname_len; + wcsncpy(buf_in, watch->wpath_longname, root_len); + wcsncpy(buf_in + root_len, wpath, wpath_len); + buf_in[root_len + wpath_len] = 0; + + /* + * We don't actually know if the source pathname is a + * shortname or a longname. This routine allows either to be + * given as input. + */ + if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) { + /* + * The shortname to longname conversion can fail for + * various reasons, for example if the file has been + * deleted. (That is, if we just received a + * delete-file notification event and the file is + * already gone, we can't ask the file system to + * lookup the longname for it. Likewise, for moves + * and renames where we are given the old name.) + * + * Since deleting or moving a file or directory by its + * shortname is rather obscure, I'm going ignore the + * failure and ask the caller to report the original + * relative path. This seems kinder than failing here + * and forcing a resync. Besides, forcing a resync on + * every file/directory delete would effectively + * cripple monitoring. + * + * We might revisit this in the future. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (!wcscmp(buf_in, buf_out)) { + /* + * The path does not have a shortname alias. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (wcsncmp(buf_in, buf_out, root_len)) { + /* + * The spelling of the root directory portion of the computed + * longname has changed. This should not happen. Basically, + * it means that we don't know where (without recomputing the + * longname of just the root directory) to split out the + * relative path. Since this should not happen, I'm just + * going to let this fail and force a shutdown (because all + * subsequent events are probably going to see the same + * mismatch). + */ + return GRR_SHUTDOWN; + } + + /* Return the worktree root-relative portion of the longname. */ + + wcscpy(wpath_longname, buf_out + root_len); + return GRR_HAVE_CONVERSION; +} + void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); @@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, DWORD share_mode = FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; HANDLE hDir; - wchar_t wpath[MAX_PATH]; + DWORD len_longname; + wchar_t wpath[MAX_PATH + 1]; + wchar_t wpath_longname[MAX_PATH + 1]; if (xutftowcs_path(wpath, path) < 0) { error(_("could not convert to wide characters: '%s'"), path); @@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, return NULL; } + if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) { + error(_("[GLE %ld] could not get longname of '%s'"), + GetLastError(), path); + CloseHandle(hDir); + return NULL; + } + + len_longname = wcslen(wpath_longname); + if (wpath_longname[len_longname - 1] != L'/' && + wpath_longname[len_longname - 1] != L'\\') { + wpath_longname[len_longname++] = L'/'; + wpath_longname[len_longname] = 0; + } + CALLOC_ARRAY(watch, 1); watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ @@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, strbuf_init(&watch->path, 0); strbuf_addstr(&watch->path, path); + wcscpy(watch->wpath_longname, wpath_longname); + watch->wpath_longname_len = len_longname; + watch->hDir = hDir; watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); @@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch) watch->is_active = FALSE; } +/* + * Process a single relative pathname event. + * Return 1 if we should shutdown. + */ +static int process_1_worktree_event( + struct string_list *cookie_list, + struct fsmonitor_batch **batch, + const struct strbuf *path, + enum fsmonitor_path_type t, + DWORD info_action) +{ + const char *slash; + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path->buf); + string_list_append(cookie_list, + slash ? slash + 1 : path->buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info_action == FILE_ACTION_REMOVED) || + (info_action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + return 1; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!*batch) + *batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(*batch, path->buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path->buf); + } + + return 0; +} + /* * Process filesystem events that happen anywhere (recursively) under the * root directory. For a normal working directory, this includes @@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) struct string_list cookie_list = STRING_LIST_INIT_DUP; struct fsmonitor_batch *batch = NULL; const char *p = watch->buffer; + wchar_t wpath_longname[MAX_PATH + 1]; /* * If the kernel gets more events than will fit in the kernel @@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) */ for (;;) { FILE_NOTIFY_INFORMATION *info = (void *)p; - const char *slash; + wchar_t *wpath = info->FileName; + DWORD wpath_len = info->FileNameLength / sizeof(WCHAR); enum fsmonitor_path_type t; + enum get_relative_result grr; + + if (watch->has_shortnames) { + if (!wcscmp(wpath, watch->dotgit_shortname)) { + /* + * This event exactly matches the + * spelling of the shortname of + * ".git", so we can skip some steps. + * + * (This case is odd because the user + * can "rm -rf GIT~1" and we cannot + * use the filesystem to map it back + * to ".git".) + */ + strbuf_reset(&path); + strbuf_addstr(&path, ".git"); + t = IS_DOT_GIT; + goto process_it; + } - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) - goto skip_this_path; - - t = fsmonitor_classify_path_workdir_relative(path.buf); - - switch (t) { - case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: - /* special case cookie files within .git */ - - /* Use just the filename of the cookie file. */ - slash = find_last_dir_sep(path.buf); - string_list_append(&cookie_list, - slash ? slash + 1 : path.buf); - break; - - case IS_INSIDE_DOT_GIT: - /* ignore everything inside of "/.git/" */ - break; + if (watch->has_tilda && !wcschr(wpath, L'~')) { + /* + * Shortnames on this filesystem have tildas + * and the notification path does not have + * one, so we assume that it is a longname. + */ + goto normalize_it; + } - case IS_DOT_GIT: - /* "/.git" was deleted (or renamed away) */ - if ((info->Action == FILE_ACTION_REMOVED) || - (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { - trace2_data_string("fsmonitor", NULL, - "fsm-listen/dotgit", - "removed"); + grr = get_relative_longname(watch, wpath, wpath_len, + wpath_longname); + switch (grr) { + case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */ + break; + case GRR_HAVE_CONVERSION: + wpath = wpath_longname; + wpath_len = wcslen(wpath); + break; + default: + case GRR_SHUTDOWN: goto force_shutdown; } - break; + } - case IS_WORKDIR_PATH: - /* queue normal pathname */ - if (!batch) - batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, path.buf); - break; +normalize_it: + if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1) + goto skip_this_path; - case IS_GITDIR: - case IS_INSIDE_GITDIR: - case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: - default: - BUG("unexpected path classification '%d' for '%s'", - t, path.buf); - } + t = fsmonitor_classify_path_workdir_relative(path.buf); + +process_it: + if (process_1_worktree_event(&cookie_list, &batch, &path, t, + info->Action)) + goto force_shutdown; skip_this_path: if (!info->NextEntryOffset) @@ -382,6 +630,9 @@ force_shutdown: * Note that we DO NOT get filesystem events on the external * itself (it is not inside something that we are watching). In particular, * we do not get an event if the external is deleted. + * + * Also, we do not care about shortnames within the external , since + * we never send these paths to clients. */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { @@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state) const char *slash; enum fsmonitor_path_type t; - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) + if (normalize_path_in_utf8( + info->FileName, + info->FileNameLength / sizeof(WCHAR), + &path) == -1) goto skip_this_path; t = fsmonitor_classify_path_gitdir_relative(path.buf); @@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) if (!data->watch_worktree) goto failed; + check_for_shortnames(data->watch_worktree); + if (state->nr_paths_watching > 1) { data->watch_gitdir = create_watch(state, state->path_gitdir_watch.buf); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 389ebf431c6..d28a74feeb9 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -173,6 +173,71 @@ test_expect_success 'implicit daemon stop (rename .git)' ' test_must_fail git -C test_implicit_2 fsmonitor--daemon status ' +# File systems on Windows may or may not have shortnames. +# This is a volume-specific setting on modern systems. +# "C:/" drives are required to have them enabled. Other +# hard drives default to disabled. +# +# This is a crude test to see if shortnames are enabled +# on the volume containing the test directory. It is +# crude, but it does not require elevation like `fsutil`. +# +test_lazy_prereq SHORTNAMES ' + mkdir .foo && + test -d "FOO~1" +' + +# Here we assume that the shortname of ".git" is "GIT~1". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s" && + + git init test_implicit_1s && + + start_daemon -C test_implicit_1s && + + # renaming the .git directory will implicitly stop the daemon. + # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + # this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}. + mv test_implicit_1s/.gitxyz test_implicit_1s/.git && + + test_must_fail git -C test_implicit_1s fsmonitor--daemon status +' + +# Here we first create a file with LONGNAME of "GIT~1" before +# we create the repo. This will cause the shortname of ".git" +# to be "GIT~2". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s2" && + + mkdir test_implicit_1s2 && + echo HELLO >test_implicit_1s2/GIT~1 && + git init test_implicit_1s2 && + + test_path_is_file test_implicit_1s2/GIT~1 && + test_path_is_dir test_implicit_1s2/GIT~2 && + + start_daemon -C test_implicit_1s2 && + + # renaming the .git directory will implicitly stop the daemon. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git && + + test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status +' + test_expect_success 'cannot start multiple daemons' ' test_when_finished "stop_daemon_delete_repo test_multiple" && From patchwork Tue Mar 22 18:22:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6164BC433EF for ; Tue, 22 Mar 2022 18:23:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240275AbiCVSY4 (ORCPT ); Tue, 22 Mar 2022 14:24:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240260AbiCVSYk (ORCPT ); Tue, 22 Mar 2022 14:24:40 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70D1CBFA for ; Tue, 22 Mar 2022 11:23:11 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id m30so14799631wrb.1 for ; Tue, 22 Mar 2022 11:23:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sCXgYeR36GUqmZmNkmliiej5zTY6qgDP8cvb8xuW+qw=; b=Kg8E0aAyULo0LgPZeaKvqpmuo1dJlqCpvv0gN3sRSmI2zbjHg8cMJ6BE2ow8KQBQiC LjxpJLoka7m2A1FpG7uDUgjLkl+Cyjc6u/DtWyes5ObmN7Y/BQ0WnNBsE+/scEYK2JGI OWuPhRi8TwpwtHlnM1ZntwCWCnaDrx3Fm6z8jPPRnQe1PEYhQj2i08hhCHbRKdmfrjm/ nB0hJHRooPYQCljkCz3zezlfq+K1F+EyxDT5c5kXMH6DyVjZQb8YAcaJqfRVVr0F02cx CA3TLYT1IBOUa24S0d0akQdWTF7dURzqIg4F2VdOH9JTs0QBXFb71f+b7OUSaEe1ZKgs YY+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sCXgYeR36GUqmZmNkmliiej5zTY6qgDP8cvb8xuW+qw=; b=5753yPhu4/OGKPb/Sz1cmTq/5xGQxG55YrQn3kC33pSjZJBGQvjGWLC3eWyJ7Sg0g7 C6GhclUlW0IAKN+Y4tj35gcKIyR/iToxVrUSh0BVK40NOMpu5W6pBE20xI3JwXGZUDO9 xNWNOddKER72BqVpY1sxikao2LWH5Vad/lqnsM8lFidyleRBAIjpIvUnTYVk+775k4it npdhhBD8uosLv1kr/jzISXv3NCHt49gjNwIIUQ1dTjC7EbwctSIoOJs5Jn//8mSlO4gl BuHtElaoyrisHCUSPr10XNuvzk6ZBV+1VVC2PCw35nd+jTrPpIswUI3wqhmp4FSwlY23 EkFg== X-Gm-Message-State: AOAM5310m6HjlToeThJv5AkVi4/VbfIkr0Vez57X3+pcj4NgZt/79tEt iTiGJjB4tBBWoPmXHn3GYFtjYoVuq5E= X-Google-Smtp-Source: ABdhPJxSVI0SRUMG2oDxmQBfe787q5GcNGjk+knqmYvdtHQq9E9ke+F0d9pUoRh9pBK6yAUdLZ/b1Q== X-Received: by 2002:adf:8148:0:b0:203:f970:301c with SMTP id 66-20020adf8148000000b00203f970301cmr17517215wrm.508.1647973389872; Tue, 22 Mar 2022 11:23:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u12-20020a5d6dac000000b00204119d37d0sm5930342wrs.26.2022.03.22.11.23.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:07 -0700 (PDT) Message-Id: <11d4a17b69239bdce17a7b4d76a3491c29976c13.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:35 +0000 Subject: [PATCH v3 02/27] t7527: test FSMonitor on repos with Unicode root paths Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create some test repos with UTF8 characters in the pathname of the root directory and verify that the builtin FSMonitor can watch them. This test is mainly for Windows where we need to avoid `*A()` routines. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index d28a74feeb9..429029fcadd 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -656,4 +656,27 @@ do done done +# Test Unicode UTF-8 characters in the pathname of the working +# directory root. Use of "*A()" routines rather than "*W()" routines +# on Windows can sometimes lead to odd failures. +# +u1=$(printf "u_c3_a6__\xC3\xA6") +u2=$(printf "u_e2_99_ab__\xE2\x99\xAB") +u_values="$u1 $u2" +for u in $u_values +do + test_expect_success "Unicode in repo root path: $u" ' + test_when_finished "stop_daemon_delete_repo $u" && + + git init "$u" && + echo 1 >"$u"/file1 && + git -C "$u" add file1 && + git -C "$u" config core.fsmonitor true && + + start_daemon -C "$u" && + git -C "$u" status >actual && + grep "new file: file1" actual + ' +done + test_done From patchwork Tue Mar 22 18:22:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCE3FC433F5 for ; Tue, 22 Mar 2022 18:23:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240286AbiCVSY5 (ORCPT ); Tue, 22 Mar 2022 14:24:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240282AbiCVSYz (ORCPT ); Tue, 22 Mar 2022 14:24:55 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18B411F623 for ; Tue, 22 Mar 2022 11:23:14 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id v130-20020a1cac88000000b00389d0a5c511so3810022wme.5 for ; Tue, 22 Mar 2022 11:23:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8V1FoXCb86xqZ2XOaLWrGr4avTXqor6TX1WNeRZ9avw=; b=J42ZzPVvrZZyqS7Bunm/ZXZula4c8FXnn50t4fnm+4IYEnCZpO3haeZSGpzBndt3i/ xoWmVNcyEBZMkIaOQCnOlaGJS3TPQjfKh2D22xYkQn9uEUjK3DMaSmfGa8k7hLMjlNnm CoOnMPxlGp+KrxUygs8TC60JCrKZuvFOLNkDgvv9vt5NMVRaAN78DTavyON7DSDJCEqM kI1eAtxphCzVqA6Os7yt38HTSXYT49LeHscQK5i7bOzThK+RSH+BWyMIV6b1+/XDKYmb 9BOc62NRHUEKSEqJ++cPn00QJVs0VfyUv4zLvcdv4h8GomX2rqZN0nUrXb+ECYzr2BW1 m1zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8V1FoXCb86xqZ2XOaLWrGr4avTXqor6TX1WNeRZ9avw=; b=KHEhEyEdFelI7wCudvC0rpqJCrdUdQvXqG02DgFwXbfsW+t7aJfTrxSIzRzT/48TGk VzD7HEx0nyD4apI940dWjTlFX8Rx1p1lQU0QXeAhqdrpFgxpc/miir0qx3+CWV8IHY4K JFUPVEMeFXtMKOKOGOp6My7yLwWQCIaSC4iXDcGuK3WtdSH5McNwacr9JYHFbpaZ/H7z tPxVO0PE58dWncwS4gHsD6rYSB4LWFiPDxP4DlRn8sZtvPM/FXfhZarX5sISVjaJ4v9L SvncctTClL4zhwdJiwyvyAudnGAHYyFsacKBWSyVK+nsMI1PCpsks260hF1Ueq56FU+L Ydng== X-Gm-Message-State: AOAM530vaCRQ0CiF9fqNag832g2D3GfP62HmIku6HltwhX9W3UoNfty2 8OJ7MyIcj0MXhfr7j3yroeuIqmEFgLc= X-Google-Smtp-Source: ABdhPJxkFiYMMbnzRdXzYLLP38v2nF5myKw8mKK5YqkJPD+JPaIyAEOqqqg3HL1gabh7DB54L0i/tQ== X-Received: by 2002:a7b:c8d5:0:b0:38c:9eb9:129 with SMTP id f21-20020a7bc8d5000000b0038c9eb90129mr5264547wml.65.1647973392489; Tue, 22 Mar 2022 11:23:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c24-20020a7bc018000000b0038a18068cf5sm2306801wmb.15.2022.03.22.11.23.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:11 -0700 (PDT) Message-Id: <901fa32f6eacab7ba76aedbb020855c8f9718943.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:36 +0000 Subject: [PATCH v3 03/27] t/helper/fsmonitor-client: create stress test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a stress test to hammer on the fsmonitor daemon. Create a client-side thread pool of n threads and have each of them make m requests as fast as they can. We do not currently inspect the contents of the response. We're only interested in placing a heavy request load on the daemon. This test is useful for interactive testing and various experimentation. For example, to place additional load on the daemon while another test is running. We currently do not have a test script that actually uses this helper. We might add such a test in the future. Signed-off-by: Jeff Hostetler --- t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c index 3062c8a3c2b..54a4856c48c 100644 --- a/t/helper/test-fsmonitor-client.c +++ b/t/helper/test-fsmonitor-client.c @@ -7,6 +7,8 @@ #include "cache.h" #include "parse-options.h" #include "fsmonitor-ipc.h" +#include "thread-utils.h" +#include "trace2.h" #ifndef HAVE_FSMONITOR_DAEMON_BACKEND int cmd__fsmonitor_client(int argc, const char **argv) @@ -79,20 +81,121 @@ static int do_send_flush(void) return 0; } +struct hammer_thread_data +{ + pthread_t pthread_id; + int thread_nr; + + int nr_requests; + const char *token; + + int sum_successful; + int sum_errors; +}; + +static void *hammer_thread_proc(void *_hammer_thread_data) +{ + struct hammer_thread_data *data = _hammer_thread_data; + struct strbuf answer = STRBUF_INIT; + int k; + int ret; + + trace2_thread_start("hammer"); + + for (k = 0; k < data->nr_requests; k++) { + strbuf_reset(&answer); + + ret = fsmonitor_ipc__send_query(data->token, &answer); + if (ret < 0) + data->sum_errors++; + else + data->sum_successful++; + } + + strbuf_release(&answer); + trace2_thread_exit(); + return NULL; +} + +/* + * Start a pool of client threads that will each send a series of + * commands to the daemon. + * + * The goal is to overload the daemon with a sustained series of + * concurrent requests. + */ +static int do_hammer(const char *token, int nr_threads, int nr_requests) +{ + struct hammer_thread_data *data = NULL; + int k; + int sum_join_errors = 0; + int sum_commands = 0; + int sum_errors = 0; + + if (!token || !*token) + token = get_token_from_index(); + if (nr_threads < 1) + nr_threads = 1; + if (nr_requests < 1) + nr_requests = 1; + + CALLOC_ARRAY(data, nr_threads); + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + p->thread_nr = k; + p->nr_requests = nr_requests; + p->token = token; + + if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) { + warning("failed to create thread[%d] skipping remainder", k); + nr_threads = k; + break; + } + } + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + + if (pthread_join(p->pthread_id, NULL)) + sum_join_errors++; + sum_commands += p->sum_successful; + sum_errors += p->sum_errors; + } + + fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n", + nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors); + + free(data); + + /* + * Return an error if any of the _send_query requests failed. + * We don't care about thread create/join errors. + */ + return sum_errors > 0; +} + int cmd__fsmonitor_client(int argc, const char **argv) { const char *subcmd; const char *token = NULL; + int nr_threads = 1; + int nr_requests = 1; const char * const fsmonitor_client_usage[] = { "test-tool fsmonitor-client query []", "test-tool fsmonitor-client flush", + "test-tool fsmonitor-client hammer [] [] []", NULL, }; struct option options[] = { OPT_STRING(0, "token", &token, "token", "command token to send to the server"), + + OPT_INTEGER(0, "threads", &nr_threads, "number of client threads"), + OPT_INTEGER(0, "requests", &nr_requests, "number of requests per thread"), + OPT_END() }; @@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv) if (!strcmp(subcmd, "flush")) return !!do_send_flush(); + if (!strcmp(subcmd, "hammer")) + return !!do_hammer(token, nr_threads, nr_requests); + die("Unhandled subcommand: '%s'", subcmd); } #endif From patchwork Tue Mar 22 18:22:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 550D9C433EF for ; Tue, 22 Mar 2022 18:23:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240281AbiCVSY6 (ORCPT ); Tue, 22 Mar 2022 14:24:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240287AbiCVSYz (ORCPT ); Tue, 22 Mar 2022 14:24:55 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5DC76E4DE for ; Tue, 22 Mar 2022 11:23:16 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id v130-20020a1cac88000000b00389d0a5c511so3810098wme.5 for ; Tue, 22 Mar 2022 11:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=f7rf5XJgM6jW8M+rq9uT0VeICIGviBdUswkTE7UTxew=; b=REW/s6QoTeKzFg+vOpnsintxevGtVpm5gY9YogExAdgUj5IuJ00WXU1dfN9ibObrxq tD0zPaMipiqWnHSATcYwKKa7u1pHl25qe4Bcu7/ly+UYRqhEPqdp5wD+a4febxYxmAX1 r5AOBm9mQgQgYLcO7zjp8GaX6iO71/oBuLKbgCqCeDJ4MtTfGet7P4n8eijFwdSt2Hxb PabM6QGvsgb/hRHvDNHpu9/T3RyZVtbYz4jF+Dz/1xEDVRarSsOOLGjZiHGfGDXgsQK7 u+3jIlHbYTaEHMnFgRCDcA0qI7JzyRJGxMHjRszd8g+dcnPX7TbQGbLkgoWiFPeMSf9m +IrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=f7rf5XJgM6jW8M+rq9uT0VeICIGviBdUswkTE7UTxew=; b=j9PJZLWWGP1cOB89cWT7/AzwncDD/ooJWLhTE0IEz9iiWit8+Bf+uSUuqHfGUIUrr+ d0AsIY9lanDY2KPAXO60XXLxSwdvyRwQK2V3qYOHanzRTxFiFMrEoszvKICIWO2q7RsO pQ0IbBs9zR6mF3ZkPNRm4/TZTL/sQbHgxbBmG467np/EiaSsAEi9wEU+FI/cTi69Joq6 4tnaFSC6wcxRNxQr1N3y08Jd+X1Pxza7D/NzFul5KIDINeczA5U6G0eXwHqIXhNWUyr/ 0DPKcMbDfleRN5zVaCvPjO8YttvYtWAp5VENeQh7l+o0xKiLkLk2AFttZEtGrKcCCa5L eZOw== X-Gm-Message-State: AOAM533FAkr1feUCIwQBNsMllUwywrolMM0ToJNu1Kh0N8BlSh2RMkjL XpFrbeRRFu3tVP8Biow6L151oHrg1nU= X-Google-Smtp-Source: ABdhPJzfyfdflDnh06QUVKHaunFq2pr0Ezb5785K+HvtnZsk4U8pMAiR9N1x7FJB3RNjNDf3HcU2TA== X-Received: by 2002:a05:600c:3c8b:b0:37f:1546:40c9 with SMTP id bg11-20020a05600c3c8b00b0037f154640c9mr5019530wmb.161.1647973394784; Tue, 22 Mar 2022 11:23:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u8-20020a5d4348000000b00203dbfa4ff2sm16198635wrr.34.2022.03.22.11.23.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:13 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:37 +0000 Subject: [PATCH v3 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Bare repos do not have a worktree, so there is nothing for the daemon watch. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 6 ++++ builtin/update-index.c | 4 +++ fsmonitor-settings.c | 60 +++++++++++++++++++++++++++++++++++++ fsmonitor-settings.h | 12 ++++++++ t/t7519-status-fsmonitor.sh | 23 ++++++++++++++ 5 files changed, 105 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index a30ecf6cfac..edca8a667ab 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1450,6 +1450,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + prepare_repo_settings(the_repository); + fsm_settings__set_ipc(the_repository); + + if (fsm_settings__error_if_incompatible(the_repository)) + return 1; + if (!strcmp(subcmd, "start")) return !!try_to_start_background_daemon(); diff --git a/builtin/update-index.c b/builtin/update-index.c index 876112abb21..d29048f16f2 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) if (fsmonitor > 0) { enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); + + if (fsm_settings__error_if_incompatible(the_repository)) + return 1; + if (fsm_mode == FSMONITOR_MODE_DISABLED) { warning(_("core.fsmonitor is unset; " "set it if you really want to " diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 757d230d538..86c09bd35fe 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -9,9 +9,33 @@ */ struct fsmonitor_settings { enum fsmonitor_mode mode; + enum fsmonitor_reason reason; char *hook_path; }; +static void set_incompatible(struct repository *r, + enum fsmonitor_reason reason) +{ + struct fsmonitor_settings *s = r->settings.fsmonitor; + + s->mode = FSMONITOR_MODE_INCOMPATIBLE; + s->reason = reason; +} + +static int check_for_incompatible(struct repository *r) +{ + if (!r->worktree) { + /* + * Bare repositories don't have a working directory and + * therefore have nothing to watch. + */ + set_incompatible(r, FSMONITOR_REASON_BARE); + return 1; + } + + return 0; +} + static void lookup_fsmonitor_settings(struct repository *r) { struct fsmonitor_settings *s; @@ -23,6 +47,7 @@ static void lookup_fsmonitor_settings(struct repository *r) CALLOC_ARRAY(s, 1); s->mode = FSMONITOR_MODE_DISABLED; + s->reason = FSMONITOR_REASON_OK; r->settings.fsmonitor = s; @@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r) lookup_fsmonitor_settings(r); + if (check_for_incompatible(r)) + return; + r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } @@ -97,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path) lookup_fsmonitor_settings(r); + if (check_for_incompatible(r)) + return; + r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK; FREE_AND_NULL(r->settings.fsmonitor->hook_path); r->settings.fsmonitor->hook_path = strdup(path); @@ -110,5 +141,34 @@ void fsm_settings__set_disabled(struct repository *r) lookup_fsmonitor_settings(r); r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED; + r->settings.fsmonitor->reason = FSMONITOR_REASON_OK; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } + +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + return r->settings.fsmonitor->reason; +} + +int fsm_settings__error_if_incompatible(struct repository *r) +{ + enum fsmonitor_reason reason = fsm_settings__get_reason(r); + + switch (reason) { + case FSMONITOR_REASON_OK: + return 0; + + case FSMONITOR_REASON_BARE: + error(_("bare repository '%s' is incompatible with fsmonitor"), + xgetcwd()); + return 1; + } + + BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'", + reason); +} diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index a4c5d7b4889..8654edf33d8 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -4,11 +4,20 @@ struct repository; enum fsmonitor_mode { + FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */ FSMONITOR_MODE_DISABLED = 0, FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */ FSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */ }; +/* + * Incompatibility reasons. + */ +enum fsmonitor_reason { + FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ + FSMONITOR_REASON_BARE, +}; + void fsm_settings__set_ipc(struct repository *r); void fsm_settings__set_hook(struct repository *r, const char *path); void fsm_settings__set_disabled(struct repository *r); @@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r); enum fsmonitor_mode fsm_settings__get_mode(struct repository *r); const char *fsm_settings__get_hook_path(struct repository *r); +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r); +int fsm_settings__error_if_incompatible(struct repository *r); + struct fsmonitor_settings; #endif /* FSMONITOR_SETTINGS_H */ diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index a6308acf006..9a8e21c5608 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE ' test $ret -ne 1 ' +# Test that we detect and disallow repos that are incompatible with FSMonitor. +test_expect_success 'incompatible bare repo' ' + test_when_finished "rm -rf ./bare-clone actual expect" && + git init --bare bare-clone && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=foo \ + update-index --fsmonitor 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=true \ + update-index --fsmonitor 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual +' + +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' + test_when_finished "rm -rf ./bare-clone actual" && + git init --bare bare-clone && + test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Tue Mar 22 18:22:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BAAAC433EF for ; Tue, 22 Mar 2022 18:23:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229529AbiCVSZB (ORCPT ); Tue, 22 Mar 2022 14:25:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240293AbiCVSYz (ORCPT ); Tue, 22 Mar 2022 14:24:55 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC7A98D6B0 for ; Tue, 22 Mar 2022 11:23:17 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id bg31-20020a05600c3c9f00b00381590dbb33so2469839wmb.3 for ; Tue, 22 Mar 2022 11:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=77SaOZDqR5GCe8rGTE1gWxph/6mABa7uqFiGsLfjuL0=; b=MIE5c4J9MozwUS7FbnvSS2qpD2yfKj3FXtNuPI6SFIyvPNp5hqqACi4N2wz1WcTA1w N9Dqkj49867YdWtSJX9FSwwvvimqr9cArl3WK0MK50FbmjtaQ99Vu+9xrpBDl1R3m9Gq QekGdGd7VUcQLeFzz4fVt4JbzHyIDchsBpQ5YPdYrZqqQ8cSqEiTK5CNT/lSEzyU/zD/ Rsanx8aExLz4Lv6vvNMrCz5AmsD7IvfNYvExZVmhPc+LCA+Oyvhs2ogpjE8cw/OnBgPf XhAFQzUdUNGonpxMVgUAkHsjkJulovbGpOcz9QMTAmnANf6Je42G4yDoGOVsIaXs+0HS p4LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=77SaOZDqR5GCe8rGTE1gWxph/6mABa7uqFiGsLfjuL0=; b=P1qKRsaofL1J2ExJJi4loYZODjAfRbq8xOvV7lLYVilcbmHyOIzXhdXsIOnrk8R/CG YcG5qaFrNXryVLtTU96RZGbWfyaZ+2npdvVuE5QctvB/Kad4S1fKV8hz6yZrm9VUvWOz 9qQWgT6N+YJwS89/uTdZzg7mvKshPQ/wdhtjK1DjyGRoaMGlM8mdiA6MSA7v6htSJJRU LRctF14l/qP6sOfaM8hTOpwHKxFQIR/4SkU3OPVW4b6syFLLlgxdXY8K7OA1Z7Sfgxlz /RSApjxugleQFYLRKiCZU+mCR7JctyWJyc9pw50qw+y0p3BGlQL/BfM++n7pJOYVGvdq qhig== X-Gm-Message-State: AOAM531tHUUNy3k/ra0cTACU3snTIYpvGbcnusTLfrRUVWczH7QQiXe+ JzYzz12HAT2d4tlFhJRyRp8TQ8USMVk= X-Google-Smtp-Source: ABdhPJwTuBlkmhId6D6hkkm7KKR5cSD8eH86db2zQCnQMlrtiDEw1+4krwEIemhrf6E0CxfULeVRug== X-Received: by 2002:a1c:f018:0:b0:37b:c13c:3128 with SMTP id a24-20020a1cf018000000b0037bc13c3128mr5067122wmb.157.1647973396172; Tue, 22 Mar 2022 11:23:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u11-20020a05600c19cb00b00389efe9c512sm2403498wmq.23.2022.03.22.11.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:38 +0000 Subject: [PATCH v3 05/27] fsmonitor-settings: stub in Win32-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend generic incompatibility checkout with platform-specific mechanism. Stub in Win32 version. In the existing fsmonitor-settings code we have a way to mark types of repos as incompatible with fsmonitor (whether via the hook and IPC APIs). For example, we do this for bare repos, since there are no files to watch. Extend this exclusion mechanism for platform-specific reasons. This commit just creates the framework and adds a stub for Win32. Signed-off-by: Jeff Hostetler --- Makefile | 13 +++++++++++++ compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++ config.mak.uname | 4 ++++ contrib/buildsystems/CMakeLists.txt | 3 +++ fsmonitor-settings.c | 12 ++++++++++++ fsmonitor-settings.h | 13 +++++++++++++ 6 files changed, 54 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-win32.c diff --git a/Makefile b/Makefile index daa21bed6c3..93604fe8ef7 100644 --- a/Makefile +++ b/Makefile @@ -475,6 +475,11 @@ all:: # `compat/fsmonitor/fsm-listen-.c` that implements the # `fsm_listen__*()` routines. # +# If your platform has OS-specific ways to tell if a repo is incompatible with +# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS +# to the "" of the corresponding `compat/fsmonitor/fsm-settings-.c` +# that implements the `fsm_os_settings__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o endif +ifdef FSMONITOR_OS_SETTINGS + COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE ifdef FSMONITOR_DAEMON_BACKEND @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ endif +ifdef FSMONITOR_OS_SETTINGS + @echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index 501970902da..cf224768ad6 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index ee0d7257b77..16705da2000 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 86c09bd35fe..8ff55f8c3fd 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r) return 1; } +#ifdef HAVE_FSMONITOR_OS_SETTINGS + { + enum fsmonitor_reason reason; + + reason = fsm_os__incompatible(r); + if (reason != FSMONITOR_REASON_OK) { + set_incompatible(r, reason); + return 1; + } + } +#endif + return 0; } diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 8654edf33d8..4b35f051fb1 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -30,4 +30,17 @@ int fsm_settings__error_if_incompatible(struct repository *r); struct fsmonitor_settings; +#ifdef HAVE_FSMONITOR_OS_SETTINGS +/* + * Ask platform-specific code whether the repository is incompatible + * with fsmonitor (both hook and ipc modes). For example, if the working + * directory is on a remote volume and mounted via a technology that does + * not support notification events, then we should not pretend to watch it. + * + * fsm_os__* routines should considered private to fsm_settings__ + * routines. + */ +enum fsmonitor_reason fsm_os__incompatible(struct repository *r); +#endif /* HAVE_FSMONITOR_OS_SETTINGS */ + #endif /* FSMONITOR_SETTINGS_H */ From patchwork Tue Mar 22 18:22:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDD02C433F5 for ; Tue, 22 Mar 2022 18:23:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240320AbiCVSZE (ORCPT ); Tue, 22 Mar 2022 14:25:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240296AbiCVSYz (ORCPT ); Tue, 22 Mar 2022 14:24:55 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7949E1FCE5 for ; Tue, 22 Mar 2022 11:23:19 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id i186so2473274wma.3 for ; Tue, 22 Mar 2022 11:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ekJrkqFg1yJzdNh+egoYJeP3EUXrY0TIhPEKqSMHHf8=; b=YEYQkxVEXHurfr0DMuHdVPtfOOMrXIk5ki9NQJSEA6hrd+Bi4dEXLP0u2OdrDPwCNN M2r4/ItkCl54aqAnCVxL4+pN9P4J/GD8RlkeHqRx5RDe5pHRe+1zJlHV88Gqwzl0D5kD 9RevwAw/wvsjUCNYzmqZpf3r7zj5sHQ2B2QdaUBo4FG/LXD1jKU95w8uUt4DXN/8tWg0 yepoSuyXUD3BnAAht/DD3EU9LcHP72n2IGJwLMDUt5apXA+x9CGUcTRrrQuDpSm/pZtg b33NjjNA8Ez81UrhX4tDx/1hn7f7DKmwtZvWd8c+F+QDR9jJHnhPaIlR+BlcFXOFujpc DfFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ekJrkqFg1yJzdNh+egoYJeP3EUXrY0TIhPEKqSMHHf8=; b=CjuclAdzLVGWgrM98QogxfQCEN15vfvdsEEyxRMYlaIEw7WD5k4f/juEtSpeeT4pnC DwvODKzXVGmrgRJYO35GgCgemyq+8zVooJIhVuvaKQzNK1EGlpco49mmixkgGXr8JXQv DsU1uPzykX19jfEAtBncBS6qxKHmO03WjZImzvnmWOQQg7xMWnZNl59oKy++N7/2Otzq PfR573erqX10EIpXtDRML243nL/JaVNTwlhDr8l2JXpalDIjkrMFbuZKNfhqiSoVtzIR oEQi48zqcJVTe7LA1/dptwmjdxOZrMcE2xMlO/OHztBJoSJ6eI0/b1O/Nor3qLKB5PXf Ee1A== X-Gm-Message-State: AOAM530KCdaGYfRfNPkVhk2Gj3xelqmMMJnYr9UW6+sYP5Yiwg3Tm9fF 8d6EXyXsJCBJ58m/YjVdZa5RnxygkeI= X-Google-Smtp-Source: ABdhPJwldKXQC5OZCvMEQmaQrrLyEhbs/oLuAEuJXWP5FVgbjiO97v9Cpr4st4HyMhrPqvNpz5z2dw== X-Received: by 2002:a05:600c:ca:b0:38c:b426:3124 with SMTP id u10-20020a05600c00ca00b0038cb4263124mr5181052wmm.59.1647973398060; Tue, 22 Mar 2022 11:23:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i14-20020a0560001ace00b00203da1fa749sm28498072wry.72.2022.03.22.11.23.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:16 -0700 (PDT) Message-Id: <5546339d963caf37828da4b32deb896162cd5db9.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:39 +0000 Subject: [PATCH v3 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler VFS for Git virtual repositories are incompatible with FSMonitor. VFS for Git is a downstream fork of Git. It contains its own custom file system watcher that is aware of the virtualization. If a working directory is being managed by VFS for Git, we should not try to watch it because we may get incomplete results. We do not know anything about how VFS for Git works, but we do know that VFS for Git working directories contain a well-defined config setting. If it is set, mark the working directory as incompatible. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++ fsmonitor-settings.c | 5 +++++ fsmonitor-settings.h | 1 + t/t7519-status-fsmonitor.sh | 9 +++++++++ 4 files changed, 41 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index 7fce32a3c5b..ee78bba38e3 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -3,7 +3,33 @@ #include "repository.h" #include "fsmonitor-settings.h" +/* + * VFS for Git is incompatible with FSMonitor. + * + * Granted, core Git does not know anything about VFS for Git and we + * shouldn't make assumptions about a downstream feature, but users + * can install both versions. And this can lead to incorrect results + * from core Git commands. So, without bringing in any of the VFS for + * Git code, do a simple config test for a published config setting. + * (We do not look at the various *_TEST_* environment variables.) + */ +static enum fsmonitor_reason check_vfs4git(struct repository *r) +{ + const char *const_str; + + if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str)) + return FSMONITOR_REASON_VFS4GIT; + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_vfs4git(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 8ff55f8c3fd..1efb6e17a20 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -179,6 +179,11 @@ int fsm_settings__error_if_incompatible(struct repository *r) error(_("bare repository '%s' is incompatible with fsmonitor"), xgetcwd()); return 1; + + case FSMONITOR_REASON_VFS4GIT: + error(_("virtual repository '%s' is incompatible with fsmonitor"), + r->worktree); + return 1; } BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 4b35f051fb1..6361fcbf6b0 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -16,6 +16,7 @@ enum fsmonitor_mode { enum fsmonitor_reason { FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; void fsm_settings__set_ipc(struct repository *r); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 9a8e21c5608..156895f9437 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' grep "bare repository .* is incompatible with fsmonitor" actual ' +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' ' + test_when_finished "rm -rf ./fake-virtual-clone actual" && + git init fake-virtual-clone && + test_must_fail git -C ./fake-virtual-clone \ + -c core.virtualfilesystem=true \ + fsmonitor--daemon run 2>actual && + grep "virtual repository .* is incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Tue Mar 22 18:22:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C148C433EF for ; Tue, 22 Mar 2022 18:23:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240387AbiCVSZY (ORCPT ); Tue, 22 Mar 2022 14:25:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240303AbiCVSY5 (ORCPT ); Tue, 22 Mar 2022 14:24:57 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 846F78F997 for ; Tue, 22 Mar 2022 11:23:21 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id r13so9595359wrr.9 for ; Tue, 22 Mar 2022 11:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=Fep8wKJi05oR1j2xfKPO4QQpP5vwL62wvm8weTkIToGEgq8KQtFR2O3q1UQmU3jCEF FbJyJswonHTOhFgiwzVTVFT7ckQutklclENrY4rAlZlqZoNKfxPb7U74djN1/B4p6dXo Q6FJSkQJOOx2NNG9yqHEhvs1tIOTG2C4zFyGP893qCP6dNL0jSmfSJkt2O5AeqKwd7Iz Gh1uHfLLPmeRr7pttmNbpT522lxQeMHBY6VMbMMmjA50qeuiojOuTdBQXCisTBhP1dVt 19BiBjUXp6WiN1wzbyGcBMlTMIlp9Ck8EZlwuHdNvXeFTmGJ+nhoNLzHYBtDTOioVLxU 1S2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=M+BwpB2+rvcbIhjbxICPZll+HSN+DS4Qkfxapk3UDNQ59pasUy/cKEk19icfJnrpj6 KnrYW1GXzhU3ppO+V7jMZINt84nzgWKxxW6S0OzNbn7ggkPA0bgXs3FIpb9Z6250BVIU 1QJVluF62rGd1gAKcUn1xSsLD6cD4epCqSXBlEcH7OVsYzBdt4u8BfGggtO3ddi0aKjf 6AHZVftsDFRl1OPfghAJUgHvtQOVbt4bUGAVDuEBCfjYeD73TSGxgr3rw1iEQmppQ8mp 2gxQIbP1+MJWW3/T2W5uDn7uZK5JOsxtTSWunfiU+TbKL9BvvYNjmQiO3yFMgkgb7aw9 xCxw== X-Gm-Message-State: AOAM530lXYa3PN4Ny7U7eYX4bBH7iJxV7BQsgeKc5/7X/41/2ak3j8kN z2SMX3Fgml9EJQOYt7155UEEK9R1izA= X-Google-Smtp-Source: ABdhPJxLahBXJD9USo9eQCchUaAXs2kghFaP9AvOhQwKWHFujSg9DxGZ3a5F7/Iz7d4QFqSzy8CH/g== X-Received: by 2002:adf:8003:0:b0:1f1:d81b:d752 with SMTP id 3-20020adf8003000000b001f1d81bd752mr23275906wrk.429.1647973400144; Tue, 22 Mar 2022 11:23:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u11-20020a056000038b00b00203e5c9aa09sm22556221wrf.27.2022.03.22.11.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:18 -0700 (PDT) Message-Id: <1d2877efda0fce9103598ad92c2cf7d618bb72a0.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:40 +0000 Subject: [PATCH v3 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++ config.mak.uname | 1 + contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 13 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-darwin.c diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index cf224768ad6..cf911d141f2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin + FSMONITOR_OS_SETTINGS = darwin endif endif diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 16705da2000..b8f9f7a0388 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) endif() endif() From patchwork Tue Mar 22 18:22:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0884C433EF for ; Tue, 22 Mar 2022 18:24:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240315AbiCVSZ5 (ORCPT ); Tue, 22 Mar 2022 14:25:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240305AbiCVSY5 (ORCPT ); Tue, 22 Mar 2022 14:24:57 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 852E38F9A9 for ; Tue, 22 Mar 2022 11:23:24 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id t11so24662121wrm.5 for ; Tue, 22 Mar 2022 11:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=k5ZREbqCqHvt+JSRQlVPZDSjI/JECTPQcuz7v45GQYU=; b=Z6O6+2J/GKtKKGGgNcdeBbf3VwknsQWq/limCUvp7M3y9JRKWaiiaYkt3m9bgKUQli iqm5LYz0jhu8p1ZPKTO5OByP0TrE8q8ezg5O7k/GzNbk5HtDCB9nhqupcE64+D4U3Kst ua9nttNnkskiDZ2jjxfFYyCT20BQ3Auv1DEKrBugZoJajydwy1GFyMkGoHcZfC/g3PLV a2ZOEi/4gqbgyN2q7vxq9HIhfaREzVYLxasedOTx+WYgZX9EpxbrVZ5ksJx3UZlpC55D ep/YG+KeCLf7iMH4b1cNsQB9WaN7WNUSjOP2eZNUy/L2hOATkQHihbwLLe3fbQ4SotXA 2KKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=k5ZREbqCqHvt+JSRQlVPZDSjI/JECTPQcuz7v45GQYU=; b=WOgxZVU9l0gpqvyUEUX52Cd2CVYbUyaLpcAe6iO2zSuBUFULuLIVOGMlPFuiokSyBc YvTYXJp6+mH5uzw0PG1ldmqbnFBhNndubgHb20TkdPBMSQjHUuf4Zv2xYIeSFH0J29DC /cU/HU8baEh6W585BKWdJW9cKxZTWGq8cmxX1sUtzpLltoft9dkhZ/QhsIlwZea0tQgm +NzO5oQTaKSusJBeqPFDhZFXzsplAB2uAF4aATq1itrERtfwnqrZx6IZCpIq39DwfSlG 1LWA+NiAvLQaDO5xmyig6uPTDUAAknnwuC7DP9gOIAdZrPuZWhVPhFTrYA1LEGs2wezU hO6w== X-Gm-Message-State: AOAM530k7VkjysBfwFbQ/CTWULpG54fQpDfR9krWtjn4uLkG3xA/I74S 6J3Vh3HJ8Drs1x7rdxW98Znhat1O7Zc= X-Google-Smtp-Source: ABdhPJxQJSHQZVqixzGKg8cI8j0mhl5IVYkVzWSECntr+Ft+SeNiglGwPwL0otrJUvr1oD8D3Fabsw== X-Received: by 2002:a5d:40ce:0:b0:203:f29f:7447 with SMTP id b14-20020a5d40ce000000b00203f29f7447mr19015087wrq.477.1647973403043; Tue, 22 Mar 2022 11:23:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k2-20020a1ca102000000b0038c78fdd59asm2389350wme.39.2022.03.22.11.23.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:21 -0700 (PDT) Message-Id: <06d7f18676d311b2ae246f078239739c5d1aa787.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:41 +0000 Subject: [PATCH v3 08/27] fsmonitor-settings: remote repos on macOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on macOS and mark them as incompatible with FSMonitor. With this, `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, like `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++ fsmonitor-settings.c | 10 ++++ fsmonitor-settings.h | 2 + 3 files changed, 78 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index 7fce32a3c5b..fdd762bf79d 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -2,8 +2,74 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" +#include +#include + +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type (NFS, SAMBA, etc.) dictates whether notification events + * are available at all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * For the builtin FSMonitor, we create the Unix domain socket for the + * IPC in the .git directory. If the working directory is remote, + * then the socket will be created on the remote file system. This + * can fail if the remote file system does not support UDS file types + * (e.g. smbfs to a Windows server) or if the remote kernel does not + * allow a non-local process to bind() the socket. (These problems + * could be fixed by moving the UDS out of the .git directory and to a + * well-known local directory on the client machine, but care should + * be taken to ensure that $HOME is actually local and not a managed + * file share.) + * + * So (for now at least), mark remote working directories as + * incompatible. + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + struct statfs fs; + + if (statfs(r->worktree, &fs) == -1) { + int saved_errno = errno; + trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s", + r->worktree, strerror(saved_errno)); + errno = saved_errno; + return FSMONITOR_REASON_ERROR; + } + + trace_printf_key(&trace_fsmonitor, + "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'", + r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename); + + if (!(fs.f_flags & MNT_LOCAL)) + return FSMONITOR_REASON_REMOTE; + + return FSMONITOR_REASON_OK; +} enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 1efb6e17a20..0a1811ff004 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -180,6 +180,16 @@ int fsm_settings__error_if_incompatible(struct repository *r) xgetcwd()); return 1; + case FSMONITOR_REASON_ERROR: + error(_("repository '%s' is incompatible with fsmonitor due to errors"), + r->worktree); + return 1; + + case FSMONITOR_REASON_REMOTE: + error(_("remote repository '%s' is incompatible with fsmonitor"), + r->worktree); + return 1; + case FSMONITOR_REASON_VFS4GIT: error(_("virtual repository '%s' is incompatible with fsmonitor"), r->worktree); diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 6361fcbf6b0..34391b583b3 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -16,6 +16,8 @@ enum fsmonitor_mode { enum fsmonitor_reason { FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ + FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; From patchwork Tue Mar 22 18:22:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED8B4C433FE for ; Tue, 22 Mar 2022 18:24:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240280AbiCVSZ0 (ORCPT ); Tue, 22 Mar 2022 14:25:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240304AbiCVSY5 (ORCPT ); Tue, 22 Mar 2022 14:24:57 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C697FBE7 for ; Tue, 22 Mar 2022 11:23:27 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id q8so13275767wrc.0 for ; Tue, 22 Mar 2022 11:23:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=ToRrZPXLHB2+JwCRtvNF9XR1IlnsN+Zzil6FSgZpyyXVqQ8ZVDf1y6WGVIv8Iliibr LHaJuFV+3S7sQN/wv+6JNlLtz8QFk/GIe7sYiIdyAbKdusbp8W8N3EUPOqPEBs8ad81I aAymKXlm52dlAdgxFl0nOHCbG210+pSoozuscr5UVgIF5Pwbge5TpFLdLZbG7gwoYosS RZ9EaHsl1atkG+F5Q5A6OVre/fP5nZzXLCx4TTbLTHjPiRVPzMth0vB9Q9Ag1+EKUyDB CHClV2afYxXm50HlEZ2n10Gon7BsXmnHnkO7X2xJyZQypJqPnLLqi7OyCrE8Ze97+nv2 oZzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=WCK4I6q16bPkGTIa47XwdkD7BCeIChi73FPS/u+Oalm7iTmoA0bTaUZz17Lh1eBOvV MPa9W1RXLBWDYm7IfsKhzDNmBCU3hvtbWMgu5t6ulL5LRWVUOs5obUspIzZXlLzxSmbT mHcVMBDt6QOY5wRl2pwNsteXYxTFAC3eRqTtowTfd4kcQh6Fp2hfkVeLCQy6hl2KbWyA iMkflfL7wHl1dcVUiE8ZTk/DyCJ/oQiN6Hr12tQ8lg14KNsMdNpPWllH+LW8kFk6FKiV Ln7pIueMCWQ9fDao0Ptv+ARMAM40J+609p5bIsXsh52YTNeSDmpRwK7esRuMdammvejz OHrw== X-Gm-Message-State: AOAM530zC8s4CtNxFWOZUhU04hunnLAl4I0Li1N1GBnr7wq4yQ1p4aqB MNaVQUgKcTEle05ztyDK86Np3YWEmXo= X-Google-Smtp-Source: ABdhPJxesXx4NO7CHlkZufC7rfEUQ1kcJg6s8OTkSa34PymjJm3P7qCIZCft/e+cKKol88MxU//GYQ== X-Received: by 2002:a5d:6da7:0:b0:205:7ccc:a001 with SMTP id u7-20020a5d6da7000000b002057ccca001mr2793540wrs.389.1647973405626; Tue, 22 Mar 2022 11:23:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r14-20020a7bc08e000000b0038ca55f9bcasm2313799wmh.42.2022.03.22.11.23.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:24 -0700 (PDT) Message-Id: <5ca97f482d06d61ff912a9b509e4c2884f01c8b9.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:42 +0000 Subject: [PATCH v3 09/27] fsmonitor-settings: remote repos on Windows are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on Windows and mark them as incompatible with FSMonitor. With this `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, such as `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index ee78bba38e3..907655720bb 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -2,6 +2,7 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" /* * VFS for Git is incompatible with FSMonitor. @@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r) return FSMONITOR_REASON_OK; } +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type dictates whether notification events are available at + * all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * So (for now at least), mark remote working directories as + * incompatible. + * + * Notes for testing: + * + * (a) Windows allows a network share to be mapped to a drive letter. + * (This is the normal method to access it.) + * + * $ NET USE Z: \\server\share + * $ git -C Z:/repo status + * + * (b) Windows allows a network share to be referenced WITHOUT mapping + * it to drive letter. + * + * $ NET USE \\server\share\dir + * $ git -C //server/share/repo status + * + * (c) Windows allows "SUBST" to create a fake drive mapping to an + * arbitrary path (which may be remote) + * + * $ SUBST Q: Z:\repo + * $ git -C Q:/ status + * + * (d) Windows allows a directory symlink to be created on a local + * file system that points to a remote repo. + * + * $ mklink /d ./link //server/share/repo + * $ git -C ./link status + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + wchar_t wpath[MAX_PATH]; + wchar_t wfullpath[MAX_PATH]; + size_t wlen; + UINT driveType; + + /* + * Do everything in wide chars because the drive letter might be + * a multi-byte sequence. See win32_has_dos_drive_prefix(). + */ + if (xutftowcs_path(wpath, r->worktree) < 0) + return FSMONITOR_REASON_ERROR; + + /* + * GetDriveTypeW() requires a final slash. We assume that the + * worktree pathname points to an actual directory. + */ + wlen = wcslen(wpath); + if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') { + wpath[wlen++] = L'\\'; + wpath[wlen] = 0; + } + + /* + * Normalize the path. If nothing else, this converts forward + * slashes to backslashes. This is essential to get GetDriveTypeW() + * correctly handle some UNC "\\server\share\..." paths. + */ + if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL)) + return FSMONITOR_REASON_ERROR; + + driveType = GetDriveTypeW(wfullpath); + trace_printf_key(&trace_fsmonitor, + "DriveType '%s' L'%ls' (%u)", + r->worktree, wfullpath, driveType); + + if (driveType == DRIVE_REMOTE) { + trace_printf_key(&trace_fsmonitor, + "check_remote('%s') true", + r->worktree); + return FSMONITOR_REASON_REMOTE; + } + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; @@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) if (reason != FSMONITOR_REASON_OK) return reason; + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } From patchwork Tue Mar 22 18:22:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFED3C433F5 for ; Tue, 22 Mar 2022 18:24:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240312AbiCVSZz (ORCPT ); Tue, 22 Mar 2022 14:25:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240311AbiCVSY7 (ORCPT ); Tue, 22 Mar 2022 14:24:59 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F88F15FEC for ; Tue, 22 Mar 2022 11:23:29 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id h23so24021632wrb.8 for ; Tue, 22 Mar 2022 11:23:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KKDUtqfcz4RAKt3TKC0Ge43bGODedopwH2p3XKXOALk=; b=drdUne9vXfWX3YoU+6pCEAj2TYWwqZYPHGbc4I/QN8IPD3cyh2MAD+tHSanaLdMxYV OPc9st1XBeJBJksVqAGhwgHmZLPFykVO/cHdnI/KWK+Y2muJCluCSgf3eSA0msjzx3K9 ne7R8ZOQSJxidH7L6v5JRRnvBKhTQzdTTZ3uGXX5q/gLgJMyUWwQLMigNR69LHVTTor1 cYVh32mlb941XvOrKO7WdA9g64NPw+Absu/7dyilElqOwIJuvBn49mMzFa4VVxbFVRxt QhGI1vK3ho42+3s8nepLDTODb8oSAy6X+9P0w+uUdgG2O+7qUc61Vd5eCPcxnYJrxKwW 9cEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KKDUtqfcz4RAKt3TKC0Ge43bGODedopwH2p3XKXOALk=; b=L8HhMA73ybunwzDFLrzC7uZmO5gi61eH22cJLaG5aiQLUKW+7DCsNqn4051mE8/r12 xZH0RzuahzVB93f3FzT6pwZvRK1yngEwUxe9dEC5YLB1tISeQmWOqVO+asbhm+xlOTXB /Zm5aYcY1O8z9Y/f+AUs52ftdccQz28eL9R156mQKbGwbhpDnbZTODKbcRegqUcZ811T +9rCLPf/Dz4kxR2vD9ByJVTEkCwF0Q26gHbh30awaJVt8AGwXf8XWJTViT0fT39IHFME sFROO+g+NJRwNHGIukrgwovLzfLPperPOTdbk0iYd/PwgJGy2ocLsCFlRk90IEae6kFs 2DgQ== X-Gm-Message-State: AOAM5319hviKFQYUwC8fbRuatdL+0StsODRdL6Q0yil+NgWQA+ek0Mj5 bLPfGect1gbyxqgP/CRuYZMHwx7WvRs= X-Google-Smtp-Source: ABdhPJzG/eTYX76qdtuHwVUiAX7SI/d3OFh4fKJW76D22sO3PbaMmifnJe0coHfqEKXf+DSVcF/56g== X-Received: by 2002:adf:d4c2:0:b0:203:da50:12c5 with SMTP id w2-20020adfd4c2000000b00203da5012c5mr24026732wrk.100.1647973408155; Tue, 22 Mar 2022 11:23:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h188-20020a1c21c5000000b0038c6c37efc3sm2374468wmh.12.2022.03.22.11.23.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:27 -0700 (PDT) Message-Id: <6715143724586bd53de5070c61dba30dd8fe2bf8.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:43 +0000 Subject: [PATCH v3 10/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler On MacOS mark repos on NTFS or FAT32 volumes as incompatible. The builtin FSMonitor used Unix domain sockets on MacOS for IPC with clients. These sockets are kept in the .git directory. Unix sockets are not supported by NTFS and FAT32, so the daemon cannot start up. Test for this during our compatibility checking so that client commands do not keep trying to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++--- fsmonitor-settings.c | 5 +++++ fsmonitor-settings.h | 1 + 3 files changed, 23 insertions(+), 3 deletions(-) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index fdd762bf79d..efc732c0f31 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -7,7 +7,7 @@ #include /* - * Remote working directories are problematic for FSMonitor. + * [1] Remote working directories are problematic for FSMonitor. * * The underlying file system on the server machine and/or the remote * mount type (NFS, SAMBA, etc.) dictates whether notification events @@ -40,8 +40,16 @@ * * So (for now at least), mark remote working directories as * incompatible. + * + * + * [2] FAT32 and NTFS working directories are problematic too. + * + * The builtin FSMonitor uses a Unix domain socket in the .git + * directory for IPC. These Windows drive formats do not support + * Unix domain sockets, so mark them as incompatible for the daemon. + * */ -static enum fsmonitor_reason check_remote(struct repository *r) +static enum fsmonitor_reason check_volume(struct repository *r) { struct statfs fs; @@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r) if (!(fs.f_flags & MNT_LOCAL)) return FSMONITOR_REASON_REMOTE; + if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */ + return FSMONITOR_REASON_NOSOCKETS; + + if (!strcmp(fs.f_fstypename, "ntfs")) + return FSMONITOR_REASON_NOSOCKETS; + return FSMONITOR_REASON_OK; } @@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; - reason = check_remote(r); + reason = check_volume(r); if (reason != FSMONITOR_REASON_OK) return reason; diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 0a1811ff004..60d5eaee497 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -194,6 +194,11 @@ int fsm_settings__error_if_incompatible(struct repository *r) error(_("virtual repository '%s' is incompatible with fsmonitor"), r->worktree); return 1; + + case FSMONITOR_REASON_NOSOCKETS: + error(_("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"), + r->worktree); + return 1; } BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 34391b583b3..23d5676c8c8 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -19,6 +19,7 @@ enum fsmonitor_reason { FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ + FSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */ }; void fsm_settings__set_ipc(struct repository *r); From patchwork Tue Mar 22 18:22:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 383FCC433F5 for ; Tue, 22 Mar 2022 18:24:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240327AbiCVSZ7 (ORCPT ); Tue, 22 Mar 2022 14:25:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240322AbiCVSZS (ORCPT ); Tue, 22 Mar 2022 14:25:18 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B30428D6B0 for ; Tue, 22 Mar 2022 11:23:32 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id u16so23630301wru.4 for ; Tue, 22 Mar 2022 11:23:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=frAgk+GnUmXFwY7nudp4DgXTjAEOmpIoQGmvYysxzYlRMMD5kt6AJdVaYoPDyUYwVF Dr/fiopz/kzJwarL0rfoEhrw+9rfQYymc8dJebSNktrUB0xXsSRIUlvrWNLcydYDj6XQ b5ocULpCKMPSlRb8IqQE49JOIHzooIofC/O1LuDnSwscR66Mh8kPNDbnN/a+/c6GGsA2 vHt2N6FDyDQXaFwxXh8RE5zzpRBWRoavf6fvvAs89yQfSqkOZ7q4ZFpWY4z3fkGmY26c iC30BA3Eee3a0sp+igkILoa/+v5+Gcl4hQR036zbWhaWbXKIV1ho0qO7GQN3mZV3rWta vtVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=SPT3MlTCNu5i/4qhN1sG42YJgG7+6XRW+MqNoeMh9BeMh6CUVMt1TrJmlDpJOm0jsn pSejQDeR/n3FLRV2Th41UEyw1nWQrjNOJ4t0GQZ7JOV2dm35bRDEmPTyMaT+7Ve12dk5 +s/4XKbD5b7a5FnJRCpphZhFP93BCI1HuYfTp1y0EtvrrGJtru1s7VbXHlYbKtAUafe/ w/AUzma3VWg8cozNia0vlV3fEtwBDUO8T2PbVzY/ZBe9d3aB0EPmNOt3orv9CnAkP7nC Vcwovr5q0OHcsel907Qn9bslX2lk6FMvqCrkxt1UIk49azvG+WfGSyIcfnEf81DCqku2 376g== X-Gm-Message-State: AOAM532bOdfYmgDIDJP6/eZG+U8G99IP4hxaCsS5yCsGYFgGW+vn01U5 C5/Az93O4moOIYaukJWzfLEUNRIXWJ0= X-Google-Smtp-Source: ABdhPJz/ANozmo2eopnJLVLPp5HS59+wwF8ETgyAHtHU5xkdHnsdBglwb2NSWW/O4t8yOZu/qU060A== X-Received: by 2002:adf:f102:0:b0:205:7d05:d250 with SMTP id r2-20020adff102000000b002057d05d250mr2750857wro.350.1647973411000; Tue, 22 Mar 2022 11:23:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm2254437wmb.22.2022.03.22.11.23.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:29 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:44 +0000 Subject: [PATCH v3 11/27] unpack-trees: initialize fsmonitor_has_run_once in o->result Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Initialize `o->result.fsmonitor_has_run_once` based upon value in `o->src_index->fsmonitor_has_run_once` to prevent a second fsmonitor query during the tree traversal and possibly getting a skewed view of the working directory. The checkout code has already talked to the fsmonitor and the traversal is updating the index as it traverses, so there is no need to query the fsmonitor. Signed-off-by: Jeff Hostetler --- unpack-trees.c | 1 + 1 file changed, 1 insertion(+) diff --git a/unpack-trees.c b/unpack-trees.c index 360844bda3a..888cff81f9c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options o->result.fsmonitor_last_update = xstrdup_or_null(o->src_index->fsmonitor_last_update); + o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once; /* * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries From patchwork Tue Mar 22 18:22:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DA43C4332F for ; Tue, 22 Mar 2022 18:24:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240321AbiCVSZ5 (ORCPT ); Tue, 22 Mar 2022 14:25:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240323AbiCVSZS (ORCPT ); Tue, 22 Mar 2022 14:25:18 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C8BB8F9AC for ; Tue, 22 Mar 2022 11:23:34 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id v2-20020a7bcb42000000b0037b9d960079so3863566wmj.0 for ; Tue, 22 Mar 2022 11:23:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5W2x8t68NoDAVIFeMr0BVtQdXxzEzVw8wanSKIA/+K4=; b=euXWwXv4Huay4kBJIaraV0X9wTjhVbD8VV+wBUhUau7lNcl5PXYzpC2vbqX7OksqZB WkoCILPUoQ6S0k+jgdZQ77q6dyhtCHT/O3T5A3p1Gwx+7kGcw4RPx19eY9gZGY34rVhE 4ymuc5yWs8fwEC9Cq3J4KzKktyr+YYJkDSGjNLaLrXTuL39jzGzajpmteKD9rfybRAz7 5XjuZYm36Ld1H+mmBjPFMZITwDbr54Vv14k+OQv+4NoboLfVqZd9BuS5AjYK8PMqaOgH gd9Z6eTkmhrJuunqb6udVk/6dw93KG5hnL49mZFjBmwricoW+wjqVPCDVW15HNJJqucC eH/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5W2x8t68NoDAVIFeMr0BVtQdXxzEzVw8wanSKIA/+K4=; b=hRV4Ig2bFPFjGZ9V9Geux+iYWLTBLOYbedJh9TmIcOHzqNsBx8kVzOvjMWp1GjqVI+ yLReW21iMxlrrDtd54h69mzq2Kq/STMnWOMxN2bWhr6lzZUIvspI7YGGLZvr7L40KQEG qYZtEXr/L7txrNN0R5e4magwRpQWBopzrpXxiIm0RTMtd0ztpFaOEYptQBUIWaQEuo/m HIibLKZzARcDvm510nOAWaX7MTcX+b1DFg7JSSj29svXkqXVHe4O+QOwElDd9aoaaxAO yf68nnMFsIGiAtBXhe8E2XYDOuhcB6XQvM2gdBvVAl4XvXMh2RcZxYzQx5ypGET9XnXl kIlw== X-Gm-Message-State: AOAM5320cbGPsS7OgJp6c4iVNbwHeGPyMPm1hgdU7yG3yaoKQ3pfEvRs tZtNGcP+OUTaHg78h+kSIMRj9aeYwSY= X-Google-Smtp-Source: ABdhPJzhWrcb8KuDTcjg3Nb1s2LF+qDxsvMpnXzJsXk79k61PxwafsPYVm3i/Q0cWFKCRq8iDspcgQ== X-Received: by 2002:a05:600c:35d4:b0:38c:9933:41c0 with SMTP id r20-20020a05600c35d400b0038c993341c0mr5071123wmq.149.1647973412767; Tue, 22 Mar 2022 11:23:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 10-20020a5d47aa000000b00204012e4373sm10444986wrb.101.2022.03.22.11.23.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:31 -0700 (PDT) Message-Id: <35c77b854bdb5ea3a339c9a57865d4d0debc4df5.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:45 +0000 Subject: [PATCH v3 12/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Ignore FSEvents resulting from `xattr` changes. Git does not care about xattr's or changes to xattr's, so don't waste time collecting these events in the daemon nor transmitting them to clients. Various security tools add xattrs to files and/or directories, such as to mark them as having been downloaded. We should ignore these events since it doesn't affect the content of the file/directory or the normal meta-data that Git cares about. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 0741fe834c3..14105f45c18 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) if (flag & kFSEventStreamEventFlagItemCloned) strbuf_addstr(&msg, "ItemCloned|"); - trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s", path, flag, msg.buf); strbuf_release(&msg); @@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef) ef & kFSEventStreamEventFlagUserDropped); } +/* + * If an `xattr` change is the only reason we received this event, + * then silently ignore it. Git doesn't care about xattr's. We + * have to be careful here because the kernel can combine multiple + * events for a single path. And because events always have certain + * bits set, such as `ItemIsFile` or `ItemIsDir`. + * + * Return 1 if we should ignore it. + */ +static int ef_ignore_xattr(const FSEventStreamEventFlags ef) +{ + static const FSEventStreamEventFlags mask = + kFSEventStreamEventFlagItemChangeOwner | + kFSEventStreamEventFlagItemCreated | + kFSEventStreamEventFlagItemFinderInfoMod | + kFSEventStreamEventFlagItemInodeMetaMod | + kFSEventStreamEventFlagItemModified | + kFSEventStreamEventFlagItemRemoved | + kFSEventStreamEventFlagItemRenamed | + kFSEventStreamEventFlagItemXattrMod | + kFSEventStreamEventFlagItemCloned; + + return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); +} + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_ignore_xattr(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "ignore-xattr: '%s', flags=0x%x", + path_k, event_flags[k]); + continue; + } + switch (fsmonitor_classify_path_absolute(state, path_k)) { case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: From patchwork Tue Mar 22 18:22:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19F56C433FE for ; Tue, 22 Mar 2022 18:24:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240348AbiCVS0B (ORCPT ); Tue, 22 Mar 2022 14:26:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240365AbiCVSZU (ORCPT ); Tue, 22 Mar 2022 14:25:20 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E1E890FCB for ; Tue, 22 Mar 2022 11:23:37 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id p184-20020a1c29c1000000b0037f76d8b484so2472651wmp.5 for ; Tue, 22 Mar 2022 11:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Z8Ge1p0oYcNFMDPX3A37UzYPM/95/xrV3EtYMnhaxr8=; b=IM9j0btIkHRn95l8Dh8bP4g+yZsu3bi0dmvI8nmD6myindlNMaQsoLfbcyodUOb5rM Yw5AJP+iBhLfEbpxlF7aUvr2BrqSJpuzfdEnjm5P1d06JOu11RVTkwZrHmVU8548b5IF uLCoDH59JVrebj43ySBfbN9fSIiMNY8pj37iWVPGzoIv69mYhjdMwMis6fp4LzhPC3iZ p4vuKCUzMuCusdsieI42TkeC3Efehq2ti0zTdVp2vfFRZ8QSpO7Jg3tSsf+A7IlV0P3l dndxsMULwL50KFkvx4+AIr7q/mgJiQr8YDMfPYFJzZJyPU4VM+kMuo9sDGqJaDwlv5Gc id1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Z8Ge1p0oYcNFMDPX3A37UzYPM/95/xrV3EtYMnhaxr8=; b=CMBmdGSjDI9sTZek/6gvbhAzBruC+FsaYN3IsaEv2rDOQv/IijnDZmfXgR6ueM+cf/ 41FIbqWF7yF+sP+UdLehs0mqYWWGJTyDAwN48h4IsxLRm3Q/H1dw3BvqteYoUOPV6Et3 NlPGvK7oehHQSzoEJhJnGBUN8PUpD/cToIYPkbFncSCc4XYBndAFq2AIqzM/6Xb05Ne+ mE7VcuZmMsTvtPov4m3iMPwynfalT6wvifuULkkkt1YxUQorsXBDRLocvkyl8RDZRmBg fF6odagJhCHRImhlJuW2a2tUHHL7IuwMz+NGpCbuqGUkMlzNiGFEphAUjWWt1HMfug79 k7/g== X-Gm-Message-State: AOAM5307ku+kk4N6dozXucRs6YUoCtJ+dVdCewwQIGJ0C0coy0WPlrod VZIw9UDHuNGZyI74ShUDHb81B1hvb9s= X-Google-Smtp-Source: ABdhPJyGKU3NDDXkv8i7FAMcgzPTWipXjom4Bc0Mp1g7zNH/gPvRN+v3kKq2AgnKND5Qg79reDBEPA== X-Received: by 2002:a05:600c:2146:b0:38c:ab68:1919 with SMTP id v6-20020a05600c214600b0038cab681919mr5239173wml.52.1647973415405; Tue, 22 Mar 2022 11:23:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d15-20020a056000186f00b0020384249361sm18633662wri.104.2022.03.22.11.23.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:34 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:46 +0000 Subject: [PATCH v3 13/27] fsmonitor--daemon: cd out of worktree root Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the fsmonitor--daemon to CD outside of the worktree before starting up. The common Git startup mechanism causes the CWD of the daemon process to be in the root of the worktree. On Windows, this causes the daemon process to hold a locked handle on the CWD and prevents other processes from moving or deleting the worktree while the daemon is running. CD to HOME before entering main event loops. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++-- compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------ fsmonitor--daemon.h | 1 + 3 files changed, 47 insertions(+), 8 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index edca8a667ab..f17d5ed84c8 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1182,11 +1182,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * before we need it. */ if (ipc_server_run_async(&state->ipc_server_data, - fsmonitor_ipc__get_path(), &ipc_opts, + state->path_ipc.buf, &ipc_opts, handle_client, state)) return error_errno( _("could not start IPC thread pool on '%s'"), - fsmonitor_ipc__get_path()); + state->path_ipc.buf); /* * Start the fsmonitor listener thread to collect filesystem @@ -1221,6 +1221,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) static int fsmonitor_run_daemon(void) { struct fsmonitor_daemon_state state; + const char *home; int err; memset(&state, 0, sizeof(state)); @@ -1290,6 +1291,15 @@ static int fsmonitor_run_daemon(void) strbuf_addch(&state.path_cookie_prefix, '/'); + /* + * We create a named-pipe or unix domain socket inside of the + * ".git" directory. (Well, on Windows, we base our named + * pipe in the NPFS on the absolute path of the git + * directory.) + */ + strbuf_init(&state.path_ipc, 0); + strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path())); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1299,6 +1309,23 @@ static int fsmonitor_run_daemon(void) goto done; } + /* + * CD out of the worktree root directory. + * + * The common Git startup mechanism causes our CWD to be the + * root of the worktree. On Windows, this causes our process + * to hold a locked handle on the CWD. This prevents the + * worktree from being moved or deleted while the daemon is + * running. + * + * We assume that our FS and IPC listener threads have either + * opened all of the handles that they need or will do + * everything using absolute paths. + */ + home = getenv("HOME"); + if (home && *home && chdir(home)) + die_errno(_("could not cd home '%s'"), home); + err = fsmonitor_run_daemon_1(&state); done: @@ -1311,6 +1338,7 @@ done: strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); strbuf_release(&state.path_cookie_prefix); + strbuf_release(&state.path_ipc); return err; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 3f1b68267bd..c43d92b9620 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch) } /* - * NEEDSWORK: If an external is deleted, the above - * returns an error. I'm not sure that there's anything that - * we can do here other than failing -- the /.git - * link file would be broken anyway. We might try to check - * for that and return a better error message, but I'm not - * sure it is worth it. + * GetOverlappedResult() fails if the watched directory is + * deleted while we were waiting for an overlapped IO to + * complete. The documentation did not list specific errors, + * but I observed ERROR_ACCESS_DENIED (0x05) errors during + * testing. + * + * Note that we only get notificaiton events for events + * *within* the directory, not *on* the directory itself. + * (These might be properies of the parent directory, for + * example). + * + * NEEDSWORK: We might try to check for the deleted directory + * case and return a better error message, but I'm not sure it + * is worth it. + * + * Shutdown if we get any error. */ error(_("GetOverlappedResult failed on '%s' [GLE %ld]"), diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index bd09fffc176..223c2131b58 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -54,6 +54,7 @@ struct fsmonitor_daemon_state { struct fsmonitor_daemon_backend_data *backend_data; struct ipc_server_data *ipc_server_data; + struct strbuf path_ipc; }; /* From patchwork Tue Mar 22 18:22:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCE7CC433F5 for ; Tue, 22 Mar 2022 18:24:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240401AbiCVSZn (ORCPT ); Tue, 22 Mar 2022 14:25:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240348AbiCVSZT (ORCPT ); Tue, 22 Mar 2022 14:25:19 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA45390CC8 for ; Tue, 22 Mar 2022 11:23:38 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id d7so24639141wrb.7 for ; Tue, 22 Mar 2022 11:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Ej7vjoBKufi5X7KtDia+Ocl0y26c4e3iM4lYXqh6QBQ=; b=L6t+uIHS4oh2vUxOlTRAJ6B6oPAJyoOhEykrUguDAduXQq0ramvHeYP5NOJ3EX+zKZ XV5RBhiA2YZCc4jAvGK+QECs9GYQvzaJTIZTxyCWtW+LD3yXEmVX85WndSasKb0a6qCy WVJhY6HoOwtVGJit/EWyXvhrdUTBd2O7i8oeHJ9SGtXBUOyMbH6NjdcU5lbA/P4FBXm+ 7lox50YVaVBaUudGZehxvaRH8zKVpHtvQ//Li5HouLiaTq3NJXTz0G7fxM62FqFzK9nF IbqB7N0fqsoTT2cASBhf0ItIjdUbXi2u+jB0pIk/qny2c8iaSz+2S1Tnwea0gyEMJm4d z3Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Ej7vjoBKufi5X7KtDia+Ocl0y26c4e3iM4lYXqh6QBQ=; b=g4yP+YjhGthud1rUKvFzHjE8qkUwqojvBTKVKh8Xgb9mKKaKJ+f541zXzManhqVJB8 Eod3sclCYRv7KEeYZEjHcS+LHTg0SXTn0y0a1F4jZFvxG63V6wv93oXqUsfp0MQQUH/Y FcxkT4wgsu/9ozkAUWBNoycJeVIGClkzfE7rQnReQQPUutmdX6YVYgP9w5AHAW4u5JeZ 0RLT0B/aNBHUMrf4fAiaGCmyTiwxOfus00nCxGhnFS+YTY+BEoRUfvP/ihG2peuc+cxV pJjd5JMC4xWVMtINxFIbgKJULHBIRkT7OieZ2qTzYU/YW1zWPqqkeCOcj6+/8yCoP2KN jvcA== X-Gm-Message-State: AOAM530S0d6hHf3KiaZopFUg3rJwAkpH7mVJdhpuP5yvtW7Y2gYi6/0h tgBmofQLP13/ixOCa7e2x+ijSUPOqho= X-Google-Smtp-Source: ABdhPJzydzQnz1Fmvo38GENgZHYMADUekqpF48OZ13FdM02n7MBdccYidAkXogq2PEJFpUlCfCVxPA== X-Received: by 2002:a5d:5846:0:b0:204:1a79:f1ab with SMTP id i6-20020a5d5846000000b002041a79f1abmr7055805wrf.520.1647973417276; Tue, 22 Mar 2022 11:23:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u12-20020a5d6dac000000b00204119d37d0sm5931143wrs.26.2022.03.22.11.23.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:36 -0700 (PDT) Message-Id: <087af5dfb6315849b0079d8f6df9bc2da7682653.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:47 +0000 Subject: [PATCH v3 14/27] fsmonitor--daemon: prepare for adding health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Refactor daemon thread startup to make it easier to start a third thread class to monitor the health of the daemon. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index f17d5ed84c8..8033916aa63 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1175,6 +1175,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int listener_started = 0; + int err = 0; /* * Start the IPC thread pool before the we've started the file @@ -1195,15 +1197,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) if (pthread_create(&state->listener_thread, NULL, fsm_listen__thread_proc, state) < 0) { ipc_server_stop_async(state->ipc_server_data); - ipc_server_await(state->ipc_server_data); - - return error(_("could not start fsmonitor listener thread")); + err = error(_("could not start fsmonitor listener thread")); + goto cleanup; } + listener_started = 1; /* * The daemon is now fully functional in background threads. + * Our primary thread should now just wait while the threads + * do all the work. + */ +cleanup: + /* * Wait for the IPC thread pool to shutdown (whether by client - * request or from filesystem activity). + * request, from filesystem activity, or an error). */ ipc_server_await(state->ipc_server_data); @@ -1212,10 +1219,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * event from the IPC thread pool, but it doesn't hurt to tell * it again. And wait for it to shutdown. */ - fsm_listen__stop_async(state); - pthread_join(state->listener_thread, NULL); + if (listener_started) { + fsm_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + } - return state->error_code; + if (err) + return err; + if (state->error_code) + return state->error_code; + return 0; } static int fsmonitor_run_daemon(void) From patchwork Tue Mar 22 18:22:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFBE6C433EF for ; Tue, 22 Mar 2022 18:24:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240399AbiCVSZk (ORCPT ); Tue, 22 Mar 2022 14:25:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240386AbiCVSZV (ORCPT ); Tue, 22 Mar 2022 14:25:21 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6D5F90FF3 for ; Tue, 22 Mar 2022 11:23:40 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id u16so23630781wru.4 for ; Tue, 22 Mar 2022 11:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Wtvb5FDBX3WUfcTKnEkmdS/WLRWMd9ydoFseLbjp70o=; b=j1qMyXHMMcfyPAI+ZZvU6uF+XvZ2LGPGwb1xcSepaD8oRGNn8PULad1dFMh8aO+Yb1 hbdsrr+lAl1KqhR4BH6hRt6TAqWC7ziCOJegq4ZqWaDixwZtXg9dPb7/QPrFRG1MWEqf ypW/XOImXEMVyS+lkgXiayNmLaYtNASGrRHFeJC6+RDHSF/4n8riDyE6WcfClsk+fGw/ r5wUWXA0OBugbywibfLwYpd1xzbROA1ldC4q7QhidpJyJzBRzcdVghFIvFdGlXt9Qb1h OUEL6JQZEqay0m16Yd2QXHOQbEXDXvhkhxhNIq+In3lF2f2MCZybaont6aF75DxfLTQ5 Lfng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Wtvb5FDBX3WUfcTKnEkmdS/WLRWMd9ydoFseLbjp70o=; b=NsBewe27u5dnN0wFmPM3xIOzqH6ohUuJ2/vEnbnAuhe+U0L+Rs+h3WynN7lhUgj6cM HnxyghEDikRLgTF6OMyvdKZcvs50+Oy+DFTQdxwYuBylqy96xLbo6I8pzGNa0FsxQdJ0 4A31q1TL4zSCjsWcYdicHogZOd5mEjS+cRTocpjFutvCD8cvG8KHBwoiUI3ScromLKnz 8ifT2HQtS9FUn6pDsSTtTiHhD9Z+n2LVJCh99GVXSGssZVoLbUdI1aCjJCfp0i5wdgp4 gecgz1V2n09o/3flG78xZwKvhPAZjmkl5bC72GAd5z4pI34Rdf9WuA3xs8oiahjcOo4p gx0A== X-Gm-Message-State: AOAM531mPqM9pcVCqP9ZwMLkNUoJ1kMrzhHF7UD9ykqikdOq3hRmyz06 ehSXY3SgS8J0kD5KoDG7bXEsCnYpqPw= X-Google-Smtp-Source: ABdhPJxEXZYXe50Eot8ToMqLgCRMAvNJmL1xy5Xeh/8kKLI0//RaZ8oLkYaj44Kpr3IKGQmmrBfvuA== X-Received: by 2002:adf:806e:0:b0:1e3:2a17:bd60 with SMTP id 101-20020adf806e000000b001e32a17bd60mr23053742wrk.719.1647973418816; Tue, 22 Mar 2022 11:23:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j5-20020a05600c1c0500b0038ca4fdf7a5sm3674870wms.9.2022.03.22.11.23.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:37 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:48 +0000 Subject: [PATCH v3 15/27] fsmonitor--daemon: rename listener thread related variables Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Rename platform-specific listener thread related variables and data types as we prepare to add another backend thread type. [] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data` [] `state->backend_data` becomes `state->listen_data` [] `state->error_code` becomes `state->listen_error_code` Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 6 +++--- compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++-------------- compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++------------- compat/fsmonitor/fsm-listen.h | 2 +- fsmonitor--daemon.h | 6 +++--- 5 files changed, 36 insertions(+), 36 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 8033916aa63..af91bc2fb4c 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1226,8 +1226,8 @@ cleanup: if (err) return err; - if (state->error_code) - return state->error_code; + if (state->listen_error_code) + return state->listen_error_code; return 0; } @@ -1242,7 +1242,7 @@ static int fsmonitor_run_daemon(void) hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); - state.error_code = 0; + state.listen_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 14105f45c18..07113205a61 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -27,7 +27,7 @@ #include "fsm-listen.h" #include "fsmonitor--daemon.h" -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { CFStringRef cfsr_worktree_path; CFStringRef cfsr_gitdir_path; @@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, const FSEventStreamEventId event_ids[]) { struct fsmonitor_daemon_state *state = ctx; - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; char **paths = (char **)event_paths; struct fsmonitor_batch *batch = NULL; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) NULL, NULL }; - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; const void *dir_array[2]; CALLOC_ARRAY(data, 1); - state->backend_data = data; + state->listen_data = data; data->cfsr_worktree_path = CFStringCreateWithCString( NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); @@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) failed: error(_("Unable to create FSEventStream.")); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); return -1; } void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; if (data->stream) { if (data->stream_started) @@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state) FSEventStreamRelease(data->stream); } - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->shutdown_style = SHUTDOWN_EVENT; CFRunLoopStop(data->rl); @@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->rl = CFRunLoopGetCurrent(); @@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) switch (data->shutdown_style) { case FORCE_ERROR_STOP: - state->error_code = -1; + state->listen_error_code = -1; /* fall thru */ case FORCE_SHUTDOWN: ipc_server_stop_async(state->ipc_server_data); @@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) return; force_error_stop_without_loop: - state->error_code = -1; + state->listen_error_code = -1; ipc_server_stop_async(state->ipc_server_data); return; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index c43d92b9620..be2d93f47b2 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -54,7 +54,7 @@ struct one_watch wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { struct one_watch *watch_worktree; struct one_watch *watch_gitdir; @@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname( void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); + SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]); } static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, @@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch) free(watch); } -static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, +static int start_rdcw_watch(struct fsm_listen_data *data, struct one_watch *watch) { DWORD dwNotifyFilter = @@ -516,7 +516,7 @@ static int process_1_worktree_event( */ static int process_worktree_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_worktree; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -646,7 +646,7 @@ force_shutdown: */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_gitdir; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -704,11 +704,11 @@ skip_this_path: void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; DWORD dwWait; int result; - state->error_code = 0; + state->listen_error_code = 0; if (start_rdcw_watch(data, data->watch_worktree) == -1) goto force_error_stop; @@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) } force_error_stop: - state->error_code = -1; + state->listen_error_code = -1; force_shutdown: /* @@ -790,7 +790,7 @@ clean_shutdown: int fsm_listen__ctor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; CALLOC_ARRAY(data, 1); @@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) data->nr_listener_handles++; } - state->backend_data = data; + state->listen_data = data; return 0; failed: @@ -836,16 +836,16 @@ failed: void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; CloseHandle(data->hEventShutdown); destroy_watch(data->watch_worktree); destroy_watch(data->watch_gitdir); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h index f0539349baf..41650bf8972 100644 --- a/compat/fsmonitor/fsm-listen.h +++ b/compat/fsmonitor/fsm-listen.h @@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state); * do so if the listener thread receives a normal shutdown signal from * the IPC layer.) * - * It should set `state->error_code` to -1 if the daemon should exit + * It should set `state->listen_error_code` to -1 if the daemon should exit * with an error. */ void fsm_listen__loop(struct fsmonitor_daemon_state *state); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 223c2131b58..2c6fa1a5d91 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); */ void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); -struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ +struct fsm_listen_data; /* opaque platform-specific data for listener thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; @@ -50,8 +50,8 @@ struct fsmonitor_daemon_state { int cookie_seq; struct hashmap cookies; - int error_code; - struct fsmonitor_daemon_backend_data *backend_data; + int listen_error_code; + struct fsm_listen_data *listen_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Tue Mar 22 18:22:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41D95C433EF for ; Tue, 22 Mar 2022 18:24:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240289AbiCVSZ1 (ORCPT ); Tue, 22 Mar 2022 14:25:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240395AbiCVSZV (ORCPT ); Tue, 22 Mar 2022 14:25:21 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48E2E91340 for ; Tue, 22 Mar 2022 11:23:43 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id q8so13276614wrc.0 for ; Tue, 22 Mar 2022 11:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=L54Ib7+Uo9Wm+Z01OHIvaDm39CF5zJeSZpCeR2BT2uM=; b=ixL9+5GxyH2x2mMQtAoyYl3sRiYuLXwyUcs7CGoBqGEG0Gfk/R/VnNM6ABf0ohoRZZ TKnN/apvVwc7pLVahk4R7D+fPui/Jcfcjy/fEpnQxvX2+6knYHwJoJgkTjm6I2dTUkRN CmJ23gb7Mp9uJIoCYpJlacK6+/GanvqB4ZQEolpNUljAgT7vMUNaqAexUQw/RBAeO/ko vPao5xTcPlD5o6xW5V3MWYylELE3O0t1bVKYlfemUwysduigaW/M36MumHAIb/IRLFzs HeOnE6e/GL1vJW+vQwSoFlj9+yImvHC6+U+4XZdY9N0+9/JTWp3auc0fzE1Q+WXjKSyr +OSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=L54Ib7+Uo9Wm+Z01OHIvaDm39CF5zJeSZpCeR2BT2uM=; b=76EVPoU65VBRoHuYAlWU7IZx1jb7cxdzWPrwbBfwnQj795bJR+dSpdrpglE0ZPO1Ly 9tGqCyFRs4HfrvvR7qW9lqkX/BGxSwL9oWxPP4Po5V/8wS94fredX2zK3ahD++N1bsle AmOJYDwCB7tuXnNY/0AU5yQAL1eZzN9MmInOnbQ2wbeXJ7p5XsEnojVCgxGTXe/seZzu b4igmLz/CKD5FRYPRCijOpS0f0Uq44FsWrk+ds4klvBGssSmkJO0lZHRKn3KrNCwLJRh TaXoazlcYsCdnWXD9eE8P9eRgKz1WYaIJPDR80lCh7PfeHg+SxDaXF8EWbVClGRxJcL6 LH9A== X-Gm-Message-State: AOAM530DzMc8mM3MXqyOu4zjcX2nWo9BEwC1oRPw3P1MtlemNzMF/gwN CEURK9pELKg0cuy3fLMW6IjvWSY+1f0= X-Google-Smtp-Source: ABdhPJyUvpQTAsXP/cFxANFljtwymGmFxkDBY8saHzPhMs9KLuV85bNm8hVsEbFxhWyFtJDJkRvLtQ== X-Received: by 2002:a05:6000:156f:b0:204:b75:b62d with SMTP id 15-20020a056000156f00b002040b75b62dmr10511517wrz.489.1647973421274; Tue, 22 Mar 2022 11:23:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l126-20020a1c2584000000b00387d4f35651sm2416818wml.10.2022.03.22.11.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:39 -0700 (PDT) Message-Id: <301fff5296a9b343dc1dd41b6a8c1eafc570141e.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:49 +0000 Subject: [PATCH v3 16/27] fsmonitor--daemon: stub in health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create another thread to watch over the daemon process and automatically shut it down if necessary. This commit creates the basic framework for a "health" thread to monitor the daemon and/or the file system. Later commits will add platform-specific code to do the actual work. The "health" thread is intended to monitor conditions that would be difficult to track inside the IPC thread pool and/or the file system listener threads. For example, when there are file system events outside of the watched worktree root or if we want to have an idle-timeout auto-shutdown feature. This commit creates the health thread itself, defines the thread-proc and sets up the thread's event loop. It integrates this new thread into the existing IPC and Listener thread models. This commit defines the API to the platform-specific code where all of the monitoring will actually happen. The platform-specific code for MacOS is just stubs. Meaning that the health thread will immediately exit on MacOS, but that is OK and expected. Future work can define MacOS-specific monitoring. The platform-specific code for Windows sets up enough of the WaitForMultipleObjects() machinery to watch for system and/or custom events. Currently, the set of wait handles only includes our custom shutdown event (sent from our other theads). Later commits in this series will extend the set of wait handles to monitor other conditions. Signed-off-by: Jeff Hostetler --- Makefile | 6 ++- builtin/fsmonitor--daemon.c | 39 +++++++++++++++ compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++ compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++ compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++ contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor--daemon.h | 4 ++ 7 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-darwin.c create mode 100644 compat/fsmonitor/fsm-health-win32.c create mode 100644 compat/fsmonitor/fsm-health.h diff --git a/Makefile b/Makefile index 93604fe8ef7..5f1623baadd 100644 --- a/Makefile +++ b/Makefile @@ -472,8 +472,9 @@ all:: # # If your platform supports a built-in fsmonitor backend, set # FSMONITOR_DAEMON_BACKEND to the "" of the corresponding -# `compat/fsmonitor/fsm-listen-.c` that implements the -# `fsm_listen__*()` routines. +# `compat/fsmonitor/fsm-listen-.c` and +# `compat/fsmonitor/fsm-health-.c` files +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines. # # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS @@ -1982,6 +1983,7 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index af91bc2fb4c..522904abf36 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,6 +3,7 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsm-health.h" #include "compat/fsmonitor/fsm-listen.h" #include "fsmonitor--daemon.h" #include "simple-ipc.h" @@ -1137,6 +1138,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, pthread_mutex_unlock(&state->main_lock); } +static void *fsm_health__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-health"); + + fsm_health__loop(state); + + trace2_thread_exit(); + return NULL; +} + static void *fsm_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -1175,6 +1188,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int health_started = 0; int listener_started = 0; int err = 0; @@ -1202,6 +1216,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) } listener_started = 1; + /* + * Start the health thread to watch over our process. + */ + if (pthread_create(&state->health_thread, NULL, + fsm_health__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + err = error(_("could not start fsmonitor health thread")); + goto cleanup; + } + health_started = 1; + /* * The daemon is now fully functional in background threads. * Our primary thread should now just wait while the threads @@ -1224,10 +1249,17 @@ cleanup: pthread_join(state->listener_thread, NULL); } + if (health_started) { + fsm_health__stop_async(state); + pthread_join(state->health_thread, NULL); + } + if (err) return err; if (state->listen_error_code) return state->listen_error_code; + if (state->health_error_code) + return state->health_error_code; return 0; } @@ -1243,6 +1275,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); state.listen_error_code = 0; + state.health_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ @@ -1322,6 +1355,11 @@ static int fsmonitor_run_daemon(void) goto done; } + if (fsm_health__ctor(&state)) { + err = error(_("could not initialize health thread")); + goto done; + } + /* * CD out of the worktree root directory. * @@ -1345,6 +1383,7 @@ done: pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsm_listen__dtor(&state); + fsm_health__dtor(&state); ipc_server_free(state.ipc_server_data); diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c new file mode 100644 index 00000000000..b9f709e8548 --- /dev/null +++ b/compat/fsmonitor/fsm-health-darwin.c @@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c new file mode 100644 index 00000000000..94b1d020f25 --- /dev/null +++ b/compat/fsmonitor/fsm-health-win32.c @@ -0,0 +1,72 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +struct fsm_health_data +{ + HANDLE hEventShutdown; + + HANDLE hHandles[1]; /* the array does not own these handles */ +#define HEALTH_SHUTDOWN 0 + int nr_handles; /* number of active event handles */ +}; + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown; + data->nr_handles++; + + state->health_data = data; + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + if (!state || !state->health_data) + return; + + data = state->health_data; + + CloseHandle(data->hEventShutdown); + + FREE_AND_NULL(state->health_data); +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data = state->health_data; + + for (;;) { + DWORD dwWait = WaitForMultipleObjects(data->nr_handles, + data->hHandles, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) + goto clean_shutdown; + + error(_("health thread wait failed [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->health_error_code = -1; + ipc_server_stop_async(state->ipc_server_data); +clean_shutdown: + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ + SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]); +} diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h new file mode 100644 index 00000000000..45547ba9380 --- /dev/null +++ b/compat/fsmonitor/fsm-health.h @@ -0,0 +1,47 @@ +#ifndef FSM_HEALTH_H +#define FSM_HEALTH_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor health thread. + * This will be called from the main thread PRIOR to staring the + * thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsm_health__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the health thread. + * This will be called from the main thread AFTER joining the thread. + */ +void fsm_health__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to monitor the + * health of the daemon process. This will run in the health thread. + * + * The health thread should call `ipc_server_stop_async()` if it needs + * to cause a shutdown. (It should NOT do so if it receives a shutdown + * shutdown signal.) + * + * It should set `state->health_error_code` to -1 if the daemon should exit + * with an error. + */ +void fsm_health__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the health thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsm_health__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSM_HEALTH_H */ diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index b8f9f7a0388..16ace43d1c7 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 2c6fa1a5d91..2102a5c9ff5 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); struct fsm_listen_data; /* opaque platform-specific data for listener thread */ +struct fsm_health_data; /* opaque platform-specific data for health thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; + pthread_t health_thread; pthread_mutex_t main_lock; struct strbuf path_worktree_watch; @@ -51,7 +53,9 @@ struct fsmonitor_daemon_state { struct hashmap cookies; int listen_error_code; + int health_error_code; struct fsm_listen_data *listen_data; + struct fsm_health_data *health_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Tue Mar 22 18:22:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7465DC433EF for ; Tue, 22 Mar 2022 18:24:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240368AbiCVSZa (ORCPT ); Tue, 22 Mar 2022 14:25:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240413AbiCVSZV (ORCPT ); Tue, 22 Mar 2022 14:25:21 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5489C90FC6 for ; Tue, 22 Mar 2022 11:23:46 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id q8so13276783wrc.0 for ; Tue, 22 Mar 2022 11:23:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=B6Ob0SoY4xPD5YmOdDUhfS7NheQVJqtjlncyLzMtKDw8b2MVLEFSMCVVx1wSJZqycc qICdxoqxpH1NSbJSdv5Koj7PPghObn3HtQcZ4k18TW86gpntvqltIPLqV4zetIKx2T4X PDtexUP6rjxpnJBkJq1vftXkMbN7ONkvoLVSY1WFhGstfK6tJOEM04aE0nTe4idTRH5y oi8cYW3jxhmosappGDxR0QkTOwtueRgAVmLMtAhtKBDqsqN7LJDOfAv+gP3NqZQGHjch 4HdHuXFCUlBVrke2HN4JSBqAhzpwOL5+uLeGbyLBbeMbirn8x4VLL0cT9oCqsVjZ6qEX ra4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=n0fWinJve8xOnriisQ41I0Wuk/IAIZuZyOrMgsGnrcopOnmEd3/PBUdV8lObHaSrbg BjB94xdq0dunLLASxZ9UVlKoVduUvkNv0ILkbDyNgFZnti/uWPStZjLdgK+lUCxRills ebWrpu2YiIXpeF8z2LGJRJAAkwGsVzjQuuNI5g0Utpc8bEuS/VkcQ8ctaFxNvgyg2+WY HdDzx1V4TJQsiDK4EZXCibCTFzfTVx0Vv6Zo7o3ZYXl4XAFYQ0JGDUPzF4UExoe75TR/ xJ2nZXFcunLogpm5W81BIMYxjCPkRmlD4iVv/bYdRF6YfI/6Mkb6O/01isvDYjHAdBfq 3m5Q== X-Gm-Message-State: AOAM5321XADzxS2wwpju8qlFOEUSRAXMF+7qS4+zBUcGofPzNOhZpp/W ebh7tUiupL64PojhW71x+r1wRKyiiU4= X-Google-Smtp-Source: ABdhPJxsa3IYhRdOBb3qx5zi8GUXdf6pV/YJSY64hTImNDv5KOrFywE9v0UjL/3Y1rFJrmEjxLvEdQ== X-Received: by 2002:a05:6000:1acc:b0:204:2ae7:da14 with SMTP id i12-20020a0560001acc00b002042ae7da14mr4412444wry.609.1647973424557; Tue, 22 Mar 2022 11:23:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az19-20020a05600c601300b0038cadf3aa69sm3376931wmb.36.2022.03.22.11.23.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:42 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:50 +0000 Subject: [PATCH v3 17/27] fsm-health-win32: add polling framework to monitor daemon health Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend the Windows version of the "health" thread to periodically inspect the system and shutdown if warranted. This commit updates the thread's wait loop to use a timeout and defines a (currently empty) table of functions to poll the system. A later commit will add functions to the table to actually inspect the system. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 94b1d020f25..24fc612bf02 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -4,6 +4,24 @@ #include "fsm-health.h" #include "fsmonitor--daemon.h" +/* + * Every minute wake up and test our health. + */ +#define WAIT_FREQ_MS (60 * 1000) + +/* + * State machine states for each of the interval functions + * used for polling our health. + */ +enum interval_fn_ctx { + CTX_INIT = 0, + CTX_TERM, + CTX_TIMER +}; + +typedef int (interval_fn)(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx); + struct fsm_health_data { HANDLE hEventShutdown; @@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) FREE_AND_NULL(state->health_data); } +/* + * A table of the polling functions. + */ +static interval_fn *table[] = { + NULL, /* must be last */ +}; + +/* + * Call all of the polling functions in the table. + * Shortcut and return first error. + * + * Return 0 if all succeeded. + */ +static int call_all(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + int k; + + for (k = 0; table[k]; k++) { + int r = table[k](state, ctx); + if (r) + return r; + } + + return 0; +} + void fsm_health__loop(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data = state->health_data; + int r; + + r = call_all(state, CTX_INIT); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; for (;;) { DWORD dwWait = WaitForMultipleObjects(data->nr_handles, data->hHandles, - FALSE, INFINITE); + FALSE, WAIT_FREQ_MS); if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) goto clean_shutdown; + if (dwWait == WAIT_TIMEOUT) { + r = call_all(state, CTX_TIMER); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; + continue; + } + error(_("health thread wait failed [GLE %ld]"), GetLastError()); goto force_error_stop; @@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state) force_error_stop: state->health_error_code = -1; +force_shutdown: ipc_server_stop_async(state->ipc_server_data); clean_shutdown: + call_all(state, CTX_TERM); return; } From patchwork Tue Mar 22 18:22:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D6A4C433EF for ; Tue, 22 Mar 2022 18:24:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240395AbiCVSZg (ORCPT ); Tue, 22 Mar 2022 14:25:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240428AbiCVSZX (ORCPT ); Tue, 22 Mar 2022 14:25:23 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 811FD90247 for ; Tue, 22 Mar 2022 11:23:51 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id b19so24619215wrh.11 for ; Tue, 22 Mar 2022 11:23:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pg+pIfJCewJfl3Uu7fv/1CMTNHr8lfgpM21Z9LQ/57U=; b=CY1FswVdVhxakPUVay74MigFgkM9F3I8WOuw6lFKP+ax+/lUAOtxChu2TiJtL+yLwj g6xDNTH+hhNh9XQkIBcShAGRaMBTl4nD5s1n7DOZH2Nv8C0op5n+o+uKdElifwgQtgDB wBwVKWfw7SUpFy4T2xPqpHKAx1b0EPHrDjRz8dxTO0aZSHO/OK/4+alZRiac4vht6leR KtEblyuNar2BHDCddT6nytc2djnPVrwJulDNTFzhNcYfNpX4G6oVEIaCglGv5fR2QvdP 3W1Uzq32aEtna0RiMN1DloDPXKB0B3x7ry0uycrR/CjDQvPb+ERv/AmVmASnvbpZPuSk hqyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pg+pIfJCewJfl3Uu7fv/1CMTNHr8lfgpM21Z9LQ/57U=; b=04405BPmURreMf2N7RaZqMyFpWCyL5ZsBfXxbqT7Zbvi4Q7UqiteQgqJe4IegvS1qi lNqdX82wy1n85Em1TcqrZnHzuPUSdK/0ltZXTu+RohHF2CWyGvR1CGrSW1eh5s20t6lL cmQkyXT7e0cWyaT5Mz0WQ0Yp4eLnoVreHKjBwby8a4gD+GQcU3tGUD+RzOoLaMTS3fk+ YVywI0a0hqjrQU5LmRl6RQey/i+//IpByUc+yVMe5ackL88WgU0058FbBT7eC4zel39b G3jdXZCMULvOcKkeuCeIRQhXNpPUA/6eV95iKs+RedaEUGQBjG7WMi43oETCEBqJt39y bFHw== X-Gm-Message-State: AOAM530HzYhFpayWTjBa+uLw9H3FnUQ7Cm7YqFFkKgx7LppcwoiF0rf9 fwrrzDVhpzULAr/mJIZQmpAcJ7ZHZ+A= X-Google-Smtp-Source: ABdhPJxEVuew5E4cDp9rwM+NQU6Vn0Wopcr1qDwAM/Dfl6V8XiBKcbuBGAeiGRgH1wkiMdq0y3vkZw== X-Received: by 2002:a5d:47c4:0:b0:1f0:25ca:6d81 with SMTP id o4-20020a5d47c4000000b001f025ca6d81mr23314013wrc.247.1647973429562; Tue, 22 Mar 2022 11:23:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w7-20020a1cf607000000b00389a5390180sm2348621wmc.25.2022.03.22.11.23.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:47 -0700 (PDT) Message-Id: <13d11713a869f62677ca2f281cc527020e616eec.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:51 +0000 Subject: [PATCH v3 18/27] fsm-health-win32: force shutdown daemon if worktree root moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Force shutdown fsmonitor daemon if the worktree root directory is moved, renamed, or deleted. Use Windows low-level GetFileInformationByHandle() to get and compare the Windows system unique ID for the directory with a cached version when we started up. This lets us detect the case where someone renames the directory that we are watching and then creates a new directory with the original pathname. This is important because we are listening to a named pipe for requests and they are stored in the Named Pipe File System (NPFS) which a kernel-resident pseudo filesystem not associated with the actual NTFS directory. For example, if the daemon was watching "~/foo/", it would have a directory-watch handle on that directory and a named-pipe handle for "//./pipe/...foo". Moving the directory to "~/bar/" does not invalidate the directory handle. (So the daemon would actually be watching "~/bar" but listening on "//./pipe/...foo". If the user then does "git init ~/foo" and causes another daemon to start, the first daemon will still have ownership of the pipe and the second daemon instance will fail to start. "git status" clients in "~/foo" will ask "//./pipe/...foo" about changes and the first daemon instance will tell them about "~/bar". This commit causes the first daemon to shutdown if the system unique ID for "~/foo" changes (changes from what it was when the daemon started). Shutdown occurs after a periodic poll. After the first daemon exits and releases the lock on the named pipe, subsequent Git commands may cause another daemon to be started on "~/foo". Similarly, a subsequent Git command may cause another daemon to be started on "~/bar". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 24fc612bf02..2ea08c1d4e8 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -29,8 +29,150 @@ struct fsm_health_data HANDLE hHandles[1]; /* the array does not own these handles */ #define HEALTH_SHUTDOWN 0 int nr_handles; /* number of active event handles */ + + struct wt_moved + { + wchar_t wpath[MAX_PATH + 1]; + BY_HANDLE_FILE_INFORMATION bhfi; + } wt_moved; }; +/* + * Lookup the system unique ID for the path. This is as close as + * we get to an inode number, but this also contains volume info, + * so it is a little stronger. + */ +static int lookup_bhfi(wchar_t *wpath, + BY_HANDLE_FILE_INFORMATION *bhfi) +{ + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileW(wpath, desired_access, share_mode, NULL, + OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] health thread could not open '%ls'"), + GetLastError(), wpath); + return -1; + } + + if (!GetFileInformationByHandle(hDir, bhfi)) { + error(_("[GLE %ld] health thread getting BHFI for '%ls'"), + GetLastError(), wpath); + CloseHandle(hDir); + return -1; + } + + CloseHandle(hDir); + return 0; +} + +/* + * Compare the relevant fields from two system unique IDs. + * We use this to see if two different handles to the same + * path actually refer to the same *instance* of the file + * or directory. + */ +static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1, + const BY_HANDLE_FILE_INFORMATION *bhfi_2) +{ + return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber && + bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh && + bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow); +} + +/* + * Shutdown if the original worktree root directory been deleted, + * moved, or renamed? + * + * Since the main thread did a "chdir(getenv($HOME))" and our CWD + * is not in the worktree root directory and because the listener + * thread added FILE_SHARE_DELETE to the watch handle, it is possible + * for the root directory to be moved or deleted while we are still + * watching it. We want to detect that here and force a shutdown. + * + * Granted, a delete MAY cause some operations to fail, such as + * GetOverlappedResult(), but it is not guaranteed. And because + * ReadDirectoryChangesW() only reports on changes *WITHIN* the + * directory, not changes *ON* the directory, our watch will not + * receive a delete event for it. + * + * A move/rename of the worktree root will also not generate an event. + * And since the listener thread already has an open handle, it may + * continue to receive events for events within the directory. + * However, the pathname of the named-pipe was constructed using the + * original location of the worktree root. (Remember named-pipes are + * stored in the NPFS and not in the actual file system.) Clients + * trying to talk to the worktree after the move/rename will not + * reach our daemon process, since we're still listening on the + * pipe with original path. + * + * Furthermore, if the user does something like: + * + * $ mv repo repo.old + * $ git init repo + * + * A new daemon cannot be started in the new instance of "repo" + * because the named-pipe is still being used by the daemon on + * the original instance. + * + * So, detect move/rename/delete and shutdown. This should also + * handle unsafe drive removal. + * + * We use the file system unique ID to distinguish the original + * directory instance from a new instance and force a shutdown + * if the unique ID changes. + * + * Since a worktree move/rename/delete/unmount doesn't happen + * that often (and we can't get an immediate event anyway), we + * use a timeout and periodically poll it. + */ +static int has_worktree_moved(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + struct fsm_health_data *data = state->health_data; + BY_HANDLE_FILE_INFORMATION bhfi; + int r; + + switch (ctx) { + case CTX_TERM: + return 0; + + case CTX_INIT: + if (xutftowcs_path(data->wt_moved.wpath, + state->path_worktree_watch.buf) < 0) { + error(_("could not convert to wide characters: '%s'"), + state->path_worktree_watch.buf); + return -1; + } + + /* + * On the first call we lookup the unique sequence ID for + * the worktree root directory. + */ + return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi); + + case CTX_TIMER: + r = lookup_bhfi(data->wt_moved.wpath, &bhfi); + if (r) + return r; + if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) { + error(_("BHFI changed '%ls'"), data->wt_moved.wpath); + return -1; + } + return 0; + + default: + die(_("unhandled case in 'has_worktree_moved': %d"), + (int)ctx); + } + + return 0; +} + + int fsm_health__ctor(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data; @@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) * A table of the polling functions. */ static interval_fn *table[] = { + has_worktree_moved, NULL, /* must be last */ }; From patchwork Tue Mar 22 18:22:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FE4CC433F5 for ; Tue, 22 Mar 2022 18:24:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240394AbiCVSZc (ORCPT ); Tue, 22 Mar 2022 14:25:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240301AbiCVSZY (ORCPT ); Tue, 22 Mar 2022 14:25:24 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59E90DE83 for ; Tue, 22 Mar 2022 11:23:54 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id u16so23631494wru.4 for ; Tue, 22 Mar 2022 11:23:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=m/7nmjzQrPWZOZRTUqvWzqRTYtxoxcBSCt/eY/PHK+w=; b=HEGgBO7ZpcW2EKj9J0FrwU3eBQ/UCv7k+iKs1sSyiIfbetBeV+wsjKS8XPqoxPw3T7 pyUUHXimBmaK67pAj0wLO+pMAb2mU/u4VlYA5mLArOMpVWX/LLY8OW0p3whNnpS3bvGH nPZOgulBrtjOvfpgk4ci31kJamSCJ4xhgDD9s4d3wGxM8E3YCK6IP4XFAw53eYq9Gw15 s6O2TfFevcZwJws/sePikuT3zUmtEoqUKHn/dt1Pohakgu+BPO0WNyp/6K0cOvJDACeY 3I2kDvjbgwdXTwYklalKqZDnIlmc0TlBmU0GZKIrlvg2ohOjcxu7KhcdusC0axOux3XI iGwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=m/7nmjzQrPWZOZRTUqvWzqRTYtxoxcBSCt/eY/PHK+w=; b=EsSLQFPQiq5WF5WI6SDhZUy/OXSp2qsxOJxzMcNxhpPRdV2wew0pV7Yuwke+1tVU3m oew7vjg2F6mWyDuHrw4+HBDbgHtfs+HW9W7gJD6kU5tSCJvowclcJpQTYkqkFfPA51xC W0/1Wwf5UXCgxiRKzNSefO3Ky3JCrEQN5OXAGnpFK5y+kvqH+QmwcTY3ZEuFSUZ0TQWz zZXI5N2fN/5+9pdWJLQjYuRmSfenSTZ1sAYQuFst1YX0k1QarUg54JkgskCQotyci5x3 TyT/YInTt96pcs1hiazs4buaJTvkO/PLeqXTnnIOgekcaDeXy9+B0nFGr68jP+/ov0ay SINg== X-Gm-Message-State: AOAM5300YFgMr255ZXf0VMEfYdwKl+PC9nH6SuiwmwNKVn3v8WU+g1aT BsNC9vTaDdKeJKkdnLytTmRcsNgtBa0= X-Google-Smtp-Source: ABdhPJzH5hGpC46pWeycTxEy9fVdruCYHXAvtuIiqlIFSQ0Pa6fRGfTRhsb9esWaiL9CjN45ncl+QA== X-Received: by 2002:a05:6000:2c9:b0:204:1ce4:7c29 with SMTP id o9-20020a05600002c900b002041ce47c29mr6611388wry.51.1647973432508; Tue, 22 Mar 2022 11:23:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e8-20020a056000178800b00203da3bb4d2sm17405632wrg.41.2022.03.22.11.23.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:50 -0700 (PDT) Message-Id: <01c1a38c46243666c8af0556d5d7588cf4203ada.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:52 +0000 Subject: [PATCH v3 19/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the listener thread to shutdown the daemon if the spelling of the worktree root directory changes. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 07113205a61..83d38e8ac6c 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) strbuf_release(&msg); } +static int ef_is_root_changed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagRootChanged); +} + static int ef_is_root_delete(const FSEventStreamEventFlags ef) { return (ef & kFSEventStreamEventFlagItemIsDir && @@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_is_root_changed(event_flags[k])) { + /* + * The spelling of the pathname of the root directory + * has changed. This includes the name of the root + * directory itself or of any parent directory in the + * path. + * + * (There may be other conditions that throw this, + * but I couldn't find any information on it.) + * + * Force a shutdown now and avoid things getting + * out of sync. The Unix domain socket is inside + * the .git directory and a spelling change will make + * it hard for clients to rendezvous with us. + */ + trace_printf_key(&trace_fsmonitor, + "event: root changed"); + goto force_shutdown; + } + if (ef_ignore_xattr(event_flags[k])) { trace_printf_key(&trace_fsmonitor, "ignore-xattr: '%s', flags=0x%x", From patchwork Tue Mar 22 18:22:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC5D9C433EF for ; Tue, 22 Mar 2022 18:24:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232158AbiCVSZe (ORCPT ); Tue, 22 Mar 2022 14:25:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240330AbiCVSZY (ORCPT ); Tue, 22 Mar 2022 14:25:24 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50290692A9 for ; Tue, 22 Mar 2022 11:23:56 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id r64so11844883wmr.4 for ; Tue, 22 Mar 2022 11:23:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=GW69XiKo2T/Xo2pKsYW+LWIVSmqjRk4do21ccvdTPZJg7IcbQcpYd/qfst22QPFgUc gJCUl2AGkpry7LLAJLj10AP54Ncs7uvx4Vl2fR2BJafe/4Q7S2XBJPZVXfxTTKDp+3p0 yXqmYlLjI9U7D1nXPOPfn5HwcP3wa43ESB8xVAa1UNieiET0RI6KQv9F3FoU7GmEsUmw bMudwMkhCUw9/R4WiKZW0osXcptxc0Atp5knD2pu0gSxO0o8iBJkKUOPixzr+8X0V3nq UnnRgzX5Zik5zqYT9eRgo1b1f9rTthqA1fqQlM6SJVDWg72WXE4SB/pWSBzYQfJhHdH9 gNKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=iuNu0h9ZpthW3E2CNmHDJWocQWXHjDvFsGf0NnksZHcbS16IFVB4NutxHbnTYtCSpz lBaT8LrwTdwR2B5hJuqjWGYcXs8YdvzFKtmM0cqGaIH2jJIHZEvwljr0ZvU5tPH7MWbL oCc3jYVa7/4xBda6onT/T6m6bSnTakpis8AFaunuqbtgS4bgDCqTonKUocYmUB7MJmsu OJSQbNiZDNjjvXmaqGRIVUyqDhoDFPra/sk0JijHoRcXYf8vg+fAqjRljjxUenVDfpMI c9jZ6CfgXtgz4s3ws+lAMUiw1mkiX3qzK1r8tyiOFqAqAhdgEe4f6kiQKMyxR3xL6fZd I79w== X-Gm-Message-State: AOAM5312Qd3XKhUuWT9PMGdbML8kSfS8I2vZj0wDNwVOuOKfabMIcdzM /pCLr2FBPCVfnBa6QDBee43syT47ibY= X-Google-Smtp-Source: ABdhPJx146YANg8jgunnul68Wey41i4Zu/4w4W0eJFjbXgOmRFxWrBNrjwDku/xSK+w8mjoKs+BcHA== X-Received: by 2002:a7b:c5d0:0:b0:389:fe85:3d79 with SMTP id n16-20020a7bc5d0000000b00389fe853d79mr5263864wmk.77.1647973434443; Tue, 22 Mar 2022 11:23:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v124-20020a1cac82000000b0037c3d08e0e7sm2358758wme.29.2022.03.22.11.23.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:53 -0700 (PDT) Message-Id: <0f0a5b5ca1630985b9a91fa0e6c5fc951561679c.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:53 +0000 Subject: [PATCH v3 20/27] fsmonitor: optimize processing of directory events Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to perform binary search over the cache-entries for a directory notification and then linearly scan forward to find the immediate children. Previously, when the FSMonitor reported a modified directory Git would perform a linear search on the entire cache-entry array for all entries matching that directory prefix and invalidate them. Since the cache-entry array is already sorted, we can use a binary search to find the first matching entry and then only linearly walk forward and invalidate entries until the prefix changes. Also, the original code would invalidate anything having the same directory prefix. Since a directory event should only be received for items that are immediately within the directory (and not within sub-directories of it), only invalidate those entries and not the whole subtree. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 54 insertions(+), 17 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 292a6742b4f..e1229c289cf 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r, static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { int i, len = strlen(name); - if (name[len - 1] == '/') { + int pos = index_name_pos(istate, name, len); + + trace_printf_key(&trace_fsmonitor, + "fsmonitor_refresh_callback '%s' (pos %d)", + name, pos); + if (name[len - 1] == '/') { /* - * TODO We should binary search to find the first path with - * TODO this directory prefix. Then linearly update entries - * TODO while the prefix matches. Taking care to search without - * TODO the trailing slash -- because '/' sorts after a few - * TODO interesting special chars, like '.' and ' '. + * The daemon can decorate directory events, such as + * moves or renames, with a trailing slash if the OS + * FS Event contains sufficient information, such as + * MacOS. + * + * Use this to invalidate the entire cone under that + * directory. + * + * We do not expect an exact match because the index + * does not normally contain directory entries, so we + * start at the insertion point and scan. */ + if (pos < 0) + pos = -pos - 1; /* Mark all entries for the folder invalid */ - for (i = 0; i < istate->cache_nr; i++) { - if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID && - starts_with(istate->cache[i]->name, name)) - istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } - /* Need to remove the / from the path for the untracked cache */ + + /* + * We need to remove the traling "/" from the path + * for the untracked cache. + */ name[len - 1] = '\0'; + } else if (pos >= 0) { + /* + * We have an exact match for this path and can just + * invalidate it. + */ + istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID; } else { - int pos = index_name_pos(istate, name, strlen(name)); - - if (pos >= 0) { - struct cache_entry *ce = istate->cache[pos]; - ce->ce_flags &= ~CE_FSMONITOR_VALID; + /* + * The path is not a tracked file -or- it is a + * directory event on a platform that cannot + * distinguish between file and directory events in + * the event handler, such as Windows. + * + * Scan as if it is a directory and invalidate the + * cone under it. (But remember to ignore items + * between "name" and "name/", such as "name-" and + * "name.". + */ + pos = -pos - 1; + + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + if ((unsigned char)istate->cache[i]->name[len] > '/') + break; + if (istate->cache[i]->name[len] == '/') + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } } @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) * Mark the untracked cache dirty even if it wasn't found in the index * as it could be a new untracked file. */ - trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name); untracked_cache_invalidate_path(istate, name, 0); } From patchwork Tue Mar 22 18:22:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788924 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9734EC433EF for ; Tue, 22 Mar 2022 18:24:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240353AbiCVS0C (ORCPT ); Tue, 22 Mar 2022 14:26:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237387AbiCVSZ0 (ORCPT ); Tue, 22 Mar 2022 14:25:26 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E3B715FEC for ; Tue, 22 Mar 2022 11:23:58 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id j18so15004410wrd.6 for ; Tue, 22 Mar 2022 11:23:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=c9U8uQ8YcJe+AKnqA/7k7C5uEO6mNcd8TsCgZ3bkj2s=; b=CzsvLWT7fRvUNnqhjMcujjBdxX675DlRPzOM+sUNlp6kYO5Mb1Wie0R9e3Z0gDebLa gya5r/dAPM0e7u4NTX4zitO5AlPAGEsNOF07v847b9N37jLmtwAIH8V+KN4ckgEuf5sC g+5H66wYHpAwKVoaLjtWNtFBk7jrHSEUlZjrZqxcXxBkCQJVgTsuBmwIWHlMylGlW9CG SEpctWPI7qgj2qXCwfstU5tRPVP3n9G7npmwTPHTX0EMmEpk4WFPco0/QEqxOy7F5QpG jAnrg7NxHTAYyQ7dwVF77I4kAWFgOCXmc215KMxX50jh6WO9C9baKvbkooE5UNOkwd5O 6JNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=c9U8uQ8YcJe+AKnqA/7k7C5uEO6mNcd8TsCgZ3bkj2s=; b=bcrvYO+dES4I/GtYH47+M4vfFP9Q0rorrpS4dUDgjEiboRiQ/JJMojrvAnWip2T+Tv t4K/v43F+0BBAzMKeLv6zdLkyM6Fn4SVgH4ogehQwvDMB0acYD6MIY+4PuJcLtlV2VfJ fdQlclLjK0ygnSdcsbZ4PyZD9t045JmnHiB+bfoGaM5q6rXISb33YmrY4r29Y4tsU9Y5 rPNuiNcAhc69npag/7Lmoed5pLFcEcvF9+JtHkY4YkPMl2bqRgViHb9ijS4RqaNyUMKH Z5NPCmyqntGHiM3VgwccZqd9+WuYeRN6SYxkqMz88so+mbi8Bu2GmKomwGELWxTDn198 OpNQ== X-Gm-Message-State: AOAM530sBPusxg+UjwdIbUkhnDV1Dbch46KNKpi8qbeMB+QQAqvxmXeQ RM8JbvybYhDaZczD5TRbLGpfB/ICvSg= X-Google-Smtp-Source: ABdhPJxeexjXLhSr8Yg8Qf+Vfa/SDfmhvAApcnimYI4CfbVCJGOiUSFAeuUcqTm7+2FXGL0DSD4cWw== X-Received: by 2002:a05:6000:178e:b0:205:7acf:8fb1 with SMTP id e14-20020a056000178e00b002057acf8fb1mr3465031wrg.499.1647973436499; Tue, 22 Mar 2022 11:23:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g5-20020a5d64e5000000b00203914f5313sm16277535wri.114.2022.03.22.11.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:55 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:54 +0000 Subject: [PATCH v3 21/27] t7527: FSMonitor tests for directory moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create unit tests to move a directory. Verify that `git status` gives the same result with and without FSMonitor enabled. NEEDSWORK: This test exposes a bug in the untracked-cache on Windows when FSMonitor is disabled. These are commented out for the moment. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 429029fcadd..bc66d8285a3 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -281,6 +281,16 @@ test_expect_success 'setup' ' trace* EOF + mkdir -p T1/T2/T3/T4 && + echo 1 >T1/F1 && + echo 1 >T1/T2/F1 && + echo 1 >T1/T2/T3/F1 && + echo 1 >T1/T2/T3/T4/F1 && + echo 2 >T1/F2 && + echo 2 >T1/T2/F2 && + echo 2 >T1/T2/T3/F2 && + echo 2 >T1/T2/T3/T4/F2 && + git -c core.fsmonitor=false add . && test_tick && git -c core.fsmonitor=false commit -m initial && @@ -363,6 +373,19 @@ directory_to_file () { echo 1 >dir1 } +move_directory_contents_deeper() { + mkdir T1/_new_ && + mv T1/[A-Z]* T1/_new_ +} + +move_directory_up() { + mv T1/T2/T3 T1 +} + +move_directory() { + mv T1/T2/T3 T1/T2/NewT3 +} + # The next few test cases confirm that our fsmonitor daemon sees each type # of OS filesystem notification that we care about. At this layer we just # ensure we are getting the OS notifications and do not try to confirm what @@ -645,6 +668,22 @@ do matrix_try $uc_val $fsm_val file_to_directory matrix_try $uc_val $fsm_val directory_to_file + # NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor + # is DISABLED. Turn off a few test that cause it problems until + # we can debug it. + # + try_moves="true" + test_have_prereq UNTRACKED_CACHE,WINDOWS && \ + test $uc_val = true && \ + test $fsm_val = false && \ + try_moves="false" + if test $try_moves = true + then + matrix_try $uc_val $fsm_val move_directory_contents_deeper + matrix_try $uc_val $fsm_val move_directory_up + matrix_try $uc_val $fsm_val move_directory + fi + if test $fsm_val = true then test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" ' From patchwork Tue Mar 22 18:22:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D000DC433FE for ; Tue, 22 Mar 2022 18:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238421AbiCVS0E (ORCPT ); Tue, 22 Mar 2022 14:26:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240345AbiCVSZ3 (ORCPT ); Tue, 22 Mar 2022 14:25:29 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 845C58D6B9 for ; Tue, 22 Mar 2022 11:24:00 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id h4so6675581wrc.13 for ; Tue, 22 Mar 2022 11:24:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=ZdoPxpf6wKKPp80BSk0zp2U0ZWPCaxghwP7iawLoK5zFOkKJIcMcuyKRSgnXCPQ8/g ZCAbULUcXvwbanhh++xoOdU8yzwTO8ZuIYPAWhQxFueZydIW2Xf2pj5HBgKYyQsmhk7v ObUXw+JIXK9qBPI+ApkxlbY/5W4+eJEwvmge6seFAsHZlnmz2SRa3f/Ep/sS97Nz8xmW qg8oioFPTClhPr+BjRrhoredLAQx4rsw38CGsLy+0r1qX64k30wih02cUbLy8QJjtmBF X8f6LlP1zCQ2jmTh5iZ760stepiuWGwpSPzJZylxhIhyPZDMX6TZTcqfMps1G4fL1uIZ 1+nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=avd8Ju/HyIiMD7eJyAw2l/uwQ2otosbNW7MQNh/aRYLqpJf6R0yGjkyf2woCrYKqke kTSJqigZOP/lMk7okpDi7hOMBWRP4H8Ha5ncqav/cnczUXKkv5uDaEaSpfqmaY946Zd/ rO/s1rmc6jdcHroG3fLmYIUhfG46niStB9Ml5akPefvbpbHp/3cghqGqbE+rn20zrRh0 FOVjcQ4Iftm8KPaanG8sEZaVsjHUzahNUx7NjqrTO7Afe8DskiTflV4wwfHH9KT04KWO vX4i6wNBu7SY8FSzSFsxPJz+IFfoa+iYv+GMftXKhBfgcxkSGdejQZiA/pIDQ47McQ6a XW1Q== X-Gm-Message-State: AOAM531PYWN/+rY0fI1MeN/TcvgcCTC71BGQUUmCpjZDFOOzqJa7InoI mDDKzoS6xndpfQi7mIqh/9S4/Ee+cxA= X-Google-Smtp-Source: ABdhPJysj0XQKmYXpiEButBK/O/UJNC0+ljwLPRl0tweKeQ1CDO7iWq1Ms8ac8AAcamAFaMdY48VXQ== X-Received: by 2002:adf:a58a:0:b0:1ef:603f:fb58 with SMTP id g10-20020adfa58a000000b001ef603ffb58mr23301575wrc.472.1647973438711; Tue, 22 Mar 2022 11:23:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p19-20020a05600c359300b0038c7d1086a7sm2374547wmq.1.2022.03.22.11.23.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:57 -0700 (PDT) Message-Id: <79da369dcce69f14dab19116e23aaec36ca4d92a.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:55 +0000 Subject: [PATCH v3 22/27] t/perf/p7527: add perf test for builtin FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100755 t/perf/p7527-builtin-fsmonitor.sh diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh new file mode 100755 index 00000000000..9338b9ea008 --- /dev/null +++ b/t/perf/p7527-builtin-fsmonitor.sh @@ -0,0 +1,257 @@ +#!/bin/sh + +test_description="Perf test for the builtin FSMonitor" + +. ./perf-lib.sh + +if ! test_have_prereq FSMONITOR_DAEMON +then + skip_all="fsmonitor--daemon is not supported on this platform" + test_done +fi + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +# Lie to perf-lib and ask for a new empty repo and avoid +# the complaints about GIT_PERF_REPO not being big enough +# the perf hit when GIT_PERF_LARGE_REPO is copied into +# the trash directory. +# +# NEEDSWORK: It would be nice if perf-lib had an option to +# "borrow" an existing large repo (especially for gigantic +# monorepos) and use it in-place. For now, fake it here. +# +test_perf_fresh_repo + + +# Use a generated synthetic monorepo. If it doesn't exist, we will +# generate it. If it does exist, we will put it in a known state +# before we start our timings. +# +PARAM_D=5 +PARAM_W=10 +PARAM_F=9 + +PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F" + +BALLAST_BR=p0006-ballast +export BALLAST_BR + +TMP_BR=tmp_br +export TMP_BR + +REPO=../repos/gen-many-files-"$PARAMS".git +export REPO + +if ! test -d $REPO +then + (cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F) +fi + + +enable_uc () { + git -C $REPO config core.untrackedcache true + git -C $REPO update-index --untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +disable_uc () { + git -C $REPO config core.untrackedcache false + git -C $REPO update-index --no-untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +start_fsm () { + git -C $REPO fsmonitor--daemon start + git -C $REPO fsmonitor--daemon status + git -C $REPO config core.fsmonitor true + git -C $REPO update-index --fsmonitor + git -C $REPO status >/dev/null 2>&1 +} + +stop_fsm () { + git -C $REPO config --unset core.fsmonitor + git -C $REPO update-index --no-fsmonitor + test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null + git -C $REPO status >/dev/null 2>&1 +} + + +# Ensure that FSMonitor is turned off on the borrowed repo. +# +test_expect_success "Setup borrowed repo (fsm+uc)" " + stop_fsm && + disable_uc +" + +# Also ensure that it starts in a known state. +# +# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time +# the ballast checkout, since only the first invocation does any work and the +# subsequent ones just print "already on branch" and quit, so the reported +# time is not useful. +# +# Create a temp branch and do all work relative to it so that we don't +# accidentially alter the real ballast branch. +# +test_expect_success "Setup borrowed repo (temp ballast branch)" " + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO reset --hard && + git -C $REPO clean -d -f && + test_might_fail git -C $REPO branch -D $TMP_BR && + git -C $REPO branch $TMP_BR $BALLAST_BR && + git -C $REPO checkout $TMP_BR +" + + +echo Data >data.txt + +# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With +# FSMonitor enabled, we can get a skewed view of status times, since +# the index MAY (or may not) be updated after the first invocation +# which will update the FSMonitor Token, so the subsequent invocations +# may get a smaller response from the daemon. +# +do_status () { + msg=$1 + + test_perf "$msg" " + git -C $REPO status >/dev/null 2>&1 + " +} + +do_matrix () { + uc=$1 + fsm=$2 + + t="[uc $uc][fsm $fsm]" + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_expect_success "$t Setup matrix branch" " + git -C $REPO clean -d -f && + git -C $REPO checkout $TMP_BR && + test_might_fail git -C $REPO branch -D $MATRIX_BR && + git -C $REPO branch $MATRIX_BR $TMP_BR && + git -C $REPO checkout $MATRIX_BR + " + + if test $uc = true + then + enable_uc + else + disable_uc + fi + + if test $fsm = true + then + start_fsm + else + stop_fsm + fi + + do_status "$t status after checkout" + + # Modify many files in the matrix branch. + # Stage them. + # Commit them. + # Rollback. + # + test_expect_success "$t modify tracked files" " + find $REPO -name file1 -exec cp data.txt {} \\; + " + + do_status "$t status after big change" + + # Don't bother timing the "add" because _REPEAT_COUNT + # issue described above. + # + test_expect_success "$t add all" " + git -C $REPO add -A + " + + do_status "$t status after add all" + + test_expect_success "$t add dot" " + git -C $REPO add . + " + + do_status "$t status after add dot" + + test_expect_success "$t commit staged" " + git -C $REPO commit -a -m data + " + + do_status "$t status after commit" + + test_expect_success "$t reset HEAD~1 hard" " + git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1 + " + + do_status "$t status after reset hard" + + # Create some untracked files. + # + test_expect_success "$t create untracked files" " + cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1 + " + + do_status "$t status after create untracked files" + + # Remove the new untracked files. + # + test_expect_success "$t clean -df" " + git -C $REPO clean -d -f + " + + do_status "$t status after clean" + + if test $fsm = true + then + stop_fsm + fi +} + +# Begin testing each case in the matrix that we care about. +# +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" + +fsm_values="false true" + +for uc_val in $uc_values +do + for fsm_val in $fsm_values + do + do_matrix $uc_val $fsm_val + done +done + +cleanup () { + uc=$1 + fsm=$2 + + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_might_fail git -C $REPO branch -D $MATRIX_BR +} + + +# We're borrowing this repo. We should leave it in a clean state. +# +test_expect_success "Cleanup temp and matrix branches" " + git -C $REPO clean -d -f && + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO branch -D $TMP_BR && + for uc_val in $uc_values + do + for fsm_val in $fsm_values + do + cleanup $uc_val $fsm_val + done + done +" + +test_done From patchwork Tue Mar 22 18:22:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED393C433EF for ; Tue, 22 Mar 2022 18:24:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235877AbiCVS0W (ORCPT ); Tue, 22 Mar 2022 14:26:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240340AbiCVS0B (ORCPT ); Tue, 22 Mar 2022 14:26:01 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB2528F997 for ; Tue, 22 Mar 2022 11:24:02 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id u16so23631963wru.4 for ; Tue, 22 Mar 2022 11:24:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3LdO1VFXn1omyNsVhABS+UE1aKRNR5fWXMxTnb1/fGo=; b=gbCpPx5zWlbPbzBIBHYvrz34aNUzqYgPFOlUP+7mDLO7NtxI8aud4aASd5wc/xuPte K+I/KxSp06TSnVLB7xS7QwyfeMFwUsK5iGpbLL9hMYwutGmHrkgMxmSsRSUH0aB25riF x12tNbrAm3ci1bZA5D8dk+bpU/JIBcRQUaRbJhYuJzEQAx8C1H7pD7tlZQfBoLSZ8Nko eawxxm3W/cvaRhaqABoj8Nd8ZnW0M/yqmsDG+/b6V/UcVx7+SKiGkulYypHmh1l8TmKm oC/Kr34IHIjTg6Yq7D1k+xzaTkanoYTuFhN8IpWxsXuPzJsMujRlAUFm1V9wPoCGS8dl 1m9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3LdO1VFXn1omyNsVhABS+UE1aKRNR5fWXMxTnb1/fGo=; b=Dc0x6mNF4f0+pglFOQwOrw5N2CjAO7iRWdSdGhWyerFRctzmDbNcbklgg4PmV7/7TO 3ZS3NGGHYlqBhPsxyi0sJbkkEuarFytSJHUobtUf/SGnCHYW1ckqbRHd5IWzgYsnK91S 5c2YTnYhR/eFbDonzcnkllNVarhWTJT1cFVQ1amvx8Q12MEId6DBIFMwpatfVnNZsR6b mzErG4p/zr/AK1nmyP5IyvoyEK1Cy/wemUBovhKK+ZxyCxHT48NzxEht0WZa3rAT11J+ lrhvAo+rDx2PfKi/b6qsCwA0ZuoE84bdNl96HM/U6jDHfz3e8i0YRj+NQ4kWAIaJYrOT Rqtw== X-Gm-Message-State: AOAM5331R6V2yUqgsd+fp5UoLgoVJhNECUI0ZB385KrkXdJMvZaRZ2CQ jMujwjg1DmKk6WraqXheqlQU1WeKTnw= X-Google-Smtp-Source: ABdhPJwTlywDQp2kxi+UrQzwOcZeCbfs0lnEuZrMyP7lWPL2tFlkbr0o6/uhC9PLdZOqKuB/ZxGXIQ== X-Received: by 2002:a5d:59ac:0:b0:205:7e94:bc0f with SMTP id p12-20020a5d59ac000000b002057e94bc0fmr2310929wrr.484.1647973441019; Tue, 22 Mar 2022 11:24:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f7-20020a1cc907000000b0038c756fe683sm2331432wmb.43.2022.03.22.11.23.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:23:59 -0700 (PDT) Message-Id: <4ab4306adab355cbad98457c34730c88197fd422.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:56 +0000 Subject: [PATCH v3 23/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Never set CE_FSMONITOR_VALID on the cache-entry of submodule directories. During a client command like 'git status', we may need to recurse into each submodule to compute a status summary for the submodule. Since the purpose of the ce_flag is to let Git avoid scanning a cache-entry, setting the flag causes the recursive call to be avoided and we report incorrect (no status) for the submodule. We created an OS watch on the root directory of our working directory and we receive events for everything in the cone under it. When submodules are present inside our working directory, we receive events for both our repo (the super) and any subs within it. Since our index doesn't have any information for items within the submodules, we can't use those events. We could try to truncate the paths of those events back to the submodule boundary and mark the GITLINK as dirty, but that feels expensive since we would have to prefix compare every FS event that we receive against a list of submodule roots. And it still wouldn't be sufficient to correctly report status on the submodule, since we don't have any space in the cache-entry to cache the submodule's status (the 'SCMU' bits in porcelain V2 speak). That is, the CE_FSMONITOR_VALID bit just says that we don't need to scan/inspect it because we already know the answer -- it doesn't say that the item is clean -- and we don't have space in the cache-entry to store those answers. So we should always do the recursive scan. Therefore, we should never set the flag on GITLINK cache-entries. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 2 + fsmonitor.h | 11 +++++ t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++ 3 files changed, 106 insertions(+) diff --git a/fsmonitor.c b/fsmonitor.c index e1229c289cf..57d6a483bee 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate) if (fsmonitor_enabled) { /* Mark all entries valid */ for (i = 0; i < istate->cache_nr; i++) { + if (S_ISGITLINK(istate->cache[i]->ce_mode)) + continue; istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; } diff --git a/fsmonitor.h b/fsmonitor.h index 3f41f653691..edf7ce5203b 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate) * Set the given cache entries CE_FSMONITOR_VALID bit. This should be * called any time the cache entry has been updated to reflect the * current state of the file on disk. + * + * However, never mark submodules as valid. When commands like "git + * status" run they might need to recurse into the submodule (using a + * child process) to get a summary of the submodule state. We don't + * have (and don't want to create) the facility to translate every + * FS event that we receive and that happens to be deep inside of a + * submodule back to the submodule root, so we cannot correctly keep + * track of this bit on the gitlink directory. Therefore, we never + * set it on submodules. */ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce) { @@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache if (fsm_mode > FSMONITOR_MODE_DISABLED && !(ce->ce_flags & CE_FSMONITOR_VALID)) { + if (S_ISGITLINK(ce->ce_mode)) + return; istate->cache_changed = 1; ce->ce_flags |= CE_FSMONITOR_VALID; trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index bc66d8285a3..39efed42a69 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -718,4 +718,97 @@ do ' done +# Test fsmonitor interaction with submodules. +# +# If we start the daemon in the super, it will see FS events for +# everything in the working directory cone and this includes any +# files/directories contained *within* the submodules. +# +# A `git status` at top level will get events for items within the +# submodule and ignore them, since they aren't named in the index +# of the super repo. This makes the fsmonitor response a little +# noisy, but it doesn't alter the correctness of the state of the +# super-proper. +# +# When we have submodules, `git status` normally does a recursive +# status on each of the submodules and adds a summary row for any +# dirty submodules. (See the "S..." bits in porcelain V2 output.) +# +# It is therefore important that the top level status not be tricked +# by the FSMonitor response to skip those recursive calls. + +my_match_and_clean () { + git -C super --no-optional-locks status --porcelain=v2 >actual.with && + git -C super --no-optional-locks -c core.fsmonitor=false \ + status --porcelain=v2 >actual.without && + test_cmp actual.with actual.without && + + git -C super/dir_1/dir_2/sub reset --hard && + git -C super/dir_1/dir_2/sub clean -d -f +} + +test_expect_success "Submodule" ' + test_when_finished "git -C super fsmonitor--daemon stop" && + + git init "super" && + echo x >super/file_1 && + echo y >super/file_2 && + echo z >super/file_3 && + mkdir super/dir_1 && + echo a >super/dir_1/file_11 && + echo b >super/dir_1/file_12 && + mkdir super/dir_1/dir_2 && + echo a >super/dir_1/dir_2/file_21 && + echo b >super/dir_1/dir_2/file_22 && + git -C super add . && + git -C super commit -m "initial super commit" && + + git init "sub" && + echo x >sub/file_x && + echo y >sub/file_y && + echo z >sub/file_z && + mkdir sub/dir_x && + echo a >sub/dir_x/file_a && + echo b >sub/dir_x/file_b && + mkdir sub/dir_x/dir_y && + echo a >sub/dir_x/dir_y/file_a && + echo b >sub/dir_x/dir_y/file_b && + git -C sub add . && + git -C sub commit -m "initial sub commit" && + + git -C super submodule add ../sub ./dir_1/dir_2/sub && + git -C super commit -m "add sub" && + + start_daemon -C super && + git -C super config core.fsmonitor true && + git -C super update-index --fsmonitor && + git -C super status && + + # Now run pairs of commands w/ and w/o FSMonitor while we make + # some dirt in the submodule and confirm matching output. + + # Completely clean status. + my_match_and_clean && + + # .M S..U + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u && + my_match_and_clean && + + # .M S.M. + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M S.M. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M SC.. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + git -C super/dir_1/dir_2/sub commit -m "SC.." && + my_match_and_clean +' + test_done From patchwork Tue Mar 22 18:22:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 870A3C433FE for ; Tue, 22 Mar 2022 18:24:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240340AbiCVS0Y (ORCPT ); Tue, 22 Mar 2022 14:26:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240347AbiCVS0B (ORCPT ); Tue, 22 Mar 2022 14:26:01 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BA458FE45 for ; Tue, 22 Mar 2022 11:24:04 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id q20so11867388wmq.1 for ; Tue, 22 Mar 2022 11:24:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Fpp+VSFtO8Ks/JINfvbBSypJdw7fJ1NuYrbBZJ7ri20=; b=CObvgVuLO5hAI02DWKcFva6SDl73muVO1/i0cPqgcd8SQryab/pJarv2Tt0NjTCY/0 lkIMEC19fGMWuqjWKzR2AgbF4dDaYV2Xt1BoJl3X9TbiFvG3d9u/AVOhjEMJNCI0LjQ4 v/feF2v2VXLlz5jI7Rjk8OcOuGCGBhFw6hSRjUjJP0Rrrc9tmS2xOluDi0mOeC1iOQSS SDDwkmd2hPsBd8/N3imSbtWSWOboV8cZ/rpTcoI50UkBAT+BV4fwF9mq20oRZ6B48zko bgub7WHRnzxfN+pGH4RaMPFCNZvcPcDYH4yuJqOllTIjgWWnC0Ef+90kWbS2wkEmxMRF C0AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Fpp+VSFtO8Ks/JINfvbBSypJdw7fJ1NuYrbBZJ7ri20=; b=02GdpKvdC84Ed34aBNEbXfVtGyutTEaSTlJFiTpJRe+kP2zVCbdS9vofgz4PqiXwjF irtAJOrDdXMUilcQ1yipVJ6/dF1y+P0M6lSZEqb15UFOrUP/9Zl3pK7WgqqN70RQV9GP kGQ8njR+0w+3rdVubm1FYtkgHxANh1Dptt/q9CdzLM3zFTmg9j0DPk0f7jr3am8I41/B 4mDZDaD99wJN3wt7FkVLGbM7xwrffWEZI/U1Tv9ACUp5m4q63RcOwK+IAIlxfgDun9rz 4hWVW2KM8w4hUDSW8+KBGOUc9f3qEAICz+VjXPoOCscpn0mHsIluw00xF7ciISEOF5Wd aJPA== X-Gm-Message-State: AOAM532UW4lJCEeHKLAY69L9Za5jVGK1uxa9S+3jej/NaVFVzYPqZzio jDJNja2PJzNCxmqMWlT5kUn75Y7pUN0= X-Google-Smtp-Source: ABdhPJx9oKSlH/KxwUlGj2XKc/BUBpn0iMJcmFWEUqdJwJ5PTuy2bww0d8/ZqGNk4Ru+3zX87NynDA== X-Received: by 2002:a05:600c:22c4:b0:38c:70f5:25a9 with SMTP id 4-20020a05600c22c400b0038c70f525a9mr5078963wmg.119.1647973443287; Tue, 22 Mar 2022 11:24:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z12-20020a5d4d0c000000b002057d6f7053sm1203356wrt.47.2022.03.22.11.24.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:24:01 -0700 (PDT) Message-Id: <5d0fa19929d8b326786f967f8e7ba40b2bc834d2.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:57 +0000 Subject: [PATCH v3 24/27] t7527: test FSMonitor on case insensitive+preserving file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Test that FS events from the OS are received using the preserved, on-disk spelling of files/directories rather than spelling used to make the change. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 39efed42a69..51cdf05e7ec 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -811,4 +811,40 @@ test_expect_success "Submodule" ' my_match_and_clean ' +# On a case-insensitive file system, confirm that the daemon +# notices when the .git directory is moved/renamed/deleted +# regardless of how it is spelled in the the FS event. +# That is, does the FS event receive the spelling of the +# operation or does it receive the spelling preserved with +# the file/directory. +# +test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' +# test_when_finished "stop_daemon_delete_repo test_insensitive" && + + git init test_insensitive && + + start_daemon -C test_insensitive -tf "$PWD/insensitive.trace" && + + mkdir -p test_insensitive/abc/def && + echo xyz >test_insensitive/ABC/DEF/xyz && + + test_path_is_dir test_insensitive/.git && + test_path_is_dir test_insensitive/.GIT && + + # Rename .git using an alternate spelling to verify that that + # daemon detects it and automatically shuts down. + mv test_insensitive/.GIT test_insensitive/.FOO && + sleep 1 && + mv test_insensitive/.FOO test_insensitive/.git && + test_must_fail git -C test_insensitive fsmonitor--daemon status && + + # Verify that events were reported using on-disk spellings of the + # directories and files that we touched. We may or may not get a + # trailing slash on modified directories. + # + egrep "^event: abc/?$" ./insensitive.trace && + egrep "^event: abc/def/?$" ./insensitive.trace && + egrep "^event: abc/def/xyz$" ./insensitive.trace +' + test_done From patchwork Tue Mar 22 18:22:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45522C433EF for ; Tue, 22 Mar 2022 18:25:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240405AbiCVS0i (ORCPT ); Tue, 22 Mar 2022 14:26:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240365AbiCVS0I (ORCPT ); Tue, 22 Mar 2022 14:26:08 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 469BE90CFC for ; Tue, 22 Mar 2022 11:24:07 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id b19so24620111wrh.11 for ; Tue, 22 Mar 2022 11:24:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bIorACD9MvGhf+EOOe9S7KEedrit1duOzRZ5v8bUeZQ=; b=WrUgnjcGMGoGGKU/Ehif714ADbETA5wXYLRFv0bS7YcWzVqRFwCggAU7TM0KVbUCz3 7BGldD50SUPclN7ZloXy+ms9idH2FEBciLaI3mA0uw2Oy2aDc8VuUOgvr1iJH2Y809eh IDtRPAwYYNLD08yIkFhxP1JvB6mMvUsnOsiZdcbRb6zTQXBJfGYB8CP1kqYTxf+kc2v3 UD4j0WOFd8n/peP8uAzCF1C+9dJu3WztgxXfvuT1QzHA37sUi/8K3pQxJiXKBXyYId9R CprtyclWrf93DKOZ7VZdCHuDV4LdHaZ3MR5Xwbu8fWZFSN9kgnRiPeTGLVvdTY0a4Vyd m1QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bIorACD9MvGhf+EOOe9S7KEedrit1duOzRZ5v8bUeZQ=; b=GS6lWzF0EvrYMJqSIrew81y/GrQ59BMyu/o8XCeNLZ4alGTUNOUDEjuTfuN7ZHRGL+ YnPyWX2LXPCd18aMze3dS1/9Pkxjexa3N7O+VogYNQn3RaLsjdUaCjEg2+EF9RUsrVJT nxVLPqO/NNXq0YGmj69w/kOvAwCF1mfFJW6eGOh5LBsxRP/fg+FJLPPeTKK/VDsKDCPL HSHSiW+lhae012Q1xGGSMSqX4j/g5ei3SDhLn6ekrglRlIkz9OZZflWjbqgIuVt3c6aq Ez44l7hSD9Gxx52LQOVdPrB6go3Ocxgvj66qY1Ae5VRxvTJP58pJ1y9Cqw/tnPyuZjc4 f10A== X-Gm-Message-State: AOAM533bufbU10s5iZmIqeO3jaQNZnfdNedivAMPD5nEhbDGnJmbIafR +SFEFdt/BwpmabAeQNkIHcw5EWhu19Y= X-Google-Smtp-Source: ABdhPJyyRx/Wvth+f+VqpBNAcWXriNXevrXSUzZER2BZaA/13AoBZsoq+FGpUw8mq6OaM+e0QJEbfg== X-Received: by 2002:adf:da50:0:b0:204:7d8:b654 with SMTP id r16-20020adfda50000000b0020407d8b654mr11964230wrl.607.1647973445565; Tue, 22 Mar 2022 11:24:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o16-20020a05600c4fd000b0038cbd198602sm1291032wmq.34.2022.03.22.11.24.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:24:04 -0700 (PDT) Message-Id: <264397e8bd4b53ecf1d260b99d1895b8d153f82b.1647973380.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:58 +0000 Subject: [PATCH v3 25/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Emit NFC or NFC and NFD spellings of pathnames on macOS. MacOS is Unicode composition insensitive, so NFC and NFD spellings are treated as aliases and collide. While the spelling of pathnames in filesystem events depends upon the underlying filesystem, such as APFS, HFS+ or FAT32, the OS enforces such collisions regardless of filesystem. Teach the daemon to always report the NFC spelling and to report the NFD spelling when stored in that format on the disk. This is slightly more general than "core.precomposeUnicode". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 83d38e8ac6c..823cf63999e 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef) return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); } +/* + * On MacOS we have to adjust for Unicode composition insensitivity + * (where NFC and NFD spellings are not respected). The different + * spellings are essentially aliases regardless of how the path is + * actually stored on the disk. + * + * This is related to "core.precomposeUnicode" (which wants to try + * to hide NFD completely and treat everything as NFC). Here, we + * don't know what the value the client has (or will have) for this + * config setting when they make a query, so assume the worst and + * emit both when the OS gives us an NFD path. + */ +static void my_add_path(struct fsmonitor_batch *batch, const char *path) +{ + char *composed; + + /* add the NFC or NFD path as received from the OS */ + fsmonitor_batch__add_path(batch, path); + + /* if NFD, also add the corresponding NFC spelling */ + composed = (char *)precompose_string_if_needed(path); + if (!composed || composed == path) + return; + + fsmonitor_batch__add_path(batch, composed); + free(composed); +} + + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, rel); + my_add_path(batch, rel); } if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { @@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, tmp.buf); + my_add_path(batch, tmp.buf); } break; From patchwork Tue Mar 22 18:22:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D2E1C433EF for ; Tue, 22 Mar 2022 18:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240463AbiCVS0s (ORCPT ); Tue, 22 Mar 2022 14:26:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240375AbiCVS0U (ORCPT ); Tue, 22 Mar 2022 14:26:20 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A591490FED for ; Tue, 22 Mar 2022 11:24:09 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id r7so11863679wmq.2 for ; Tue, 22 Mar 2022 11:24:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=E8xmQ7dYip7SaSRf9IXmrYL6K0sMohTDrDq1DYcKE54=; b=SG5csjTGLJ7Y03TCQusHTZB7T9TtNWpcYrXPGEGe4nBGN+13s/nSjRm3ecUhwdHhH+ 2yNlF/OjqLnX/kWqECiOGPGGUO9zJiKP7lO6MIfJdQUq+EeONeHr29hfVe2H8haP1EMd O1ZXiLHCt90piZgV3spQOLhEHU4yxXk2YM46lqcD/E5lWU8qQm2W4vVzcSpLWRlFgnAc 6OrInUcWZS1urdAQE0mRIvQOlt2yyyDJFQsxMnlFDf4jCEpenWAJ1XUed22coxp3QVST vJk1gpPSFZauFDrbuxgRJYhIZ4JHdDpXQeyCAtO3UwYNxnyT8FsHLxcJQNM0xlmDWV2g h5aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=E8xmQ7dYip7SaSRf9IXmrYL6K0sMohTDrDq1DYcKE54=; b=7OB/5umwo7XJkMmxk4VatlgloHMXm84WnrEcTZWqhxI8iHDCawweRdV1IoB730KPem QvpSJuMZL+vNiZy4vh+0BdXcWuBuOpFY9wBuJaQRfAr/RbasaHdpT21pn9XhVUQPTSzU knQ7ZzioWCwJcbD8VGC236N0WQpoT3rkpssrFSLlquRpzbVbMzudlYMzoogZCqZ+etOz gHeMzmpSGWcpuCp8ABmGURuWhDJGzr8kHn+3w+ePzDz/JmVDLnvrgWQK2JTCZF+wZxA1 si8Ujxtjx4Mumzz37dahiS1SQnXANI3l5d6/Zb+TiO2KxHbU5Y+6BaGrmCk13YClSEBC aU8w== X-Gm-Message-State: AOAM531Dtj2TMZuwnJc5YytZZZfZqgOXFsDbd60zLot/NroR9DVycd0U WJ0kd0Nx7hHFRcdMhhBgn17THYvvxdQ= X-Google-Smtp-Source: ABdhPJy6Hmu0eRILK0GpRXiBHxpJ3oClQr7iJjgUiBm29TGSBay+/tbwg22aupwILFrHdEfQar9ZBw== X-Received: by 2002:a7b:c20b:0:b0:38c:6ed9:f4f with SMTP id x11-20020a7bc20b000000b0038c6ed90f4fmr5287096wmi.131.1647973448035; Tue, 22 Mar 2022 11:24:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g7-20020a5d64e7000000b00204a13925dcsm2277591wri.11.2022.03.22.11.24.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:24:06 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:22:59 +0000 Subject: [PATCH v3 26/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a set of prereqs to help understand how file names are handled by the filesystem when they contain NFC and NFD Unicode characters. Signed-off-by: Jeff Hostetler --- t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100755 t/lib-unicode-nfc-nfd.sh diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh new file mode 100755 index 00000000000..cf9c26d1e22 --- /dev/null +++ b/t/lib-unicode-nfc-nfd.sh @@ -0,0 +1,167 @@ +# Help detect how Unicode NFC and NFD are handled on the filesystem. + +# A simple character that has a NFD form. +# +# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE +# UTF8(NFC): \xc3 \xa9 +# +# NFD: U+0065 LATIN SMALL LETTER E +# U+0301 COMBINING ACUTE ACCENT +# UTF8(NFD): \x65 + \xcc \x81 +# +utf8_nfc=$(printf "\xc3\xa9") +utf8_nfd=$(printf "\x65\xcc\x81") + +# Is the OS or the filesystem "Unicode composition sensitive"? +# +# That is, does the OS or the filesystem allow files to exist with +# both the NFC and NFD spellings? Or, does the OS/FS lie to us and +# tell us that the NFC and NFD forms are equivalent. +# +# This is or may be independent of what type of filesystem we have, +# since it might be handled by the OS at a layer above the FS. +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a +# collision, for example. +# +# This does not tell us how the Unicode pathname will be spelled +# on disk, but rather only that the two spelling "collide". We +# will examine the actual on disk spelling in a later prereq. +# +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE ' + mkdir trial_${utf8_nfc} && + mkdir trial_${utf8_nfd} +' + +# Is the spelling of an NFC pathname preserved on disk? +# +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD +# and on APFS, NFC paths are preserved. As we have established +# above, this is independent of "composition sensitivity". +# +# 0000000 63 5f c3 a9 +# +# (/usr/bin/od output contains different amount of whitespace +# on different platforms, so we need the wildcards here.) +# +test_lazy_prereq UNICODE_NFC_PRESERVED ' + mkdir c_${utf8_nfc} && + ls | od -t x1 | grep "63 *5f *c3 *a9" +' + +# Is the spelling of an NFD pathname preserved on disk? +# +# 0000000 64 5f 65 cc 81 +# +test_lazy_prereq UNICODE_NFD_PRESERVED ' + mkdir d_${utf8_nfd} && + ls | od -t x1 | grep "64 *5f *65 *cc *81" +' + mkdir c_${utf8_nfc} && + mkdir d_${utf8_nfd} && + +# The following _DOUBLE_ forms are more for my curiosity, +# but there may be quirks lurking when there are multiple +# combining characters in non-canonical order. + +# Unicode also allows multiple combining characters +# that can be decomposed in pieces. +# +# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI +# UTF8(NFC): \xe1 \xbd \xa7 +# +# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA +# U+0342 COMBINING GREEK PERISPOMENI +# UTF8(NFD1): \xe1 \xbd \xa1 + \xcd \x82 +# +# But U+1f61 decomposes into +# NFD2: U+03c9 GREEK SMALL LETTER OMEGA +# U+0314 COMBINING REVERSED COMMA ABOVE +# UTF8(NFD2): \xcf \x89 + \xcc \x94 +# +# Yielding: \xcf \x89 + \xcc \x94 + \xcd \x82 +# +# Note that I've used the canonical ordering of the +# combinining characters. It is also possible to +# swap them. My testing shows that that non-standard +# ordering also causes a collision in mkdir. However, +# the resulting names don't draw correctly on the +# terminal (implying that the on-disk format also has +# them out of order). +# +greek_nfc=$(printf "\xe1\xbd\xa7") +greek_nfd1=$(printf "\xe1\xbd\xa1\xcd\x82") +greek_nfd2=$(printf "\xcf\x89\xcc\x94\xcd\x82") + +# See if a double decomposition also collides. +# +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE ' + mkdir trial_${greek_nfc} && + mkdir trial_${greek_nfd2} +' + +# See if the NFC spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED ' + mkdir c_${greek_nfc} && + ls | od -t x1 | grep "63 *5f *e1 *bd *a7" +' + +# See if the NFD spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED ' + mkdir d_${greek_nfd2} && + ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82" +' + +# The following is for debugging. I found it useful when +# trying to understand the various (OS, FS) quirks WRT +# Unicode and how composition/decomposition is handled. +# For example, when trying to understand how (macOS, APFS) +# and (macOS, HFS) and (macOS, FAT32) compare. +# +# It is rather noisy, so it is disabled by default. +# +if test "$unicode_debug" = "true" +then + if test_have_prereq UNICODE_COMPOSITION_SENSITIVE + then + echo NFC and NFD are distinct on this OS/filesystem. + else + echo NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_NFC_PRESERVED + then + echo NFC maintains original spelling. + else + echo NFC is modified. + fi + + if test_have_prereq UNICODE_NFD_PRESERVED + then + echo NFD maintains original spelling. + else + echo NFD is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE + then + echo DOUBLE NFC and NFD are distinct on this OS/filesystem. + else + echo DOUBLE NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED + then + echo Double NFC maintains original spelling. + else + echo Double NFC is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED + then + echo Double NFD maintains original spelling. + else + echo Double NFD is modified. + fi +fi From patchwork Tue Mar 22 18:23:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12788930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD96C433EF for ; Tue, 22 Mar 2022 18:25:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240373AbiCVS0t (ORCPT ); Tue, 22 Mar 2022 14:26:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240381AbiCVS0U (ORCPT ); Tue, 22 Mar 2022 14:26:20 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2094190FFD for ; Tue, 22 Mar 2022 11:24:11 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id a1so23265248wrh.10 for ; Tue, 22 Mar 2022 11:24:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=hFzBo7NPjbtbf/vT0CCaUkDt4/D28hCpl05V0u76UiM=; b=PuAPlyF29vZvmiStQlJCWIVGKCzXfE4SoaVV2eXBKMXOACsOtgnOH6lJch0I/8EEnF JcyWeMMlwOnanbrsilFgMWdIHM8vhmvrJSLz1ukrLwOas7hhhdHXBZ7cnhwudyMyUeqo 5m/0bGV3efjwTj8vFTkAoIGoggvSqSfcZC/BSWbWs/P7ckx4Kv8Ym7fZiUU5/HJ6kOxg MiZXB8tH2PgC9IShTXfU9fIw8bjFgRIY61ao8K0TmhPqZF87WEh4qOlJMiWfag5amo5w BG/SDpDI4Qb4xbPoidMF2PwrjRYJj0XKbQIHlga5gNGrZDbBvH4RKZyjEzFH/VOFVk4Z Wv+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=hFzBo7NPjbtbf/vT0CCaUkDt4/D28hCpl05V0u76UiM=; b=pwDvuuUn39pc4U98hFk4IjmoY20BpAJiQRX/VA3EplJoRJAQD4x260fZYZGhn7TApR zNl6jH+SLa+H91IbdYgRQnjeofMs/HUif4uyd5CStv/g9z/P/rj66ooEnZs0BEelZDTZ PfKsFt5whpCNX8m2rAPNNkjyGZ/6j0TzXbgdcz7+nghskffKCDUrcQ2srJr7BgmPmgEp fHgMU27VdnfzZXDQ8J8a/ku+kv44Nny9+W+7vEQV5WBxYqyBVBHgLSwW6VrNY+tvqY/Z +yu66akijPM5G6Xzp7Du4Qr0nrVYUup9xNgK4hkbld9fAwYuZxANTw7v7d1cvfxL13/P wKdw== X-Gm-Message-State: AOAM532xHYb0ppOK1d4Hpp3qQAR7HZuSZ6bXvwAaOZpaH+/EB3b2CEdI RPh9EUFrHgflJzrsD/YCqRvSt4YTaWw= X-Google-Smtp-Source: ABdhPJw26sfZct5bgkg3cFjj3fiJZ1NbKVeIC34rC7faIe/MIvOiEtPPsBePiWPtjKL/mHv4jeOHAg== X-Received: by 2002:a5d:6c68:0:b0:205:80f5:4210 with SMTP id r8-20020a5d6c68000000b0020580f54210mr1562914wrz.512.1647973449496; Tue, 22 Mar 2022 11:24:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r13-20020adfbb0d000000b00203e0efdd3bsm16070038wrg.107.2022.03.22.11.24.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Mar 2022 11:24:08 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 22 Mar 2022 18:23:00 +0000 Subject: [PATCH v3 27/27] t7527: test Unicode NFC/NFD handling on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Confirm that the daemon reports events using the on-disk spelling for Unicode NFC/NFD characters. On APFS we still have Unicode aliasing, so we cannot create two files that only differ by NFC/NFD, but the on-disk format preserves the spelling used to create the file. On HFS+ we also have aliasing, but the path is always stored on disk in NFD. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 51cdf05e7ec..126bb2dc8de 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -847,4 +847,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' egrep "^event: abc/def/xyz$" ./insensitive.trace ' +# The variable "unicode_debug" is defined in the following library +# script to dump information about how the (OS, FS) handles Unicode +# composition. Uncomment the following line if you want to enable it. +# +# unicode_debug=true + +. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh" + +# See if the OS or filesystem does NFC/NFD aliasing/munging. +# +# The daemon should err on the side of caution and send BOTH the +# NFC and NFD forms. It does not know the original spelling of +# the pathname (how the user thinks it should be spelled), so +# emit both and let the client decide (when necessary). This is +# similar to "core.precomposeUnicode". +# +test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' ' + test_when_finished "stop_daemon_delete_repo test_unicode" && + + git init test_unicode && + + start_daemon -C test_unicode -tf "$PWD/unicode.trace" && + + # Create a directory using an NFC spelling. + # + mkdir test_unicode/nfc && + mkdir test_unicode/nfc/c_${utf8_nfc} && + + # Create a directory using an NFD spelling. + # + mkdir test_unicode/nfd && + mkdir test_unicode/nfd/d_${utf8_nfd} && + + git -C test_unicode fsmonitor--daemon stop && + + if test_have_prereq UNICODE_NFC_PRESERVED + then + # We should have seen NFC event from OS. + # We should not have synthesized an NFD event. + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace && + egrep -v "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace + else + # We should have seen NFD event from OS. + # We should have synthesized an NFC event. + egrep "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace + fi && + + # We assume UNICODE_NFD_PRESERVED. + # We should have seen explicit NFD from OS. + # We should have synthesized an NFC event. + egrep "^event: nfd/d_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfd/d_${utf8_nfc}/?$" ./unicode.trace +' + test_done