From patchwork Sat May 22 13:56:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 975BAC2B9F2 for ; Sat, 22 May 2021 13:57:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B9C26115C for ; Sat, 22 May 2021 13:57:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230492AbhEVN6j (ORCPT ); Sat, 22 May 2021 09:58:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbhEVN6j (ORCPT ); Sat, 22 May 2021 09:58:39 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CFF6C061574 for ; Sat, 22 May 2021 06:57:13 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id z137-20020a1c7e8f0000b02901774f2a7dc4so8024216wmc.0 for ; Sat, 22 May 2021 06:57:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0G0+zH4+7jfcyLkZGW9tHTKGcBQuzIuX7ETSUbhHH5I=; b=m/+DhRYkg8/WsPU6VloAgWizo7Uvz85kQowoFVm9dW4apRzndLsxO0G1rFGk4hiC7B ZL69hV+98C/r5otXOoPmQdoBjV3W6tp7MwMk0HVsUXGWf5hTl5kIHkWwtpNHHgYXw/Ku 6nKpqCHAqZJQkGlBf0XsO9OTDqNGosppmPAMhqKhSNbCaPOyu/oGBi75574oFZU1pXmV 1CAwrI06NowO7Pg5XH30/51fdC4NdVulwU4Uyd53ILP/fN1pVfDKYVA0KH5uQub07fMB UL3OCWQ+gVLnLl+PMCffDTSrBm92NkriTkSmqlMf4n3DlG0v2Go+FvpWBHIvsgXXHDlm xmig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0G0+zH4+7jfcyLkZGW9tHTKGcBQuzIuX7ETSUbhHH5I=; b=tGnYQM9RHNzrNigohXpp0SBVs0RCx+RbFiAKYksF+qnjaEoevvo5GTnEqHgfgK1bL9 9yG6wGHHxNhRinvEROKwxE9V4vCinajOeUrn59pYxaE2Qz0JshIv7xXA50wl4eoD+GKi lX+OqMomXA9Sus0UqIiC7TURgdYbBXzqqvd3GEoJ/jTXZgmq6XbDS0Idwfwsh5w1+5yu 7JLEKOZg6wLoyZZOXSd7CQiNrNZ8TeItLNl5VCruwqjH09rfOuhipCJKWLAiun/kKPa6 yVoW58GNjSpBLf+XOL7yEDN0coVwFpGmww0im+ZOE9/W/9fNBAbziq0vPpIfE6y7Lz5/ 7Qvw== X-Gm-Message-State: AOAM531dsn0FUJrs3tCkeFkMxBoY9SXwF7aeLrqkhm3dMtLy4IbhTSsF xoseh36ijjK065aTDt1Rz3hHvywMF6I= X-Google-Smtp-Source: ABdhPJzRzXWZlQ/NRzdr1rBJSHvZ/d6sIN9Da7+W1FY/0db8jPfI9dSF6G4VvUB55tDUCojQbXFycg== X-Received: by 2002:a1c:c911:: with SMTP id f17mr13182248wmb.45.1621691831505; Sat, 22 May 2021 06:57:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q13sm5630516wrw.56.2021.05.22.06.57.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:10 -0700 (PDT) Message-Id: <763fa1ee7bb61d236cb0e7e2f01a3a03af87ac2d.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:40 +0000 Subject: [PATCH v2 01/28] simple-ipc: preparations for supporting binary messages. Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add `command_len` argument to the Simple IPC API. In my original Simple IPC API, I assumed that the request would always be a null-terminated string of text characters. The command arg was just a `const char *`. I found a caller that would like to pass a binary command to the daemon, so I want to ammend the Simple IPC API to take `const char *command, size_t command_len` and pass that to the daemon. (Really, the first arg should just be a `void *` or `const unsigned byte *` to make that clearer.) Note, the response side has always been a `struct strbuf` which includes the buffer and length, so we already support returning a binary answer. (Yes, it feels a little weird returning a binary buffer in a `strbuf`, but it works.) Signed-off-by: Jeff Hostetler --- compat/simple-ipc/ipc-unix-socket.c | 14 +++++++----- compat/simple-ipc/ipc-win32.c | 14 +++++++----- simple-ipc.h | 7 ++++-- t/helper/test-simple-ipc.c | 34 +++++++++++++++++++---------- 4 files changed, 46 insertions(+), 23 deletions(-) diff --git a/compat/simple-ipc/ipc-unix-socket.c b/compat/simple-ipc/ipc-unix-socket.c index 38689b278df3..0a2d1c1162b9 100644 --- a/compat/simple-ipc/ipc-unix-socket.c +++ b/compat/simple-ipc/ipc-unix-socket.c @@ -164,7 +164,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection) int ipc_client_send_command_to_connection( struct ipc_client_connection *connection, - const char *message, struct strbuf *answer) + const char *message, size_t message_len, + struct strbuf *answer) { int ret = 0; @@ -172,7 +173,7 @@ int ipc_client_send_command_to_connection( trace2_region_enter("ipc-client", "send-command", NULL); - if (write_packetized_from_buf_no_flush(message, strlen(message), + if (write_packetized_from_buf_no_flush(message, message_len, connection->fd) < 0 || packet_flush_gently(connection->fd) < 0) { ret = error(_("could not send IPC command")); @@ -193,7 +194,8 @@ int ipc_client_send_command_to_connection( int ipc_client_send_command(const char *path, const struct ipc_client_connect_options *options, - const char *message, struct strbuf *answer) + const char *message, size_t message_len, + struct strbuf *answer) { int ret = -1; enum ipc_active_state state; @@ -204,7 +206,9 @@ int ipc_client_send_command(const char *path, if (state != IPC_STATE__LISTENING) return ret; - ret = ipc_client_send_command_to_connection(connection, message, answer); + ret = ipc_client_send_command_to_connection(connection, + message, message_len, + answer); ipc_client_close_connection(connection); @@ -499,7 +503,7 @@ static int worker_thread__do_io( if (ret >= 0) { ret = worker_thread_data->server_data->application_cb( worker_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data); packet_flush_gently(reply_data.fd); } diff --git a/compat/simple-ipc/ipc-win32.c b/compat/simple-ipc/ipc-win32.c index 8f89c02037e3..632fb3c7ea24 100644 --- a/compat/simple-ipc/ipc-win32.c +++ b/compat/simple-ipc/ipc-win32.c @@ -204,7 +204,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection) int ipc_client_send_command_to_connection( struct ipc_client_connection *connection, - const char *message, struct strbuf *answer) + const char *message, size_t message_len, + struct strbuf *answer) { int ret = 0; @@ -212,7 +213,7 @@ int ipc_client_send_command_to_connection( trace2_region_enter("ipc-client", "send-command", NULL); - if (write_packetized_from_buf_no_flush(message, strlen(message), + if (write_packetized_from_buf_no_flush(message, message_len, connection->fd) < 0 || packet_flush_gently(connection->fd) < 0) { ret = error(_("could not send IPC command")); @@ -235,7 +236,8 @@ int ipc_client_send_command_to_connection( int ipc_client_send_command(const char *path, const struct ipc_client_connect_options *options, - const char *message, struct strbuf *response) + const char *message, size_t message_len, + struct strbuf *response) { int ret = -1; enum ipc_active_state state; @@ -246,7 +248,9 @@ int ipc_client_send_command(const char *path, if (state != IPC_STATE__LISTENING) return ret; - ret = ipc_client_send_command_to_connection(connection, message, response); + ret = ipc_client_send_command_to_connection(connection, + message, message_len, + response); ipc_client_close_connection(connection); @@ -454,7 +458,7 @@ static int do_io(struct ipc_server_thread_data *server_thread_data) if (ret >= 0) { ret = server_thread_data->server_data->application_cb( server_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data); packet_flush_gently(reply_data.fd); diff --git a/simple-ipc.h b/simple-ipc.h index dc3606e30bd6..c4d5225b41c2 100644 --- a/simple-ipc.h +++ b/simple-ipc.h @@ -111,7 +111,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection); */ int ipc_client_send_command_to_connection( struct ipc_client_connection *connection, - const char *message, struct strbuf *answer); + const char *message, size_t message_len, + struct strbuf *answer); /* * Used by the client to synchronously connect and send and receive a @@ -123,7 +124,8 @@ int ipc_client_send_command_to_connection( */ int ipc_client_send_command(const char *path, const struct ipc_client_connect_options *options, - const char *message, struct strbuf *answer); + const char *message, size_t message_len, + struct strbuf *answer); /* * Simple IPC Server Side API. @@ -148,6 +150,7 @@ typedef int (ipc_server_reply_cb)(struct ipc_server_reply_data *, */ typedef int (ipc_server_application_cb)(void *application_data, const char *request, + size_t request_len, ipc_server_reply_cb *reply_cb, struct ipc_server_reply_data *reply_data); diff --git a/t/helper/test-simple-ipc.c b/t/helper/test-simple-ipc.c index 42040ef81b1e..913451807509 100644 --- a/t/helper/test-simple-ipc.c +++ b/t/helper/test-simple-ipc.c @@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb, /* * The client sent a command followed by a (possibly very) large buffer. */ -static int app__sendbytes_command(const char *received, +static int app__sendbytes_command(const char *received, size_t received_len, ipc_server_reply_cb *reply_cb, struct ipc_server_reply_data *reply_data) { @@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received, int errs = 0; int ret; + /* + * The test is setup to send: + * "sendbytes" SP + */ + if (received_len < strlen("sendbytes ")) + BUG("received_len is short in app__sendbytes_command"); + if (skip_prefix(received, "sendbytes ", &p)) len_ballast = strlen(p); @@ -160,7 +167,7 @@ static ipc_server_application_cb test_app_cb; * by this application. */ static int test_app_cb(void *application_data, - const char *command, + const char *command, size_t command_len, ipc_server_reply_cb *reply_cb, struct ipc_server_reply_data *reply_data) { @@ -173,7 +180,7 @@ static int test_app_cb(void *application_data, if (application_data != (void*)&my_app_data) BUG("application_cb: application_data pointer wrong"); - if (!strcmp(command, "quit")) { + if (command_len == 4 && !strncmp(command, "quit", 4)) { /* * The client sent a "quit" command. This is an async * request for the server to shutdown. @@ -193,22 +200,23 @@ static int test_app_cb(void *application_data, return SIMPLE_IPC_QUIT; } - if (!strcmp(command, "ping")) { + if (command_len == 4 && !strncmp(command, "ping", 4)) { const char *answer = "pong"; return reply_cb(reply_data, answer, strlen(answer)); } - if (!strcmp(command, "big")) + if (command_len == 3 && !strncmp(command, "big", 3)) return app__big_command(reply_cb, reply_data); - if (!strcmp(command, "chunk")) + if (command_len == 5 && !strncmp(command, "chunk", 5)) return app__chunk_command(reply_cb, reply_data); - if (!strcmp(command, "slow")) + if (command_len == 4 && !strncmp(command, "slow", 4)) return app__slow_command(reply_cb, reply_data); - if (starts_with(command, "sendbytes ")) - return app__sendbytes_command(command, reply_cb, reply_data); + if (command_len >= 10 && starts_with(command, "sendbytes ")) + return app__sendbytes_command(command, command_len, + reply_cb, reply_data); return app__unhandled_command(command, reply_cb, reply_data); } @@ -488,7 +496,9 @@ static int client__send_ipc(void) options.wait_if_busy = 1; options.wait_if_not_found = 0; - if (!ipc_client_send_command(cl_args.path, &options, command, &buf)) { + if (!ipc_client_send_command(cl_args.path, &options, + command, strlen(command), + &buf)) { if (buf.len) { printf("%s\n", buf.buf); fflush(stdout); @@ -556,7 +566,9 @@ static int do_sendbytes(int bytecount, char byte, const char *path, strbuf_addstr(&buf_send, "sendbytes "); strbuf_addchars(&buf_send, byte, bytecount); - if (!ipc_client_send_command(path, options, buf_send.buf, &buf_resp)) { + if (!ipc_client_send_command(path, options, + buf_send.buf, buf_send.len, + &buf_resp)) { strbuf_rtrim(&buf_resp); printf("sent:%c%08d %s\n", byte, bytecount, buf_resp.buf); fflush(stdout); From patchwork Sat May 22 13:56:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B41EC2B9F2 for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5278F6115C for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231206AbhEVN6n (ORCPT ); Sat, 22 May 2021 09:58:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231142AbhEVN6k (ORCPT ); Sat, 22 May 2021 09:58:40 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B529C0613ED for ; Sat, 22 May 2021 06:57:14 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id a4so23771771wrr.2 for ; Sat, 22 May 2021 06:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=twSYMslETglXq2k4sozymNKhyot6vWiv7GZce15S1Ec=; b=LA4SOGuGxfPT8TQesKS+MBidL1/Fcfq469dyhMNgPk/rb5eTBiVexEpe9xXd8CUdhY G5V5lv5ktcS8GZTILPYomiHF/XIf/DtPviBLbgMjZniJI0LXLHQL7RoKMU10kA6F+K9m wsxsMiNGrcfgSSsR/UQSQCmKO0hUDoKrPPK7T3kY1rqZ51g9IEkiSCxuwp1UZgaQ+f5D zmg9wHy7R5U6Bk/UfwDoUwYUcI4MUCpxCjaHfJs0+D3INNtMVNb84D6/X2ph1V5oziov Gxo0DHG9HsoLKbov5yrBXchdbz/6Tv5VjskfPvih+2RANB7OAN6hyo+twX6k09Q9BLt1 EdnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=twSYMslETglXq2k4sozymNKhyot6vWiv7GZce15S1Ec=; b=f5y9hSi2Ce4wDZkqI1zRlLihOFgYCl9vFeCItaA3E8Gkf8vRK7xHKyHtNPrVxwpb2O rntguv6XLXUZEdlc6IyfeRW8pqDUxMkoMZ4ygarbCn9isr9pQHdOwl6Hw2thVwrdl9TP ViegszaJFvV6z4fYVmaXDvyKVSbAVot6gC+0PbJqkYw4sg+ELaWmUL/LLI3UnK9Ve11d mlypJWstOne7+7hhN4W4mWp5ddY3y0iEwwBrtx+1lS+uHVFf1IPJg7zn7Te4ZbLcOcRm ty/HZwhiKqSkiLYL5ljXS5Af2B5JDsGGs/y/36GBTO2oVBXlsIW6K3PcgaoHG53YlsD5 cgUA== X-Gm-Message-State: AOAM5323FZaUtMqQezDDoVuS28aQU2P+FcCgV93XwQDEjm/hSe/6sDNg 2eeNLWnYguVxXZJ4LcaU0RicQQhs3kw= X-Google-Smtp-Source: ABdhPJysfEMQNd7taRi8lQrtz7LtbkwyYWf8IkjP3k5Dix0WTquwHP2iAxEJc7pSlUsIIVMnakqHhQ== X-Received: by 2002:a05:6000:18ad:: with SMTP id b13mr14428979wri.343.1621691832123; Sat, 22 May 2021 06:57:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm5412040wrt.97.2021.05.22.06.57.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:11 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:41 +0000 Subject: [PATCH v2 02/28] fsmonitor--daemon: man page Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a manual page describing the `git fsmonitor--daemon` feature. Signed-off-by: Jeff Hostetler --- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/git-fsmonitor--daemon.txt diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt new file mode 100644 index 000000000000..154e7684daae --- /dev/null +++ b/Documentation/git-fsmonitor--daemon.txt @@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does +not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite From patchwork Sat May 22 13:56:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF52EC2B9F2 for ; Sat, 22 May 2021 13:57:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 92DFE6115C for ; Sat, 22 May 2021 13:57:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231174AbhEVN6m (ORCPT ); Sat, 22 May 2021 09:58:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230295AbhEVN6k (ORCPT ); Sat, 22 May 2021 09:58:40 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E023C06174A for ; Sat, 22 May 2021 06:57:14 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id b7so11957155wmh.5 for ; Sat, 22 May 2021 06:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=V/9CxnpqDwArhVx2MQERVzriRRP+9XRB4pNi0Oe7LQI=; b=IMdGdW8NiaJjjNlS64UyXF8eOSYoqq2yxfG4BX067Bh89OxMybvfXcO1Pye2KSRRu6 jZgtUuLEzu98ZKOI+5dOoT9V3mql5npGQdQMoQWBQRviPvCvjZiyLDNwCmi5FrVf6ghw Tl942tgKxaHCtsbHAS7dUjbU4hUIq8u02vdm/ENVDtBRwOn2VcsJkHOQw4TYcPdksw4q ZHRinwHmO+mfhZky0H4aHvdN5IEnsjmzp9DgJxu7o22vLqq805ZbsQTZFvNR3B3qhAaR vyWJPHOxsNEaJDbKv4TvPYeKCTsKVxCP9gnvTiCCQZKGhlVzvkwEIKupXJAWy10HiXfp P5QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=V/9CxnpqDwArhVx2MQERVzriRRP+9XRB4pNi0Oe7LQI=; b=XqyuQCPjcoc4thk0smu6JlLjg6EoYukLeCNQFc8PlS9EtllqytDJrG4RIqlduXiwDb SShwfAosdc3WKO3TuGVZysaiJeQLYYXk11eBoONRItUim7LKVv+IwKbMtMQu4oxm0v4s /N/3ZB/uQvpi7GKoyCa6+vrGCbOcXs7MysvV5Cr7+yYfubbIWWInhtw/kzFa8ULM5sWY zP4DquDcuhbm/dacbUVXmM6JS0rGExuo/hIAxPU+wQTE/ehsP4YomlAbyJ1nTmxGYx2A mcDvBdb6MDZPIbG+AFVE6b8oChSmStwsnzB8ENKkp/+dM232s/Knagub6PEmpjocTfMK uSTg== X-Gm-Message-State: AOAM532MtcjwC3r/qdUfGfVJVzvaoya8odMINZ0AZbTbAdhb8kF17n11 I4dWJZSFNQeRLtJbq7qxrz7n9jqUBvk= X-Google-Smtp-Source: ABdhPJzZ4tcBnDAgkG5b20oV0dy2TWO4LBNF1zJhM3qr++tx5xhaHr8QztcI2iWTkYV10djZZsuoBA== X-Received: by 2002:a7b:c85a:: with SMTP id c26mr13638524wml.131.1621691832735; Sat, 22 May 2021 06:57:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c15sm5848155wrd.49.2021.05.22.06.57.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:12 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:42 +0000 Subject: [PATCH v2 03/28] fsmonitor--daemon: update fsmonitor documentation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and pointers to `Watchman` to mention the new built-in `fsmonitor--daemon`. Signed-off-by: Jeff Hostetler --- Documentation/config/core.txt | 56 ++++++++++++++++++++++-------- Documentation/git-update-index.txt | 27 +++++++------- Documentation/githooks.txt | 3 +- 3 files changed, 59 insertions(+), 27 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a15..4f6e519bc025 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -62,22 +62,50 @@ core.protectNTFS:: Defaults to `true` on Windows, and `false` elsewhere. core.fsmonitor:: - If set, the value of this variable is used as a command which - will identify all files that may have changed since the - requested date/time. This information is used to speed up git by - avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. + If set, this variable contains the pathname of the "fsmonitor" + hook command. ++ +This hook command is used to identify all files that may have changed +since the requested date/time. This information is used to speed up +git by avoiding unnecessary scanning of files that have not changed. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the protocol version to be used when invoking the + "fsmonitor" hook. ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like Watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). + +core.useBuiltinFSMonitor:: + If set to true, enable the built-in file system monitor + daemon for this working directory (linkgit:git-fsmonitor--daemon[1]). ++ +Like hook-based file system monitors, the built-in file system monitor +can speed up Git commands that need to refresh the Git index +(e.g. `git status`) in a working directory with many files. The +built-in monitor eliminates the need to install and maintain an +external third-party tool. ++ +The built-in file system monitor is currently available only on a +limited set of supported platforms. Currently, this includes Windows +and MacOS. ++ +Note: if this config setting is set to `true`, the values of +`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored. core.trustctime:: If false, the ctime differences between the index and the diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt index 2853f168d976..c7c31b3fcf9c 100644 --- a/Documentation/git-update-index.txt +++ b/Documentation/git-update-index.txt @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files. @@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory looking for new files. If you want to enable (or disable) this feature, it is easier to use -the `core.fsmonitor` configuration variable (see -linkgit:git-config[1]) than using the `--fsmonitor` option to -`git update-index` in each repository, especially if you want to do so -across all repositories you use, because you can set the configuration -variable in your `$HOME/.gitconfig` just once and have it affect all -repositories you touch. - -When the `core.fsmonitor` configuration variable is changed, the -file system monitor is added to or removed from the index the next time -a command reads the index. When `--[no-]fsmonitor` are used, the file -system monitor is immediately added to or removed from the index. +the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable (see linkgit:git-config[1]) than using the `--fsmonitor` +option to `git update-index` in each repository, especially if you +want to do so across all repositories you use, because you can set the +configuration variable in your `$HOME/.gitconfig` just once and have +it affect all repositories you touch. + +When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable is changed, the file system monitor is added to or removed +from the index the next time a command reads the index. When +`--[no-]fsmonitor` are used, the file system monitor is immediately +added to or removed from the index. CONFIGURATION ------------- diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt index b51959ff9418..b7d5e926f7b0 100644 --- a/Documentation/githooks.txt +++ b/Documentation/githooks.txt @@ -593,7 +593,8 @@ fsmonitor-watchman This hook is invoked when the configuration option `core.fsmonitor` is set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2` -depending on the version of the hook to use. +depending on the version of the hook to use, unless overridden via +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]). Version 1 takes two arguments, a version (1) and the time in elapsed nanoseconds since midnight, January 1, 1970. From patchwork Sat May 22 13:56:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9543C4707E for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 844D3610A6 for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231233AbhEVN6p (ORCPT ); Sat, 22 May 2021 09:58:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231152AbhEVN6k (ORCPT ); Sat, 22 May 2021 09:58:40 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C29D1C06138B for ; Sat, 22 May 2021 06:57:14 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id t206so12573984wmf.0 for ; Sat, 22 May 2021 06:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=N+/eQ6PgtPRJ2aIDbNELd+uUhQT5Wu0vpZg6/GGob7o=; b=GiRV0uitXmy8pQAepg0p/+YwufQFj9V09IX6swdcKons0gP6Y260jGvEPI2ytJml4E IOTyXrx+Iz4vDW7vVG7FxEZ372W4fBQYBtJpz5P4C9t0LanO4dqVOrT+D7tlS5XeIn24 X9FCd6wm3uYUtduGj/hzXZcgGJmnIF7a1UFq8C34HUpko+rOsBy4C5e/VXcAQnUnwbXX fOGnEKFG81jJh/eqeZUpsa7eIb3I3X5MW7m/nvxmrv/5+RsIno7lPY7LyX8o5fAicL9K deqm0LsIXVWPRrkXiq6uAY3O2e9ImKHOmpQl8z09UYMyaxV8ckwl+QLxQqRz0toER2Hu Ux6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=N+/eQ6PgtPRJ2aIDbNELd+uUhQT5Wu0vpZg6/GGob7o=; b=C1Sk6qPbFmklIs5A+QzAPWa9XcOmGVXkaLVCWa5uAq8rdB2XGfz1uiJfPetzbhNa7U DyxHDTUKY3KdB7V6OBK25UW/uUWhZAOjdWBe+vd17vqu03J/JquzlPcri/OkZlpcdzKS dTyYS3a0KFHsG9I69VfwVHeLPjr4Z1/Rw286ZScuW+11OD5/HZasTVFActJF7uXy2cQS DNCSznchxcIbcNe9/BmQmoKpIzcimRwkxlwvi50lhwhDxfkVdNOHyq2jIQVY9P8eW1Ub UmRG9oKoQsvcjWl3oWAXMan0qsuWZJjn0DykBnCp7fUdvSmCVbuIbcSU0Ug9iXoS0sZB KW7A== X-Gm-Message-State: AOAM530UEs2eJtZzr+wfX1DpUvEIj66PMndLhI9QPDGcoHI3bFn7MhOz TxfBZz+Xn9MQjI+PPNeDrwr7gDQPIqo= X-Google-Smtp-Source: ABdhPJxtXpH5fpLg8RPxJRgpix/RwOkhbS4XOLQ6VG3e+QF1FWf7IjF02Xg5H6CXPOoENSuwvJbzvQ== X-Received: by 2002:a1c:7402:: with SMTP id p2mr13201378wmc.88.1621691833320; Sat, 22 May 2021 06:57:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o21sm2555101wmr.44.2021.05.22.06.57.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:13 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:43 +0000 Subject: [PATCH v2 04/28] fsmonitor-ipc: create client routines for git-fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create fsmonitor_ipc__*() client routines to spawn the built-in file system monitor daemon and send it an IPC request using the `Simple IPC` API. Stub in empty fsmonitor_ipc__*() functions for unsupported platforms. Signed-off-by: Jeff Hostetler --- Makefile | 1 + fsmonitor-ipc.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++ fsmonitor-ipc.h | 48 +++++++++++++ 3 files changed, 228 insertions(+) create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h diff --git a/Makefile b/Makefile index 21c0bf16672b..23f3b9890acd 100644 --- a/Makefile +++ b/Makefile @@ -892,6 +892,7 @@ LIB_OBJS += fetch-pack.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o +LIB_OBJS += fsmonitor-ipc.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c new file mode 100644 index 000000000000..e62901a85b5d --- /dev/null +++ b/fsmonitor-ipc.c @@ -0,0 +1,179 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "simple-ipc.h" +#include "fsmonitor-ipc.h" +#include "run-command.h" +#include "strbuf.h" +#include "trace2.h" + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int fsmonitor_ipc__is_supported(void) +{ + return 1; +} + +GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc") + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return ipc_get_active_state(fsmonitor_ipc__get_path()); +} + +static int spawn_daemon(void) +{ + const char *args[] = { "fsmonitor--daemon", "start", NULL }; + + return run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD, + "fsmonitor"); +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + int ret = -1; + int tried_to_spawn = 0; + enum ipc_active_state state = IPC_STATE__OTHER_ERROR; + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + trace2_region_enter("fsm_client", "query", NULL); + + trace2_data_string("fsm_client", NULL, "query/command", + since_token); + +try_again: + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + + switch (state) { + case IPC_STATE__LISTENING: + ret = ipc_client_send_command_to_connection( + connection, since_token, strlen(since_token), answer); + ipc_client_close_connection(connection); + + trace2_data_intmax("fsm_client", NULL, + "query/response-length", answer->len); + + if (fsmonitor_is_trivial_response(answer)) + trace2_data_intmax("fsm_client", NULL, + "query/trivial-response", 1); + + goto done; + + case IPC_STATE__NOT_LISTENING: + ret = error(_("fsmonitor_ipc__send_query: daemon not available")); + goto done; + + case IPC_STATE__PATH_NOT_FOUND: + if (tried_to_spawn) + goto done; + + tried_to_spawn++; + if (spawn_daemon()) + goto done; + + /* + * Try again, but this time give the daemon a chance to + * actually create the pipe/socket. + * + * Granted, the daemon just started so it can't possibly have + * any FS cached yet, so we'll always get a trivial answer. + * BUT the answer should include a new token that can serve + * as the basis for subsequent requests. + */ + options.wait_if_not_found = 1; + goto try_again; + + case IPC_STATE__INVALID_PATH: + ret = error(_("fsmonitor_ipc__send_query: invalid path '%s'"), + fsmonitor_ipc__get_path()); + goto done; + + case IPC_STATE__OTHER_ERROR: + default: + ret = error(_("fsmonitor_ipc__send_query: unspecified error on '%s'"), + fsmonitor_ipc__get_path()); + goto done; + } + +done: + trace2_region_leave("fsm_client", "query", NULL); + + return ret; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + int ret; + enum ipc_active_state state; + + strbuf_reset(answer); + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + if (state != IPC_STATE__LISTENING) { + die("fsmonitor--daemon is not running"); + return -1; + } + + ret = ipc_client_send_command_to_connection(connection, + command, strlen(command), + answer); + ipc_client_close_connection(connection); + + if (ret == -1) { + die("could not send '%s' command to fsmonitor--daemon", + command); + return -1; + } + + return 0; +} + +#else + +/* + * A trivial implementation of the fsmonitor_ipc__ API for unsupported + * platforms. + */ + +int fsmonitor_ipc__is_supported(void) +{ + return 0; +} + +const char *fsmonitor_ipc__get_path(void) +{ + return NULL; +} + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return IPC_STATE__OTHER_ERROR; +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + return -1; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + return -1; +} + +#endif diff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h new file mode 100644 index 000000000000..837c5e5b64ad --- /dev/null +++ b/fsmonitor-ipc.h @@ -0,0 +1,48 @@ +#ifndef FSMONITOR_IPC_H +#define FSMONITOR_IPC_H + +/* + * Returns true if built-in file system monitor daemon is defined + * for this platform. + */ +int fsmonitor_ipc__is_supported(void); + +/* + * Returns the pathname to the IPC named pipe or Unix domain socket + * where a `git-fsmonitor--daemon` process will listen. This is a + * per-worktree value. + * + * Returns NULL if the daemon is not supported on this platform. + */ +const char *fsmonitor_ipc__get_path(void); + +/* + * Try to determine whether there is a `git-fsmonitor--daemon` process + * listening on the IPC pipe/socket. + */ +enum ipc_active_state fsmonitor_ipc__get_state(void); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc + * and ask for the set of changed files since the given token. + * + * This DOES NOT use the hook interface. + * + * Spawn a daemon process in the background if necessary. + * + * Returns -1 on error; 0 on success. + */ +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc and + * send a command verb. If no daemon is available, we DO NOT try to + * start one. + * + * Returns -1 on error; 0 on success. + */ +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer); + +#endif /* FSMONITOR_IPC_H */ From patchwork Sat May 22 13:56:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7670C4707F for ; Sat, 22 May 2021 13:57:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A3BD3610A6 for ; Sat, 22 May 2021 13:57:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231178AbhEVN6n (ORCPT ); Sat, 22 May 2021 09:58:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230506AbhEVN6k (ORCPT ); Sat, 22 May 2021 09:58:40 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45699C061574 for ; Sat, 22 May 2021 06:57:15 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id n2so23815956wrm.0 for ; Sat, 22 May 2021 06:57:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vXmaiC8x6u8wcpNaHM4nwJMmiaeQYsIMT/cHQQVEbVc=; b=NLREgGNGLmXlfxv8M6Po+ml78Bi2rOnpMshI6xuG6bD8kGhqhM3l9LRcxSNBJcEZqm vGbsQxmQBxpwqCv7eMbzf3xbu+KE5N28mRLhaluNBZSyeyZB9LZnSr8utWS17Q08GZGj s8eiIWcCWv50hxTj2ujKTl3QIW+PK7CFwNHIwGRFewc22ici0dzG44WTcdGJTtEsCNjq R4AHnQd1pJWyI7C2ubNhx4W4haimpV0lSvLUsUnL6um9UfAYD3JhdEtg1/DVtqQePES5 /YD8zW1UexWemYGockznNFk1wmEmYYNOXG4tkhJwOzsB5KA7dle178ZCSGCvq3EwmxNG pvvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vXmaiC8x6u8wcpNaHM4nwJMmiaeQYsIMT/cHQQVEbVc=; b=n9NyQwoa1ncDviD3Z7DT0/UiF5BJZ0zo19dwri3rtq5jBPjc/0SBn6Byp05sHje+DW QsXNhhsX4D7Qk23X6/odk6+SzMPMQIaV1aQYsnUis8ojLypzae3j1Uc7cgR+WN1iDlH5 mTcHAxpsnuI5xEAhNeeCxPTrXvRXU8/mjpFdgbG2Y7jzxkmdsydMhGS7U9ih+tWiy7N1 j3W3ffG2wwySGdUEUiQ4VPIu9LqLl9n3JJMdcSCnSkBhYjP2svVruZj6+qG8WDSTP2Mh UwI7oFvCp7MFfSylAmjHC1d5rTXNWLrE4ECvIfPsEtvjlofI5foGV4zoHjnlc06SX1Vq QtsA== X-Gm-Message-State: AOAM530Rl6zdO7STDOLrpim+kM6CZeVt3XAgm4GLuIrHSgvMg6PS6opu oy+inn0+SnU/R0qygs1enYrvSmTCntQ= X-Google-Smtp-Source: ABdhPJwF0aox/05fQ+wjB0ZKHdhzn1wMZHT8ZBmI+IgnjMHlzUGw3ZbFUZkCArJvOWevxQ9Zcf/IdQ== X-Received: by 2002:a5d:4351:: with SMTP id u17mr14044593wrr.47.1621691833887; Sat, 22 May 2021 06:57:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n15sm5538984wrr.20.2021.05.22.06.57.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:13 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:44 +0000 Subject: [PATCH v2 05/28] help: include fsmonitor--daemon feature flag in version info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add the "feature: fsmonitor--daemon" message to the output of `git version --build-options`. This allows users to know if the built-in fsmonitor feature is supported on their platform. Signed-off-by: Jeff Hostetler --- help.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/help.c b/help.c index 3c3bdec21356..e22ba1d246a5 100644 --- a/help.c +++ b/help.c @@ -11,6 +11,7 @@ #include "version.h" #include "refs.h" #include "parse-options.h" +#include "fsmonitor-ipc.h" struct category_description { uint32_t category; @@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options) strbuf_addf(buf, "sizeof-size_t: %d\n", (int)sizeof(size_t)); strbuf_addf(buf, "shell-path: %s\n", SHELL_PATH); /* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */ + + if (fsmonitor_ipc__is_supported()) + strbuf_addstr(buf, "feature: fsmonitor--daemon\n"); } } From patchwork Sat May 22 13:56:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12274555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B91AFC2B9FB for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 96F0261244 for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231250AbhEVN6q (ORCPT ); Sat, 22 May 2021 09:58:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231159AbhEVN6m (ORCPT ); Sat, 22 May 2021 09:58:42 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1E1CC06138C for ; Sat, 22 May 2021 06:57:15 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id b7so11957184wmh.5 for ; Sat, 22 May 2021 06:57:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+6vypO9YNjV56bGEeDxuv/2CtEcB8b9KfQqp5m+Aabk=; b=RBCzG4u80jlFFFDiohN+tkbBOp0jVaY5dG2IrnR4P8UIxy+ML+UvdAqBmF+2R6++BU NTzOqsuvt/YoRpekHS6RrUEQ/e80v5LkPtpJSqUsO6vz5MvzYhjyGFGCyvSjSaCqHsHJ Pp0sbEilnaDhKFDcED6ulxOVxZgaHeAdg2q63tlC+MmFGfLB9vAV4Ohxl5AY1u5vIAL5 lzCCfW/uxIuMB5QRw7PRnxoHWVSeOFsDTkKn7ndRrB8Z86aS0AIAQZsClX5UU77mPiAU lqUdVWbTT2QzF/gJzEihnXXnNIgT1AZxdrIPlSUOq1lZtuD8WfgDjQx0VtFt2T8GDwk8 3HNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+6vypO9YNjV56bGEeDxuv/2CtEcB8b9KfQqp5m+Aabk=; b=eCs07GRyizTbNGf99wKpKnQjAtHPgPVECWJgnkXuu5ncoq5+9OzYybXGe4jCzCosDj oaHeK5JkdxTKEB/TcgGiS3TPogqO+cosFC+096xbTmkwW+wEobZhMA1nL01ebCx3x6c9 +CXqt9PEuCvq/s4N0rfUJsM8Fsv5aqHR9xdOS8XV0J2tJFEo7HjP85z0GY9yvMYONLt9 70R1H8ifZxbE9hPlzJ91a8Dh/Gl0NdLKZxIso/pvuWl034T0QGQLf1cn92oCJBfsxNkV cul2xkgYJZqs19LqdmXrqinAIcQS8w8XPsMBbLExdpQLvGb9WhtlGxPT4wuJaSxs5tTU 450Q== X-Gm-Message-State: AOAM533v+vy5nkEMy2E78+N5Sn0PnownGa6EepHb8VioDWrYquJ1yw6O aqo2SLqC7IjKO8ObuWZU4VNJpwpkeNQ= X-Google-Smtp-Source: ABdhPJw4MsbnOEMQv65E0mYkz+WZMibOAqwUvi+8/t8O2WfLTuwOTurffQthh/wK7FFjQkUvP4USsg== X-Received: by 2002:a1c:2786:: with SMTP id n128mr13730418wmn.82.1621691834447; Sat, 22 May 2021 06:57:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r11sm5596618wrp.46.2021.05.22.06.57.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:14 -0700 (PDT) Message-Id: <67bcf57f5948038a86c7aa99a6f56655a33eb045.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:45 +0000 Subject: [PATCH v2 06/28] config: FSMonitor is repository-specific Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin This commit refactors `git_config_get_fsmonitor()` into the `repo_*()` form that takes a parameter `struct repository *r`. That change prepares for the upcoming `core.useBuiltinFSMonitor` flag which will be stored in the `repo_settings` struct. Signed-off-by: Johannes Schindelin --- builtin/update-index.c | 4 ++-- config.c | 4 ++-- config.h | 2 +- fsmonitor.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/builtin/update-index.c b/builtin/update-index.c index 79087bccea4b..84793df8b2b6 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1214,14 +1214,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) } if (fsmonitor > 0) { - if (git_config_get_fsmonitor() == 0) + if (repo_config_get_fsmonitor(r) == 0) warning(_("core.fsmonitor is unset; " "set it if you really want to " "enable fsmonitor")); add_fsmonitor(&the_index); report(_("fsmonitor enabled")); } else if (!fsmonitor) { - if (git_config_get_fsmonitor() == 1) + if (repo_config_get_fsmonitor(r) == 1) warning(_("core.fsmonitor is set; " "remove it if you really want to " "disable fsmonitor")); diff --git a/config.c b/config.c index 870d9534defc..a896f44cba1f 100644 --- a/config.c +++ b/config.c @@ -2499,9 +2499,9 @@ int git_config_get_max_percent_split_change(void) return -1; /* default value */ } -int git_config_get_fsmonitor(void) +int repo_config_get_fsmonitor(struct repository *r) { - if (git_config_get_pathname("core.fsmonitor", &core_fsmonitor)) + if (repo_config_get_pathname(r, "core.fsmonitor", &core_fsmonitor)) core_fsmonitor = getenv("GIT_TEST_FSMONITOR"); if (core_fsmonitor && !*core_fsmonitor) diff --git a/config.h b/config.h index 19a9adbaa9a3..3139de81d986 100644 --- a/config.h +++ b/config.h @@ -607,7 +607,7 @@ int git_config_get_index_threads(int *dest); int git_config_get_untracked_cache(void); int git_config_get_split_index(void); int git_config_get_max_percent_split_change(void); -int git_config_get_fsmonitor(void); +int repo_config_get_fsmonitor(struct repository *r); /* This dies if the configured or default date is in the future */ int git_config_get_expiry(const char *key, const char **output); diff --git a/fsmonitor.c b/fsmonitor.c index ab9bfc60b34e..9c9b2abc9414 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -411,7 +411,7 @@ void remove_fsmonitor(struct index_state *istate) void tweak_fsmonitor(struct index_state *istate) { unsigned int i; - int fsmonitor_enabled = git_config_get_fsmonitor(); + int fsmonitor_enabled = repo_config_get_fsmonitor(istate->repo ? istate->repo : the_repository); if (istate->fsmonitor_dirty) { if (fsmonitor_enabled) { From patchwork Sat May 22 13:56:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12274561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEB92C47082 for ; Sat, 22 May 2021 13:57:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CC5C26115C for ; Sat, 22 May 2021 13:57:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231269AbhEVN6u (ORCPT ); Sat, 22 May 2021 09:58:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231185AbhEVN6n (ORCPT ); Sat, 22 May 2021 09:58:43 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7517CC06138D for ; Sat, 22 May 2021 06:57:16 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id o127so12549096wmo.4 for ; Sat, 22 May 2021 06:57:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2viLlPo1a4qnDiU6SmThgEJ8hz95NNWBvJdvSCqw4xc=; b=AfJXg5h2wBbTPbLu8jAy3nO5kJClxcqLMHm9Gdo/JxXkmRK/91ksnLLOWq9cdToTlk NZueISGKcOjyq2H3EzZLw5P/xcsLfq0A4erX+j+78He1bch4xGdoo/vGb6gIPK1XsIbA oqOjKQUa/yyvXR28QK8I2TtvwH34j8efBuzMhWcT/1AHIjgomZ+8Xwm15rV+/PPUp+Ki vfqGU8B84VQhXa5B62Rw62e55vW/7oim3io7amU0v0GE2q7rNnqVeIzKYJ/5nkVBonJq TA9w9F1xo/d9zuuHwLmM59Bfk00hyaQiN4bn9wFVeQvXf7iuykhgPNASh9+F06RmIvOB 0Xng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2viLlPo1a4qnDiU6SmThgEJ8hz95NNWBvJdvSCqw4xc=; b=gSQMvcu+PVpl80VW9g6Ye4vTdu+pso3OcNWDkDaeaCjb+/5vQUm8itvbAe34Y4z7Nv TfmagKY1MG2nQYGIimuyA/hSkDvgJDnVa5jTgNUXiiGN58E1MXoEP2lzscgaVW7KRHRN Sr+824NiGAJvH1iyDijXMZwtf6njWRMc23NCuIlmq6w/aUDwcQQIDYmVlGvNsm4MpMGF +7cZPMukstL6PPg2kqXpA2bWm09YL47He35emJryJjkF6N7+4IOyV4wTF52K0HXUlzWE zp7izNRP9nFiriQ9e9J2P6gpvlkpeTgWQOEdPawLNnX/jUIx6DmyXhxodSN5TFk+OGkc 6agQ== X-Gm-Message-State: AOAM5326rkG2KAj4xmJRoTed2AbxXkmHKwiUBbJRUhyaXccZzy/a9h1/ aCIz1Z0U6G03LyE2lJOPbz6n4l5or/I= X-Google-Smtp-Source: ABdhPJxm2lbqZzQEgBPehipBF1yBuXI4jxARtdgZhw755y9rdiCu69Bk1xuKYsf+FyJAb6CQ0b3wjA== X-Received: by 2002:a7b:c30f:: with SMTP id k15mr13593149wmj.128.1621691835047; Sat, 22 May 2021 06:57:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a11sm5505562wrr.48.2021.05.22.06.57.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:14 -0700 (PDT) Message-Id: <7e097cebc14328bc1ec6e30fc164270e0889ff06.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:46 +0000 Subject: [PATCH v2 07/28] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin Use simple IPC to directly communicate with the new builtin file system monitor daemon. Define a new config setting `core.useBuiltinFSMonitor` to enable the builtin file system monitor. The `core.fsmonitor` setting has already been defined as a HOOK pathname. Historically, this has been set to a HOOK script that will talk with Watchman. For compatibility reasons, we do not want to overload that definition (and cause problems if users have multiple versions of Git installed). Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- config.c | 5 +++++ fsmonitor.c | 43 +++++++++++++++++++++++++++++++++++++++++++ repo-settings.c | 3 +++ repository.h | 2 ++ 4 files changed, 53 insertions(+) diff --git a/config.c b/config.c index a896f44cba1f..c82f40c22b43 100644 --- a/config.c +++ b/config.c @@ -2501,6 +2501,11 @@ int git_config_get_max_percent_split_change(void) int repo_config_get_fsmonitor(struct repository *r) { + if (r->settings.use_builtin_fsmonitor > 0) { + core_fsmonitor = "(built-in daemon)"; + return 1; + } + if (repo_config_get_pathname(r, "core.fsmonitor", &core_fsmonitor)) core_fsmonitor = getenv("GIT_TEST_FSMONITOR"); diff --git a/fsmonitor.c b/fsmonitor.c index 9c9b2abc9414..c6d3c34ad78e 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -3,6 +3,7 @@ #include "dir.h" #include "ewah/ewok.h" #include "fsmonitor.h" +#include "fsmonitor-ipc.h" #include "run-command.h" #include "strbuf.h" @@ -231,6 +232,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) void refresh_fsmonitor(struct index_state *istate) { + struct repository *r = istate->repo ? istate->repo : the_repository; struct strbuf query_result = STRBUF_INIT; int query_success = 0, hook_version = -1; size_t bol = 0; /* beginning of line */ @@ -247,6 +249,46 @@ void refresh_fsmonitor(struct index_state *istate) istate->fsmonitor_has_run_once = 1; trace_printf_key(&trace_fsmonitor, "refresh fsmonitor"); + + if (r->settings.use_builtin_fsmonitor > 0) { + query_success = !fsmonitor_ipc__send_query( + istate->fsmonitor_last_update, &query_result); + if (query_success) { + /* + * The response contains a series of nul terminated + * strings. The first is the new token. + * + * Use `char *buf` as an interlude to trick the CI + * static analysis to let us use `strbuf_addstr()` + * here (and only copy the token) rather than + * `strbuf_addbuf()`. + */ + buf = query_result.buf; + strbuf_addstr(&last_update_token, buf); + bol = last_update_token.len + 1; + } else { + /* + * The builtin daemon is not available on this + * platform -OR- we failed to get a response. + * + * Generate a fake token (rather than a V1 + * timestamp) for the index extension. (If + * they switch back to the hook API, we don't + * want ambiguous state.) + */ + strbuf_addstr(&last_update_token, "builtin:fake"); + } + + /* + * Regardless of whether we successfully talked to a + * fsmonitor daemon or not, we skip over and do not + * try to use the hook. The "core.useBuiltinFSMonitor" + * config setting ALWAYS overrides the "core.fsmonitor" + * hook setting. + */ + goto apply_results; + } + /* * This could be racy so save the date/time now and query_fsmonitor * should be inclusive to ensure we don't miss potential changes. @@ -301,6 +343,7 @@ void refresh_fsmonitor(struct index_state *istate) core_fsmonitor, query_success ? "success" : "failure"); } +apply_results: /* a fsmonitor process can return '/' to indicate all entries are invalid */ if (query_success && query_result.buf[bol] != '/') { /* Mark all entries returned by the monitor as dirty */ diff --git a/repo-settings.c b/repo-settings.c index f7fff0f5ab83..93aab92ff164 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r) r->settings.core_multi_pack_index = value; UPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1); + if (!repo_config_get_bool(r, "core.usebuiltinfsmonitor", &value) && value) + r->settings.use_builtin_fsmonitor = 1; + if (!repo_config_get_bool(r, "feature.manyfiles", &value) && value) { UPDATE_DEFAULT_BOOL(r->settings.index_version, 4); UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE); diff --git a/repository.h b/repository.h index b385ca3c94b6..d6e7f61f9cf7 100644 --- a/repository.h +++ b/repository.h @@ -29,6 +29,8 @@ enum fetch_negotiation_setting { struct repo_settings { int initialized; + int use_builtin_fsmonitor; + int core_commit_graph; int commit_graph_read_changed_paths; int gc_write_commit_graph; From patchwork Sat May 22 13:56:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C021FC47081 for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A55946115C for ; Sat, 22 May 2021 13:57:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231255AbhEVN6r (ORCPT ); Sat, 22 May 2021 09:58:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231191AbhEVN6n (ORCPT ); Sat, 22 May 2021 09:58:43 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AA4BC06138E for ; Sat, 22 May 2021 06:57:17 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id o127so12549111wmo.4 for ; Sat, 22 May 2021 06:57:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3VEHOZIwLhszB9Oy3JOIWLbqD59nCCtj2Y7uYExjmVA=; b=mAQrcwtml7PhIBEc/bCPLmrhrSBQY9utRJU5KYvrcMdn6JMJTvh+VylWZ+hUiqUy1u JDc0l0+En7L05QgMiVvzhthnmIEUGDxNxw5zXIkKy4gn/R3noD5+aXnZr8Tz2VKMDdzk DJbWt9rNqlwsZouQVnJpzavYvRalGeU9mZWWIMZND4+79UkFroSE3xjBTFc3koF+Zfjh b8dC0Th9NMoESOdnpNNfOiG74RJ/pHip8RiH37S6ar58n+sH5lWC3yxU4SJoAbSRgv6I AVfRMwe3fo2C4jdDlPYfdhNt94vRdaush4snluOZm6NCjTGVUOBIDwJqjmXPWeBnv6Cv RAXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3VEHOZIwLhszB9Oy3JOIWLbqD59nCCtj2Y7uYExjmVA=; b=g5gHHlgI9R2G+dhY0fYNJAbq4zsJ2DPR86gRig+PDerN6nCpaA6Ou9P2xNZaP/Hw57 AgjUFEdXe6rmE90S9xAoFUepdF6bLSCIsxcysPsBL296KkV4VAN7/toUApfwE+TPT8SO pOhsfl0Z6xhB5iZA5DHIbQ2GSFYauNJgyhp1+JVPHNnP7FCl3Xl+cV1kkcER0isXxf6L 3T95QQxMM/QO89jp3SdfQNIGNrvsNMJ4aNt7ucwrCVsVocDtp3XKwmZoZhQyUlsh17z2 HXDZIFILgnCy40Dtotok6kFwRiCF9t5zBuFDxcTtp/PvYxKIBUQU1qZMwMhhjp/436Zl PqsQ== X-Gm-Message-State: AOAM532ykeq16LpGV6k7AtqUJRUXuZMBnzhXywUQEvdJ5d/8X7SLPnPH l164yHBapa1MwWmc1E8Ooxa7rvwZGtk= X-Google-Smtp-Source: ABdhPJwae0HugdH2+YpeaXvJdo9mZYkj3tnDIThBS3GxN3xMVpuKLGXG2huUWdZ0CgGjriPbFiuUbQ== X-Received: by 2002:a05:600c:1551:: with SMTP id f17mr10750872wmg.17.1621691835691; Sat, 22 May 2021 06:57:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v11sm6677184wrs.9.2021.05.22.06.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:47 +0000 Subject: [PATCH v2 08/28] fsmonitor--daemon: add a built-in fsmonitor daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a built-in file system monitoring daemon that can be used by the existing `fsmonitor` feature (protocol API and index extension) to improve the performance of various Git commands, such as `status`. The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and provides an alternative to hook access to existing fsmonitors such as `watchman`. This commit merely adds the new command without any functionality. Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- .gitignore | 1 + Makefile | 1 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 53 +++++++++++++++++++++++++++++++++++++ git.c | 1 + 5 files changed, 57 insertions(+) create mode 100644 builtin/fsmonitor--daemon.c diff --git a/.gitignore b/.gitignore index 3dcdb6bb5ab8..beccf34abe9e 100644 --- a/.gitignore +++ b/.gitignore @@ -71,6 +71,7 @@ /git-format-patch /git-fsck /git-fsck-objects +/git-fsmonitor--daemon /git-gc /git-get-tar-commit-id /git-grep diff --git a/Makefile b/Makefile index 23f3b9890acd..74673acc9833 100644 --- a/Makefile +++ b/Makefile @@ -1092,6 +1092,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o +BUILTIN_OBJS += builtin/fsmonitor--daemon.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o BUILTIN_OBJS += builtin/grep.o diff --git a/builtin.h b/builtin.h index b6ce981b7377..7554476f90a4 100644 --- a/builtin.h +++ b/builtin.h @@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix); int cmd_for_each_repo(int argc, const char **argv, const char *prefix); int cmd_format_patch(int argc, const char **argv, const char *prefix); int cmd_fsck(int argc, const char **argv, const char *prefix); +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix); int cmd_gc(int argc, const char **argv, const char *prefix); int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix); int cmd_grep(int argc, const char **argv, const char *prefix); diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c new file mode 100644 index 000000000000..df2bad531118 --- /dev/null +++ b/builtin/fsmonitor--daemon.c @@ -0,0 +1,53 @@ +#include "builtin.h" +#include "config.h" +#include "parse-options.h" +#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +#include "simple-ipc.h" +#include "khash.h" + +static const char * const builtin_fsmonitor__daemon_usage[] = { + NULL +}; + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + const char *subcmd; + + struct option options[] = { + OPT_END() + }; + + if (argc < 2) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + git_config(git_default_config, NULL); + + subcmd = argv[1]; + argv--; + argc++; + + argc = parse_options(argc, argv, prefix, options, + builtin_fsmonitor__daemon_usage, 0); + + die(_("Unhandled subcommand '%s'"), subcmd); +} + +#else +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + struct option options[] = { + OPT_END() + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + die(_("fsmonitor--daemon not supported on this platform")); +} +#endif diff --git a/git.c b/git.c index b53e66567138..41980c897964 100644 --- a/git.c +++ b/git.c @@ -523,6 +523,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY }, From patchwork Sat May 22 13:56:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01E76C47080 for ; Sat, 22 May 2021 13:57:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB82E610A6 for ; Sat, 22 May 2021 13:57:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231284AbhEVN6u (ORCPT ); Sat, 22 May 2021 09:58:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231193AbhEVN6n (ORCPT ); Sat, 22 May 2021 09:58:43 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A244CC06138F for ; Sat, 22 May 2021 06:57:17 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id t206so12574044wmf.0 for ; Sat, 22 May 2021 06:57:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=PHaGE7WFcR5yICY5jz4l4JOBWPxGhBX+zjT5nluTxW0=; b=bBBsofVRyyOjg2zhM/LF+YViskoPm0BpXQ+j+IdxaNY1sIskYxGZPa49GfnobEbDnp 60p5FngL7U24TN4JC202RTzEcqMLl47RICM/USWHllydKDo2SzkBBh+CF5Thzd8LwuAJ TERkPCFnQfnMPYnO4l9qXZYeXIU+sWSfWwPOyt8tJeOi+K15dFC6t7jVQHz6zQy3cck4 8i/wy0SpEY+KEyJMkapPkiIwhY701A3Y/yQSvhgstjLWPjS3Jb3wmyjkGSLGiHamBqvI R6lbX7B8dZG5GmFtg/au2cYDvvT8B8NVmIrbDiuHn/GyXlAlJAkdfgbBZcUUCzx3wofj IKXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=PHaGE7WFcR5yICY5jz4l4JOBWPxGhBX+zjT5nluTxW0=; b=EJ3l1PRlPs4TKjQNXeJ+7wd+c6AaCtsHnEmFq7R9FjIYc3hnnHosIk4k/0h7YQSejM eIs2DzaOEmitwTIgybt6nhJ3RKUQQj1HtNPiXe8bMWHM10vcO5c2x1YT5wmYW1ChvXex gBFf2CiQt0iNziBX9pVEloKXZA6w07RmaaZGtP6UkmcKZRAkcEj6pMyFcL7pkfINoc5X J25u0Y4kjqhMtB+LlJ1f3Y6PgfEb+0lPJJ4NwA4ZvwoK6QNT2hMjp9zARtFj5QmlHaE5 uWjsa0Xac7jDoS37ir621/OwZMQ95s9HQBcckZfsmN2p9dzOTkdyfyeSKz35nh0wnSAR K9dQ== X-Gm-Message-State: AOAM530awsHV+aeLmDGwakkQ+vwpR120rKsf3mfcSofSe/bheizGVp42 hRDDbH4QaE+dHRce7r6Y8atI5jRyLSE= X-Google-Smtp-Source: ABdhPJxBwA4rSV/hvmCZoMTWcFh+uBLDNo10ktUIk/df+A4lS3JC8UFftNvga/JSebaBgEM82Vms9w== X-Received: by 2002:a1c:2048:: with SMTP id g69mr13042943wmg.6.1621691836244; Sat, 22 May 2021 06:57:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j10sm5483172wrt.32.2021.05.22.06.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:15 -0700 (PDT) Message-Id: <4f401310539ea4bebf662c7ddf82bb74e9f1d8c8.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:48 +0000 Subject: [PATCH v2 09/28] fsmonitor--daemon: implement client command options Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement `stop` and `status` client commands to control and query the status of a `fsmonitor--daemon` server process (and implicitly start a server process if necessary). Later commits will implement the actual server and monitor the file system. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 49 +++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index df2bad531118..16ff68b65407 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,10 +7,53 @@ #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon stop"), + N_("git fsmonitor--daemon status"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Acting as a CLIENT. + * + * Send a "quit" command to the `git-fsmonitor--daemon` (if running) + * and wait for it to shutdown. + */ +static int do_as_client__send_stop(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("quit", &answer); + + /* The quit command does not return any response data. */ + strbuf_release(&answer); + + if (ret) + return ret; + + trace2_region_enter("fsm_client", "polling-for-daemon-exit", NULL); + while (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + sleep_millisec(50); + trace2_region_leave("fsm_client", "polling-for-daemon-exit", NULL); + + return 0; +} + +static int do_as_client__status(void) +{ + enum ipc_active_state state = fsmonitor_ipc__get_state(); + + switch (state) { + case IPC_STATE__LISTENING: + printf(_("The built-in file system monitor is active\n")); + return 0; + + default: + printf(_("The built-in file system monitor is not active\n")); + return 1; + } +} int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { @@ -35,6 +78,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc = parse_options(argc, argv, prefix, options, builtin_fsmonitor__daemon_usage, 0); + if (!strcmp(subcmd, "stop")) + return !!do_as_client__send_stop(); + + if (!strcmp(subcmd, "status")) + return !!do_as_client__status(); + die(_("Unhandled subcommand '%s'"), subcmd); } From patchwork Sat May 22 13:56:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA53DC2B9F2 for ; Sat, 22 May 2021 13:57:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9D42661244 for ; Sat, 22 May 2021 13:57:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231293AbhEVN6v (ORCPT ); Sat, 22 May 2021 09:58:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231215AbhEVN6n (ORCPT ); Sat, 22 May 2021 09:58:43 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45C1BC061574 for ; Sat, 22 May 2021 06:57:18 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id d11so23701572wrw.8 for ; Sat, 22 May 2021 06:57:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=BA7FTa5LhN0caORx5MCdbHxg65jsbpWHZ0p9QRRXVV8=; b=ey0XBfEZJi0vjPH5c2/9sE6Q5zpoK6plbyjea3qrqaN/NWLErzzrB9Q+1KmkgDd3z8 Dwd3a3Sb5+/VQYdWkIj67UUMP/mFlsuFhEU3eYPCiGisHSI7YYfb4IFJ2vDvoYKbnY9l L7mm98sMoxejOr03gum3PeA3yKNjpBcHYKVPzJlAifIE1h1DTHN1Irfp28bxovMZ+TFQ gj0uJ/llAVj1PZQWnFzqWxf7tNFrhLvrzAQhN2v0i1CYUG2zC9B32TLzRuPTRxYPOXEl ojHChEnj/6rNWZORhpSbTrJE150vBb6+Vyc9O6jB4fjtGFMzXuqUO8vqL1glWf5lJnCo /ehA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=BA7FTa5LhN0caORx5MCdbHxg65jsbpWHZ0p9QRRXVV8=; b=pg+DNKR06IoRktj7CygLyox7j6j8QAj3qGOkWQ/eZubIZ/dIMEsVzKs/yCuJJ9mAuy OSuNIxXiexNp3Y6Q7iSnHEBDa1LEb/qvfbUP7ArOMtT1vAh7WB1Yik9nc4Vz9mMzELWL RPJxPRzURysPGzKgGgevRRQEm2u9M8YSbOcY/pRVjACOLdDJccm+ZRRUIic6SDlEzQLP pQTL05d0uoYXVH/csXtSyighXenz28uzaFfSxkVtZydgbREGr3bnlUDv2A6WwKLHwrf4 zWLpmSoJJ0yoCKFEuGwvG1N92elSEzdBW5QY3GwijzKaDmyq7dfBwdgbgSqfNZZRfO9W hjnQ== X-Gm-Message-State: AOAM53162P2225n6GLoCeN//74QFkQznRaJ+y8rEfDwj7LjtOsTT/clm 4l4dD60OWp+ENGHacU7OuuEPinnoH4s= X-Google-Smtp-Source: ABdhPJyPOvT9G4IWrRz5IjwTAwlvyIlGbY87UdaXqK6cHKf6khPEByuIA+0K0aooVJWpw2xTA4yvRQ== X-Received: by 2002:a05:6000:244:: with SMTP id m4mr14318746wrz.225.1621691836867; Sat, 22 May 2021 06:57:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x2sm2538508wmc.21.2021.05.22.06.57.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:16 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:49 +0000 Subject: [PATCH v2 10/28] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create an IPC client to send query and flush commands to the daemon. Signed-off-by: Jeff Hostetler --- Makefile | 1 + t/helper/test-fsmonitor-client.c | 125 +++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 128 insertions(+) create mode 100644 t/helper/test-fsmonitor-client.c diff --git a/Makefile b/Makefile index 74673acc9833..80059032c4e3 100644 --- a/Makefile +++ b/Makefile @@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o TEST_BUILTINS_OBJS += test-hash-speed.o diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c new file mode 100644 index 000000000000..4961f28e3e02 --- /dev/null +++ b/t/helper/test-fsmonitor-client.c @@ -0,0 +1,125 @@ +/* + * test-fsmonitor-client.c: client code to send commands/requests to + * a `git fsmonitor--daemon` daemon. + */ + +#include "test-tool.h" +#include "cache.h" +#include "parse-options.h" +//#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +//#include "compat/fsmonitor/fsmonitor-fs-listen.h" +//#include "fsmonitor--daemon.h" +//#include "simple-ipc.h" + +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND +int cmd__fsmonitor_client(int argc, const char **argv) +{ + die("fsmonitor--daemon not available on this platform"); +} +#else + +/* + * Read the `.git/index` to get the last token written to the + * FSMonitor Index Extension. + */ +static const char *get_token_from_index(void) +{ + struct index_state *istate = the_repository->index; + + if (do_read_index(istate, the_repository->index_file, 0) < 0) + die("unable to read index file"); + if (!istate->fsmonitor_last_update) + die("index file does not have fsmonitor extension"); + + return istate->fsmonitor_last_update; +} + +/* + * Send an IPC query to a `git-fsmonitor--daemon` daemon and + * ask for the changes since the given token or from the last + * token in the index extension. + * + * This will implicitly start a daemon process if necessary. The + * daemon process will persist after we exit. + */ +static int do_send_query(const char *token) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + if (!token || !*token) + token = get_token_from_index(); + + ret = fsmonitor_ipc__send_query(token, &answer); + if (ret < 0) + die(_("could not query fsmonitor--daemon")); + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +/* + * Send a "flush" command to the `git-fsmonitor--daemon` (if running) + * and tell it to flush its cache. + * + * This feature is primarily used by the test suite to simulate a loss of + * sync with the filesystem where we miss kernel events. + */ +static int do_send_flush(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("flush", &answer); + if (ret) + return ret; + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +int cmd__fsmonitor_client(int argc, const char **argv) +{ + const char *subcmd; + const char *token = NULL; + + const char * const fsmonitor_client_usage[] = { + N_("test-helper fsmonitor-client query []"), + N_("test-helper fsmonitor-client flush"), + NULL, + }; + + struct option options[] = { + OPT_STRING(0, "token", &token, N_("token"), + N_("command token to send to the server")), + OPT_END() + }; + + if (argc < 2) + usage_with_options(fsmonitor_client_usage, options); + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(fsmonitor_client_usage, options); + + subcmd = argv[1]; + argv--; + argc++; + + argc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0); + + setup_git_directory(); + + if (!strcmp(subcmd, "query")) + return !!do_send_query(token); + + if (!strcmp(subcmd, "flush")) + return !!do_send_flush(); + + die("Unhandled subcommand: '%s'", subcmd); +} +#endif diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 25c6a37e93e5..b15d328f9a41 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -31,6 +31,7 @@ static struct test_cmd cmds[] = { { "dump-untracked-cache", cmd__dump_untracked_cache }, { "example-decorate", cmd__example_decorate }, { "fast-rebase", cmd__fast_rebase }, + { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, { "hashmap", cmd__hashmap }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index f03c5988b20c..a8e96b97c419 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv); int cmd__dump_untracked_cache(int argc, const char **argv); int cmd__example_decorate(int argc, const char **argv); int cmd__fast_rebase(int argc, const char **argv); +int cmd__fsmonitor_client(int argc, const char **argv); int cmd__genrandom(int argc, const char **argv); int cmd__genzeros(int argc, const char **argv); int cmd__hashmap(int argc, const char **argv); From patchwork Sat May 22 13:56:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E285C2B9F2 for ; Sat, 22 May 2021 13:57:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1FE076128A for ; Sat, 22 May 2021 13:57:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231346AbhEVN64 (ORCPT ); Sat, 22 May 2021 09:58:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231220AbhEVN6o (ORCPT ); Sat, 22 May 2021 09:58:44 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 065C9C06174A for ; Sat, 22 May 2021 06:57:19 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id z17so23709582wrq.7 for ; Sat, 22 May 2021 06:57:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2WmMYUW0ToM7xwCC94wDRArWmCNFqsAQ9TSII3wBlas=; b=FdTcn8szWGrMapSP0h/YcmUssgMNstpUA2Z0gJpgYs3LTvU4zfxGyt0n1ug2OaOrE5 G+Y49CmmnMDHgrUoOJgUxVn9i39s5i3GrzBBqsQLPbiyrhEhRdrI/qQQ2wN+h0otrp34 y6iWnw95b82q6TRkC34YH7yL2vKghGz0VWbC9w2yam69lpU9bNPYAuAKdJyGtDz27Fvh cWwc3+WsCs+2YdasUQnde0V8j3+tmnGL6IwHIRTGBCwSuXCFGEx+NiJA20ZqiHWHvA8s Fyl08r8unGoTFkHc7CO5drSK4xxGx0PJ0/jDqq0VYzSv2XhnJBN35pN4iN+UIIilAFXI kn6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2WmMYUW0ToM7xwCC94wDRArWmCNFqsAQ9TSII3wBlas=; b=rA166/ALLklVUP5lQfP+B5LsU0HTzY4XRa3RxCNNiLGVR4kHwpPdgwvZlIUALKJRZW UBUNmArgp0uXnTt76MB9iETgW/vPEsrMtWebzboBAcp4QiNYTccB8e4hqbLkMemr+Um2 9ZVC9p8GusNB6gnfU7OkU5NM0ZdvWhSOy9QW7eImiQQguUBo53HRMDv2csylkojIdmMo yi5MqTQEnED2eLfG3Q3dyrLpI8nloUa2dHeBqF8/dllQXr7NJcGZUV0KxAHJ42KGvidE 7ofnLmI4T/YQZm8wcSXFJTm86aDt0ofHV13+LLm4QTM1qiFWKhWdQOR4DsgXQoh/ibfH foCg== X-Gm-Message-State: AOAM532GC5cpomglDTX17SzNAx/lp/t6VAzAVX1fSpGJA3HQXfnNYwbm qXZpgsFQcWWtwVKiFfpKbjmDhx1DXWw= X-Google-Smtp-Source: ABdhPJzSZMBY/zfmN4e7f7iX/BvrLUm4BG35Yb1mcCU0c6IIKr9fMBUkpvtJ5s+IJdf6zs7p2TjWrw== X-Received: by 2002:a5d:4744:: with SMTP id o4mr13677238wrs.86.1621691837586; Sat, 22 May 2021 06:57:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 198sm2148278wme.15.2021.05.22.06.57.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:17 -0700 (PDT) Message-Id: <49f9e2e3d49ce6e7b56839bf44535f271216abeb.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:50 +0000 Subject: [PATCH v2 11/28] fsmonitor-fs-listen-win32: stub in backend for Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty backend for fsmonitor--daemon on Windows. Signed-off-by: Jeff Hostetler --- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h diff --git a/Makefile b/Makefile index 80059032c4e3..3f31adfd135c 100644 --- a/Makefile +++ b/Makefile @@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1906,6 +1911,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2765,6 +2775,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c new file mode 100644 index 000000000000..880446b49e35 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -0,0 +1,21 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) +{ +} + +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsmonitor-fs-listen.h b/compat/fsmonitor/fsmonitor-fs-listen.h new file mode 100644 index 000000000000..c7b5776b3b60 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen.h @@ -0,0 +1,49 @@ +#ifndef FSMONITOR_FS_LISTEN_H +#define FSMONITOR_FS_LISTEN_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread PRIOR to staring the + * fsmonitor_fs_listener thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread AFTER joining the listener. + */ +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to watch for + * filesystem events. This will run in the fsmonitor_fs_listen thread. + * + * It should call `ipc_server_stop_async()` if the listener thread + * prematurely terminates (because of a filesystem error or if it + * detects that the .git directory has been deleted). (It should NOT + * do so if the listener thread receives a normal shutdown signal from + * the IPC layer.) + * + * It should set `state->error_code` to -1 if the daemon should exit + * with an error. + */ +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the fsmonitor listener thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_FS_LISTEN_H */ diff --git a/config.mak.uname b/config.mak.uname index cb443b4e023a..fcd88b60b14a 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows) # so we don't need this: # # SNPRINTF_RETURNS_BOGUS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) NO_STRTOUMAX = YesPlease NO_MKDTEMP = YesPlease NO_SVN_TESTS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 75ed198a6a36..4e812462d955 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -256,6 +256,11 @@ else() list(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c) endif() +if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +endif() + set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) #header checks From patchwork Sat May 22 13:56:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C2BFC2B9F2 for ; Sat, 22 May 2021 13:57:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3E4FE6115C for ; Sat, 22 May 2021 13:57:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231317AbhEVN6w (ORCPT ); Sat, 22 May 2021 09:58:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231216AbhEVN6o (ORCPT ); Sat, 22 May 2021 09:58:44 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BF6CC0613ED for ; Sat, 22 May 2021 06:57:19 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id r12so23725670wrp.1 for ; Sat, 22 May 2021 06:57:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MDWyQXyPSbGQR7L5DYZ6zArCYtfAnkfQdw8AWvttV1o=; b=NF7q99ILxmgYdpvhRaMJAOG5mD6IWkeSFrvsu4t7tQKqpeDmczdHRuf0OOk32tBGGF NMYGu/qWJfNJAp+p2tqzk4KYcniX1WeA2Gy9LYbN7TVC0BZCdGMSgN0k11wOQtAcMmaF L8Arts8LR41lIk6SpfVmsytfRAG4knZZM7N+diPcT6+eyutKJ4a+lgDJ7LEfEyLDHZke yy5kOuKc3qKld1MKN0NpRuihAnYzTT45l8RINOOEmNLiOP5PasLJhibkQfRflRboFdxM CwGCLbdt3e+8w+87WSlyGGHNjJXjfVFFiUBM+7kmDaJUJfIW+vyDmNLbER3DNB9BIKeo 2QIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MDWyQXyPSbGQR7L5DYZ6zArCYtfAnkfQdw8AWvttV1o=; b=IZWrof77LYImudq4j2M9mEE0o1O17bIID+IwNoo/VYG5iQpZUBGcFQFo542PiSWlZS e+o7cdESr8IibcwmZBFO6cppUTUSLcdBp5gy9/ZiPKPd8HY+wZYMIp2CJEoGbZYOklrV Wu468vcweaw9KETMm7omDMMpJG061ERVRwPh9eh3zSkTK2LlcQKsoBt9IX9uKE5i0K52 2GuvR9qWwn3charGra9LM922ljSGhxhOsjVTu+S5lql5FV71N1++qu+jXT6jMC2MQISv iG5k87CteZawI7IJ8tfk83Ecloj7UvQ5ImS/mPgMlwz6bXnk2/63xeS+q00b91fnRnj5 2G4A== X-Gm-Message-State: AOAM530g5uhVG8l80qXY9WA7kswPjcPigjzQtSbyfKafirYckB3HD0+q 6PIX7+xrKVDDD9aHjBgvP+LbO8iLsiw= X-Google-Smtp-Source: ABdhPJw6w5KGWfs8lRlVfARZtT3hlTrWESu30gCqS9dbcuctHNoe7Oe31GKLbRixefI8xeERvydQ6g== X-Received: by 2002:adf:d20a:: with SMTP id j10mr13842963wrh.188.1621691838204; Sat, 22 May 2021 06:57:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v10sm6670768wrq.0.2021.05.22.06.57.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:17 -0700 (PDT) Message-Id: <2aa85151f03c31989e3b11979711220d05f42b1a.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:51 +0000 Subject: [PATCH v2 12/28] fsmonitor-fs-listen-macos: stub in backend for MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty implementation of fsmonitor--daemon backend for MacOS. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c new file mode 100644 index 000000000000..b91058d1c4f8 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) +{ +} diff --git a/config.mak.uname b/config.mak.uname index fcd88b60b14a..394355463e1e 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macos + BASIC_LDFLAGS += -framework CoreServices endif ifeq ($(uname_S),SunOS) NEEDS_SOCKET = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 4e812462d955..22dec4600431 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -259,6 +259,9 @@ endif() if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c) endif() set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) From patchwork Sat May 22 13:56:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 071EEC2B9FB for ; Sat, 22 May 2021 13:57:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DDF3E61164 for ; Sat, 22 May 2021 13:57:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231364AbhEVN7A (ORCPT ); Sat, 22 May 2021 09:59:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231234AbhEVN6p (ORCPT ); Sat, 22 May 2021 09:58:45 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FA67C061574 for ; Sat, 22 May 2021 06:57:20 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id x7so4115463wrt.12 for ; Sat, 22 May 2021 06:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=nmvFboLsGN1ep7Em+HstKTb5udC48mccFW+YtRtT3ho=; b=s3guk7AuHypUKy1hKeY9yEqhD/Ug4WYGFprPyBLKQdQWZK961EvNdCx5cdXoYTzRaa ZzBkDCFNhqZXh1fWqtEOpwYBGeQ7/y0fpzPgtKTCYrcOVEjkBuZnbMU3qBVTvJ85UJOw qISxiywcTIva5BALVtMwNkWJSGnjy9VPcFp2Vj/BYAKPIa1qPE56csoJz92xW5j4XOnS KHyRvx1VKd0UME0gH7IvqQFv12ISeZnzqfbEOx4J9ncTTUIS3MLYJEUW9kasicoN5xEF EGsbBI6VIc0HT3pyuwGzlmERIqs0b2k9JeNdNUeHueWcWSGLgDBotzg2PSvyQaUlnkz0 7Hog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=nmvFboLsGN1ep7Em+HstKTb5udC48mccFW+YtRtT3ho=; b=kp6W0JxoK5WH1OX11gOxBJAdAi0eFx1S7rUdixeC6bwiKCzKhlRzc/UHUuc2Q39kav kfQ+EzNm4Stl1r0hxCDeOqu2ikkKaBy8ykMX8qi6NUJyP36DL3o92tjux466qPHY/wyp 4Y3zCkMvaAInXz4rZHLUS2yDTXZfDCIwpNMoZpFpouO6PZKURS3l9vUTJ94YdLwf/CKz dU0intGhsYDxhP5ZOD30J8U6Y+JMmWB7kaImjerjHsVzgWzig+5OStdj1fVXKa2T0gnD QNsRqC8/6h5sHaJmURr63KoHJ2B2rtcu06jK3Iw9VVYBCXlHCF+4OFpnTi1dL9t0VeWf Ti8Q== X-Gm-Message-State: AOAM533ZvWBTs0ab1pBQm5HHt4uZglYgW85SAuwyENqvznr2QwIIYDXt Ghc0ijLSF1HixLLMSuicemPvpJLEW94= X-Google-Smtp-Source: ABdhPJwAXNG3R7SJQqGezvr83MDArGGM/+JQAQPGf0Rot/8+22fbb7TfSiJCzXj2LdvTQuP9BJinSQ== X-Received: by 2002:a05:6000:192:: with SMTP id p18mr14141389wrx.252.1621691838901; Sat, 22 May 2021 06:57:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m9sm6616660wrs.36.2021.05.22.06.57.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:18 -0700 (PDT) Message-Id: <2aa05ad5c67f2ce93b97c306a9723afeabd64c8c.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:52 +0000 Subject: [PATCH v2 13/28] fsmonitor--daemon: implement daemon command options Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement `run` and `start` commands to try to begin listening for file system events. This version defines the thread structure with a single fsmonitor_fs_listen thread to watch for file system events and a simple IPC thread pool to wait for connections from Git clients over a well-known named pipe or Unix domain socket. This version does not actually do anything yet because the backends are still just stubs. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 390 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 34 ++++ 2 files changed, 423 insertions(+), 1 deletion(-) create mode 100644 fsmonitor--daemon.h diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 16ff68b65407..85f99dba861f 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,16 +3,52 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon start []"), + N_("git fsmonitor--daemon run []"), N_("git fsmonitor--daemon stop"), N_("git fsmonitor--daemon status"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Global state loaded from config. + */ +#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" +static int fsmonitor__ipc_threads = 8; + +#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout" +static int fsmonitor__start_timeout_sec = 60; + +static int fsmonitor_config(const char *var, const char *value, void *cb) +{ + if (!strcmp(var, FSMONITOR__IPC_THREADS)) { + int i = git_config_int(var, value); + if (i < 1) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__IPC_THREADS, i); + fsmonitor__ipc_threads = i; + return 0; + } + + if (!strcmp(var, FSMONITOR__START_TIMEOUT)) { + int i = git_config_int(var, value); + if (i < 0) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__START_TIMEOUT, i); + fsmonitor__start_timeout_sec = i; + return 0; + } + + return git_default_config(var, value, cb); +} + /* * Acting as a CLIENT. * @@ -55,11 +91,354 @@ static int do_as_client__status(void) } } +static ipc_server_application_cb handle_client; + +static int handle_client(void *data, + const char *command, size_t command_len, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + /* struct fsmonitor_daemon_state *state = data; */ + int result; + + /* + * The Simple IPC API now supports {char*, len} arguments, but + * FSMonitor always uses proper null-terminated strings, so + * we can ignore the command_len argument. (Trust, but verify.) + */ + if (command_len != strlen(command)) + BUG("FSMonitor assumes text messages"); + + trace2_region_enter("fsmonitor", "handle_client", the_repository); + trace2_data_string("fsmonitor", the_repository, "request", command); + + result = 0; /* TODO Do something here. */ + + trace2_region_leave("fsmonitor", "handle_client", the_repository); + + return result; +} + +static void *fsmonitor_fs_listen__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-listen"); + + trace_printf_key(&trace_fsmonitor, "Watching: worktree '%s'", + state->path_worktree_watch.buf); + if (state->nr_paths_watching > 1) + trace_printf_key(&trace_fsmonitor, "Watching: gitdir '%s'", + state->path_gitdir_watch.buf); + + fsmonitor_fs_listen__loop(state); + + trace2_thread_exit(); + return NULL; +} + +static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) +{ + struct ipc_server_opts ipc_opts = { + .nr_threads = fsmonitor__ipc_threads, + + /* + * We know that there are no other active threads yet, + * so we can let the IPC layer temporarily chdir() if + * it needs to when creating the server side of the + * Unix domain socket. + */ + .uds_disallow_chdir = 0 + }; + + /* + * Start the IPC thread pool before the we've started the file + * system event listener thread so that we have the IPC handle + * before we need it. + */ + if (ipc_server_run_async(&state->ipc_server_data, + fsmonitor_ipc__get_path(), &ipc_opts, + handle_client, state)) + return error(_("could not start IPC thread pool")); + + /* + * Start the fsmonitor listener thread to collect filesystem + * events. + */ + if (pthread_create(&state->listener_thread, NULL, + fsmonitor_fs_listen__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + ipc_server_await(state->ipc_server_data); + + return error(_("could not start fsmonitor listener thread")); + } + + /* + * The daemon is now fully functional in background threads. + * Wait for the IPC thread pool to shutdown (whether by client + * request or from filesystem activity). + */ + ipc_server_await(state->ipc_server_data); + + /* + * The fsmonitor listener thread may have received a shutdown + * event from the IPC thread pool, but it doesn't hurt to tell + * it again. And wait for it to shutdown. + */ + fsmonitor_fs_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + + return state->error_code; +} + +static int fsmonitor_run_daemon(void) +{ + struct fsmonitor_daemon_state state; + int err; + + memset(&state, 0, sizeof(state)); + + pthread_mutex_init(&state.main_lock, NULL); + state.error_code = 0; + state.current_token_data = NULL; + + /* Prepare to (recursively) watch the directory. */ + strbuf_init(&state.path_worktree_watch, 0); + strbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree())); + state.nr_paths_watching = 1; + + /* + * We create/delete cookie files inside the .git directory to + * help us keep sync with the file system. If ".git" is not a + * directory, then is not inside the cone of + * , so set up a second watch for it. + */ + strbuf_init(&state.path_gitdir_watch, 0); + strbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch); + strbuf_addstr(&state.path_gitdir_watch, "/.git"); + if (!is_directory(state.path_gitdir_watch.buf)) { + strbuf_reset(&state.path_gitdir_watch); + strbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir())); + state.nr_paths_watching = 2; + } + + /* + * Confirm that we can create platform-specific resources for the + * filesystem listener before we bother starting all the threads. + */ + if (fsmonitor_fs_listen__ctor(&state)) { + err = error(_("could not initialize listener thread")); + goto done; + } + + err = fsmonitor_run_daemon_1(&state); + +done: + pthread_mutex_destroy(&state.main_lock); + fsmonitor_fs_listen__dtor(&state); + + ipc_server_free(state.ipc_server_data); + + strbuf_release(&state.path_worktree_watch); + strbuf_release(&state.path_gitdir_watch); + + return err; +} + +static int try_to_run_foreground_daemon(void) +{ + /* + * Technically, we don't need to probe for an existing daemon + * process, since we could just call `fsmonitor_run_daemon()` + * and let it fail if the pipe/socket is busy. + * + * However, this method gives us a nicer error message for a + * common error case. + */ + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + die("fsmonitor--daemon is already running."); + + return !!fsmonitor_run_daemon(); +} + +#ifndef GIT_WINDOWS_NATIVE +/* + * This is adapted from `daemonize()`. Use `fork()` to directly create + * and run the daemon in a child process. The fork-parent returns the + * child PID so that we can wait for the child to startup before exiting. + */ +static int spawn_background_fsmonitor_daemon(pid_t *pid) +{ + *pid = fork(); + + switch (*pid) { + case 0: + if (setsid() == -1) + error_errno(_("setsid failed")); + close(0); + close(1); + close(2); + sanitize_stdfds(); + + return !!fsmonitor_run_daemon(); + + case -1: + return error_errno(_("could not spawn fsmonitor--daemon in the background")); + + default: + return 0; + } +} +#else +/* + * Conceptually like `daemonize()` but different because Windows does not + * have `fork(2)`. Spawn a normal Windows child process but without the + * limitations of `start_command()` and `finish_command()`. + */ +static int spawn_background_fsmonitor_daemon(pid_t *pid) +{ + char git_exe[MAX_PATH]; + struct strvec args = STRVEC_INIT; + int in, out; + + GetModuleFileNameA(NULL, git_exe, MAX_PATH); + + in = open("/dev/null", O_RDONLY); + out = open("/dev/null", O_WRONLY); + + strvec_push(&args, git_exe); + strvec_push(&args, "fsmonitor--daemon"); + strvec_push(&args, "run"); + + *pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out); + close(in); + close(out); + + strvec_clear(&args); + + if (*pid < 0) + return error(_("could not spawn fsmonitor--daemon in the background")); + + return 0; +} +#endif + +/* + * This is adapted from `wait_or_whine()`. Watch the child process and + * let it get started and begin listening for requests on the socket + * before reporting our success. + */ +static int wait_for_background_startup(pid_t pid_child) +{ + int status; + pid_t pid_seen; + enum ipc_active_state s; + time_t time_limit, now; + + time(&time_limit); + time_limit += fsmonitor__start_timeout_sec; + + for (;;) { + pid_seen = waitpid(pid_child, &status, WNOHANG); + + if (pid_seen == -1) + return error_errno(_("waitpid failed")); + else if (pid_seen == 0) { + /* + * The child is still running (this should be + * the normal case). Try to connect to it on + * the socket and see if it is ready for + * business. + * + * If there is another daemon already running, + * our child will fail to start (possibly + * after a timeout on the lock), but we don't + * care (who responds) if the socket is live. + */ + s = fsmonitor_ipc__get_state(); + if (s == IPC_STATE__LISTENING) + return 0; + + time(&now); + if (now > time_limit) + return error(_("fsmonitor--daemon not online yet")); + } else if (pid_seen == pid_child) { + /* + * The new child daemon process shutdown while + * it was starting up, so it is not listening + * on the socket. + * + * Try to ping the socket in the odd chance + * that another daemon started (or was already + * running) while our child was starting. + * + * Again, we don't care who services the socket. + */ + s = fsmonitor_ipc__get_state(); + if (s == IPC_STATE__LISTENING) + return 0; + + /* + * We don't care about the WEXITSTATUS() nor + * any of the WIF*(status) values because + * `cmd_fsmonitor__daemon()` does the `!!result` + * trick on all function return values. + * + * So it is sufficient to just report the + * early shutdown as an error. + */ + return error(_("fsmonitor--daemon failed to start")); + } else + return error(_("waitpid is confused")); + } +} + +static int try_to_start_background_daemon(void) +{ + pid_t pid_child; + int ret; + + /* + * Before we try to create a background daemon process, see + * if a daemon process is already listening. This makes it + * easier for us to report an already-listening error to the + * console, since our spawn/daemon can only report the success + * of creating the background process (and not whether it + * immediately exited). + */ + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + die("fsmonitor--daemon is already running."); + + /* + * Run the actual daemon in a background process. + */ + ret = spawn_background_fsmonitor_daemon(&pid_child); + if (pid_child <= 0) + return ret; + + /* + * Wait (with timeout) for the background child process get + * started and begin listening on the socket/pipe. This makes + * the "start" command more synchronous and more reliable in + * tests. + */ + ret = wait_for_background_startup(pid_child); + + return ret; +} + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { const char *subcmd; struct option options[] = { + OPT_INTEGER(0, "ipc-threads", + &fsmonitor__ipc_threads, + N_("use ipc worker threads")), + OPT_INTEGER(0, "start-timeout", + &fsmonitor__start_timeout_sec, + N_("Max seconds to wait for background daemon startup")), + OPT_END() }; @@ -69,7 +448,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_fsmonitor__daemon_usage, options); - git_config(git_default_config, NULL); + git_config(fsmonitor_config, NULL); subcmd = argv[1]; argv--; @@ -77,6 +456,15 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc = parse_options(argc, argv, prefix, options, builtin_fsmonitor__daemon_usage, 0); + if (fsmonitor__ipc_threads < 1) + die(_("invalid 'ipc-threads' value (%d)"), + fsmonitor__ipc_threads); + + if (!strcmp(subcmd, "start")) + return !!try_to_start_background_daemon(); + + if (!strcmp(subcmd, "run")) + return !!try_to_run_foreground_daemon(); if (!strcmp(subcmd, "stop")) return !!do_as_client__send_stop(); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h new file mode 100644 index 000000000000..3009c1a83de7 --- /dev/null +++ b/fsmonitor--daemon.h @@ -0,0 +1,34 @@ +#ifndef FSMONITOR_DAEMON_H +#define FSMONITOR_DAEMON_H + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +#include "cache.h" +#include "dir.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "thread-utils.h" + +struct fsmonitor_batch; +struct fsmonitor_token_data; + +struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ + +struct fsmonitor_daemon_state { + pthread_t listener_thread; + pthread_mutex_t main_lock; + + struct strbuf path_worktree_watch; + struct strbuf path_gitdir_watch; + int nr_paths_watching; + + struct fsmonitor_token_data *current_token_data; + + int error_code; + struct fsmonitor_daemon_backend_data *backend_data; + + struct ipc_server_data *ipc_server_data; +}; + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_DAEMON_H */ From patchwork Sat May 22 13:56:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18C35C2B9F2 for ; Sat, 22 May 2021 13:57:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E99B761164 for ; Sat, 22 May 2021 13:57:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231300AbhEVN7B (ORCPT ); Sat, 22 May 2021 09:59:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231247AbhEVN6q (ORCPT ); Sat, 22 May 2021 09:58:46 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA1D2C06174A for ; Sat, 22 May 2021 06:57:20 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id y14so21647468wrm.13 for ; Sat, 22 May 2021 06:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sb0aZwOycW4htAtXTaVgP5R3g8GxGTSMY2FRbuZGUWc=; b=DpO94buNpNTYQq5l9yPHSSzCiYNFw9OfLBEHYP8VPry1EvGtBIVL7iz4FEybi4j6TP gcE4qbOSY+iU/A1rtw48tVHP+YU5Ei+UZzNVP8BkcCo32xu110d/yP4KYv3qR+Ll3R8V tmhyRZ4ojF8rO7NpydQjPS6n4KOlBq3HY+eg/w1ftDXq95B31lr8SNoQIhnUmzSIz4DG Z1iBwdONai2z3XKAd88wBQHDtvbf7KFnTyLjMumeegDEhJyBbxlnUGEqm3UWB0EV3cbE tLQ2eqh4yGsbUBuJ4v5osvVoz3oBGTsfHpNmWgcApAU4DRQxOpRh02RDnnTpcfVWu0ED uVmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sb0aZwOycW4htAtXTaVgP5R3g8GxGTSMY2FRbuZGUWc=; b=ucL8nmrVf0da+39yUZlCTctAjbAZVpOApzjcGQEqX5G8IOSQawBGOKr9SoMbCzbZhW Dq36XTqbqkCQGOXFysyS+yWqvemRhPiJrkt3k7te9om0iuGDlywggxga1oRpjmBJk6w5 DUjwPqGtEf2r7VTDhulJ9giI4+1TdVlRuAUn99PtoBfiNl2oxl7Del1PCX/zqwyGhiVu d6SNIuzshG8lS8gywhToY/V9AayuF97DtQ158FmbxYQZX+knx4VHT9+MGBc2zxxZ4MiA rkjovaKF80sge3B+ugMO/11VDqtj99EG/eNkQjZsGlkhEKobMk0nYhXWnbNZm5V6YjXF mzvw== X-Gm-Message-State: AOAM531iro0oO60JWGxSUWCTlmXT0AyO2/eDqrZos7Qr8PqW1jnHrUvb iDC1fPrpVjSG47GgeVgts2cKjHCX/y8= X-Google-Smtp-Source: ABdhPJyS+ojxyt6zMwD7RO0pYdU4bpc7eDrmStVBLzg0ub72sZdsBX32rvlmSggksOjBFsw0+7jY2g== X-Received: by 2002:a5d:470d:: with SMTP id y13mr14274671wrq.203.1621691839439; Sat, 22 May 2021 06:57:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q27sm5561348wrz.79.2021.05.22.06.57.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:19 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:53 +0000 Subject: [PATCH v2 14/28] fsmonitor--daemon: add pathname classification Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to classify relative and absolute pathnames and decide how they should be handled. This will be used by the platform-specific backend to respond to each filesystem event. When we register for filesystem notifications on a directory, we get events for everything (recursively) in the directory. We want to report to clients changes to tracked and untracked paths within the working directory. We do not want to report changes within the .git directory, for example. This classification will be used in a later commit by the different backends to classify paths as events are received. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 85f99dba861f..d6e35ded9f68 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -119,6 +119,87 @@ static int handle_client(void *data, return result; } +#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" + +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *rel) +{ + if (fspathncmp(rel, ".git", 4)) + return IS_WORKDIR_PATH; + rel += 4; + + if (!*rel) + return IS_DOT_GIT; + if (*rel != '/') + return IS_WORKDIR_PATH; /* e.g. .gitignore */ + rel++; + + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX; + + return IS_INSIDE_DOT_GIT; +} + +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *rel) +{ + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX; + + return IS_INSIDE_GITDIR; +} + +static enum fsmonitor_path_type try_classify_workdir_abs_path( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + + if (fspathncmp(path, state->path_worktree_watch.buf, + state->path_worktree_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_worktree_watch.len; + + if (!*rel) + return IS_WORKDIR_PATH; /* it is the root dir exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_workdir_relative(rel); +} + +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + enum fsmonitor_path_type t; + + t = try_classify_workdir_abs_path(state, path); + if (state->nr_paths_watching == 1) + return t; + if (t != IS_OUTSIDE_CONE) + return t; + + if (fspathncmp(path, state->path_gitdir_watch.buf, + state->path_gitdir_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_gitdir_watch.len; + + if (!*rel) + return IS_GITDIR; /* it is the exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_gitdir_relative(rel); +} + static void *fsmonitor_fs_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 3009c1a83de7..7bbb3a27a1ce 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -30,5 +30,66 @@ struct fsmonitor_daemon_state { struct ipc_server_data *ipc_server_data; }; +/* + * Pathname classifications. + * + * The daemon classifies the pathnames that it receives from file + * system notification events into the following categories and uses + * that to decide whether clients are told about them. (And to watch + * for file system synchronization events.) + * + * The client should only care about paths within the working + * directory proper (inside the working directory and not ".git" nor + * inside of ".git/"). That is, the client has read the index and is + * asking for a list of any paths in the working directory that have + * been modified since the last token. The client does not care about + * file system changes within the .git directory (such as new loose + * objects or packfiles). So the client will only receive paths that + * are classified as IS_WORKDIR_PATH. + * + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the + * exact ".git" directory or GITDIR. If the daemon receives a delete + * event for either of these directories, it will automatically + * shutdown, for example. + * + * Note that the daemon DOES NOT explicitly watch nor special case the + * ".git/index" file. The daemon does not read the index and does not + * have any internal index-relative state. The daemon only collects + * the set of modified paths within the working directory. + */ +enum fsmonitor_path_type { + IS_WORKDIR_PATH = 0, + + IS_DOT_GIT, + IS_INSIDE_DOT_GIT, + IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX, + + IS_GITDIR, + IS_INSIDE_GITDIR, + IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX, + + IS_OUTSIDE_CONE, +}; + +/* + * Classify a pathname relative to the root of the working directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *relative_path); + +/* + * Classify a pathname relative to a that is external to the + * worktree directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *relative_path); + +/* + * Classify an absolute pathname received from a filesystem event. + */ +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Sat May 22 13:56:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5847AC2B9F2 for ; Sat, 22 May 2021 13:57:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3186561164 for ; Sat, 22 May 2021 13:57:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231381AbhEVN7E (ORCPT ); Sat, 22 May 2021 09:59:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231252AbhEVN6r (ORCPT ); Sat, 22 May 2021 09:58:47 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F756C0613ED for ; Sat, 22 May 2021 06:57:21 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id q5so23718469wrs.4 for ; Sat, 22 May 2021 06:57:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=N5qY1y9CfNbyDb71UAUZBNInayWHSRyciKICzyYlGNc=; b=a7zIEy6p/mp+ncB6ECyifJ36s/YJC0+LOY7pwgFf3xm82nOUiFolImRqFMsV4bTDuJ lnzM2+lO0Y+W64nwajljg/+nbpz5vzZUqlQJ8quCnfCxMfhkLvLU2EDPuC7uucBOOr+P kdp6jq37scWOyuU3GjIXZXyUGKHtpABTaKAAaG8D1U43nlNAECXzL3CyXkFv4XpuZF6c Dqm1eZANuwyalDgwKwObRUXlWo/rEcQm1VUemOrVHl3msK29o22/TZ4eNK56+nD6/S+6 Tz0S7MIeZLGCmru2nNw/lDngix0o+sMOz6BOZ9z77dVc3NoXa+CUsTQ8OQY4Z2rO+wDy srhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=N5qY1y9CfNbyDb71UAUZBNInayWHSRyciKICzyYlGNc=; b=oKUUQnFho8UCeCHOfUW+n5Xkjyi4KRDnxP4xdsM6hMabyyGjrcfCNGcdqLzi+6RZS7 Jh4DdnCzHSMhkKyZTHn3FLAXUx/9e7K1l3ZV3iyU2h0RB4+NkkquV090a1ac25O0wcjr O0WyfPZanv86oBaCAdQnBYb89/2qatvGGErV22tx6pb0900OmMjaPLD9vooh3tpod82S PO8149saKpMeENTlwaqCi3pWVdBV0wXs0hRlGsMKWMP8FjgY9GFV8GONpC+NHUmbi1nc ZIofMZhBIU7KXlzT1Li29L/DrfuZK+1GLtgvCrN4uS8rnz1iRpq0i+MnztptRAjRP3wF Xd9Q== X-Gm-Message-State: AOAM531PW58YBzbqP/rG4VJyYNAqt0S3YyiTL8vhSJH52caCUcOmNVvL mBG18aQSCxGUiKjDjIqrKDC7zahhsX4= X-Google-Smtp-Source: ABdhPJys8ICuRvCxUDSGV0hmEv8wTEeLk0TBZWGD6gLiU2Yi+63WqTn+ro5e/dLhXKKLAe/XBSCd1Q== X-Received: by 2002:a5d:44cb:: with SMTP id z11mr14544923wrr.159.1621691840062; Sat, 22 May 2021 06:57:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r1sm5420587wrt.67.2021.05.22.06.57.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:19 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:54 +0000 Subject: [PATCH v2 15/28] fsmonitor--daemon: define token-ids Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to create token-ids and define the overall token naming scheme. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++- 1 file changed, 115 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index d6e35ded9f68..ecd456dc9284 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -91,6 +91,120 @@ static int do_as_client__status(void) } } +/* + * Requests to and from a FSMonitor Protocol V2 provider use an opaque + * "token" as a virtual timestamp. Clients can request a summary of all + * created/deleted/modified files relative to a token. In the response, + * clients receive a new token for the next (relative) request. + * + * + * Token Format + * ============ + * + * The contents of the token are private and provider-specific. + * + * For the built-in fsmonitor--daemon, we define a token as follows: + * + * "builtin" ":" ":" + * + * The "builtin" prefix is used as a namespace to avoid conflicts + * with other providers (such as Watchman). + * + * The is an arbitrary OPAQUE string, such as a GUID, + * UUID, or {timestamp,pid}. It is used to group all filesystem + * events that happened while the daemon was monitoring (and in-sync + * with the filesystem). + * + * Unlike FSMonitor Protocol V1, it is not defined as a timestamp + * and does not define less-than/greater-than relationships. + * (There are too many race conditions to rely on file system + * event timestamps.) + * + * The is a simple integer incremented whenever the + * daemon needs to make its state public. For example, if 1000 file + * system events come in, but no clients have requested the data, + * the daemon can continue to accumulate file changes in the same + * bin and does not need to advance the sequence number. However, + * as soon as a client does arrive, the daemon needs to start a new + * bin and increment the sequence number. + * + * The sequence number serves as the boundary between 2 sets + * of bins -- the older ones that the client has already seen + * and the newer ones that it hasn't. + * + * When a new is created, the is reset to + * zero. + * + * + * About Token Ids + * =============== + * + * A new token_id is created: + * + * [1] each time the daemon is started. + * + * [2] any time that the daemon must re-sync with the filesystem + * (such as when the kernel drops or we miss events on a very + * active volume). + * + * [3] in response to a client "flush" command (for dropped event + * testing). + * + * When a new token_id is created, the daemon is free to discard all + * cached filesystem events associated with any previous token_ids. + * Events associated with a non-current token_id will never be sent + * to a client. A token_id change implicitly means that the daemon + * has gap in its event history. + * + * Therefore, clients that present a token with a stale (non-current) + * token_id will always be given a trivial response. + */ +struct fsmonitor_token_data { + struct strbuf token_id; + struct fsmonitor_batch *batch_head; + struct fsmonitor_batch *batch_tail; + uint64_t client_ref_count; +}; + +static struct fsmonitor_token_data *fsmonitor_new_token_data(void) +{ + static int test_env_value = -1; + static uint64_t flush_count = 0; + struct fsmonitor_token_data *token; + + CALLOC_ARRAY(token, 1); + + strbuf_init(&token->token_id, 0); + token->batch_head = NULL; + token->batch_tail = NULL; + token->client_ref_count = 0; + + if (test_env_value < 0) + test_env_value = git_env_bool("GIT_TEST_FSMONITOR_TOKEN", 0); + + if (!test_env_value) { + struct timeval tv; + struct tm tm; + time_t secs; + + gettimeofday(&tv, NULL); + secs = tv.tv_sec; + gmtime_r(&secs, &tm); + + strbuf_addf(&token->token_id, + "%"PRIu64".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ", + flush_count++, + getpid(), + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (long)tv.tv_usec); + } else { + strbuf_addf(&token->token_id, "test_%08x", test_env_value++); + } + + return token; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -281,7 +395,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); state.error_code = 0; - state.current_token_data = NULL; + state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ strbuf_init(&state.path_worktree_watch, 0); From patchwork Sat May 22 13:56:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D11ECC2B9FB for ; Sat, 22 May 2021 13:57:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B58C66115C for ; Sat, 22 May 2021 13:57:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231334AbhEVN7D (ORCPT ); Sat, 22 May 2021 09:59:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231262AbhEVN6s (ORCPT ); Sat, 22 May 2021 09:58:48 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26FCFC06138A for ; Sat, 22 May 2021 06:57:22 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id a4so23772012wrr.2 for ; Sat, 22 May 2021 06:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=U6bHjkI6nlXZnaM0zvTqdYvuZwlfLsO4AySm1vblci8=; b=CSe5e8wj0zPDigQB32EkxpFXpvw2+jEy/SlKpP+h84vsLFtDdyRk03Fw/fVTSNK+Uu K0Q6h0MJQSkCKYm9Q1eUqukiFZw7GRiFZOmEf3b0vG7q7HojBbaexcDd5QcW6ZUS+s9R fWzLZ2bl0beA4uPofLJfkMI/Tn6U9Cn4QNyRIyznaVL/Ho79XPkEYsnQ99mC2dEHIDQS tIjwd2/KKM5vPYDtwi48i/DvdBGEBvJUbLWM9Tun88tk3KBCmorLaw6NjqONpLqKysyn us9gSCiP0MaY7LFLUeAfH7QuR7YvFT/2gCcKo/308tHLmgLrkL+wCAkSBMMXvkz7TneG FiKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=U6bHjkI6nlXZnaM0zvTqdYvuZwlfLsO4AySm1vblci8=; b=C1OG96eSCXbqglvorfVIAmBEuRIsSJpdnniIXFqkjSX06G/DwwMBxyQd5GH2OPIVfk 7wCnVdtmfeP+F+ZAj4QYpGPx5qBqklU3xS1qqNrVUQ4VzquTi3oqXa8vs62/hDwIdxaR r0hbef4tANbo+UIjoSEQSmSGsAwVwOXRNvxKAj26YBDqPlqrGMy56p/GK0HlxaoXqnmU 2i7DwXWu84PXYQMFIpH2lzpLnHKAdJb3a26yau24SyDnR95y2eH4ZlHuEW7wB8fJ55Cp lVBFIjJpGXqtBaM8YT+ikMP4oeHvuA++I/1FDOJ6uY8sPNaEUqPVCz0OrmLdcQgogN4M +c0A== X-Gm-Message-State: AOAM533Ij38W8gdkU2tCwp6JVUPa713oVYptxZzL0AkTGqbxT24a04Fi uyK5eas7s7sGHs48ejNy4KxwMNg+mO4= X-Google-Smtp-Source: ABdhPJy12Jw1EDExE9cilhqf9khLEmuvCUSNXks+luT8Q3arIC46Xbzau6804EQ9CpNnu3tiZpZ5OA== X-Received: by 2002:adf:f7c4:: with SMTP id a4mr14180286wrq.20.1621691840633; Sat, 22 May 2021 06:57:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f18sm1039863wmc.40.2021.05.22.06.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:20 -0700 (PDT) Message-Id: <2ed7bc3fae7aece122f1c55f59f4c3e8201de341.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:55 +0000 Subject: [PATCH v2 16/28] fsmonitor--daemon: create token-based changed path cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to build a list of changed paths and associate them with a token-id. This will be used by the platform-specific backends to accumulate changed paths in response to filesystem events. The platform-specific file system listener thread receives file system events containing one or more changed pathnames (with whatever bucketing or grouping that is convenient for the file system). These paths are accumulated (without locking) by the file system layer into a `fsmonitor_batch`. When the file system layer has drained the kernel event queue, it will "publish" them to our token queue and make them visible to concurrent client worker threads. The token layer is free to combine and/or de-dup paths within these batches for efficient presentation to clients. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 234 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 40 ++++++ 2 files changed, 272 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index ecd456dc9284..663fead0d66e 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -166,17 +166,27 @@ struct fsmonitor_token_data { uint64_t client_ref_count; }; +struct fsmonitor_batch { + struct fsmonitor_batch *next; + uint64_t batch_seq_nr; + const char **interned_paths; + size_t nr, alloc; + time_t pinned_time; +}; + static struct fsmonitor_token_data *fsmonitor_new_token_data(void) { static int test_env_value = -1; static uint64_t flush_count = 0; struct fsmonitor_token_data *token; + struct fsmonitor_batch *batch; CALLOC_ARRAY(token, 1); + batch = fsmonitor_batch__new(); strbuf_init(&token->token_id, 0); - token->batch_head = NULL; - token->batch_tail = NULL; + token->batch_head = batch; + token->batch_tail = batch; token->client_ref_count = 0; if (test_env_value < 0) @@ -202,9 +212,147 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void) strbuf_addf(&token->token_id, "test_%08x", test_env_value++); } + /* + * We created a new and are starting a new series + * of tokens with a zero . + * + * Since clients cannot guess our new (non test) + * they will always receive a trivial response (because of the + * mismatch on the ). The trivial response will + * tell them our new so that subsequent requests + * will be relative to our new series. (And when sending that + * response, we pin the current head of the batch list.) + * + * Even if the client correctly guesses the , their + * request of "builtin::0" asks for all changes MORE + * RECENT than batch/bin 0. + * + * This implies that it is a waste to accumulate paths in the + * initial batch/bin (because they will never be transmitted). + * + * So the daemon could be running for days and watching the + * file system, but doesn't need to actually accumulate any + * paths UNTIL we need to set a reference point for a later + * relative request. + * + * However, it is very useful for testing to always have a + * reference point set. Pin batch 0 to force early file system + * events to accumulate. + */ + if (test_env_value) + batch->pinned_time = time(NULL); + return token; } +struct fsmonitor_batch *fsmonitor_batch__new(void) +{ + struct fsmonitor_batch *batch; + + CALLOC_ARRAY(batch, 1); + + return batch; +} + +struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch) +{ + struct fsmonitor_batch *next; + + if (!batch) + return NULL; + + next = batch->next; + + /* + * The actual strings within the array are interned, so we don't + * own them. + */ + free(batch->interned_paths); + + return next; +} + +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, + const char *path) +{ + const char *interned_path = strintern(path); + + trace_printf_key(&trace_fsmonitor, "event: %s", interned_path); + + ALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc); + batch->interned_paths[batch->nr++] = interned_path; +} + +static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, + const struct fsmonitor_batch *batch_src) +{ + size_t k; + + ALLOC_GROW(batch_dest->interned_paths, + batch_dest->nr + batch_src->nr + 1, + batch_dest->alloc); + + for (k = 0; k < batch_src->nr; k++) + batch_dest->interned_paths[batch_dest->nr++] = + batch_src->interned_paths[k]; +} + +static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) +{ + struct fsmonitor_batch *p; + + if (!token) + return; + + assert(token->client_ref_count == 0); + + strbuf_release(&token->token_id); + + for (p = token->batch_head; p; p = fsmonitor_batch__pop(p)) + ; + + free(token); +} + +/* + * Flush all of our cached data about the filesystem. Call this if we + * lose sync with the filesystem and miss some notification events. + * + * [1] If we are missing events, then we no longer have a complete + * history of the directory (relative to our current start token). + * We should create a new token and start fresh (as if we just + * booted up). + * + * If there are no concurrent threads readering the current token data + * series, we can free it now. Otherwise, let the last reader free + * it. + * + * Either way, the old token data series is no longer associated with + * our state data. + */ +static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + struct fsmonitor_token_data *free_me = NULL; + struct fsmonitor_token_data *new_one = NULL; + + new_one = fsmonitor_new_token_data(); + + if (state->current_token_data->client_ref_count == 0) + free_me = state->current_token_data; + state->current_token_data = new_one; + + fsmonitor_free_token_data(free_me); +} + +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) +{ + pthread_mutex_lock(&state->main_lock); + with_lock__do_force_resync(state); + pthread_mutex_unlock(&state->main_lock); +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -314,6 +462,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( return fsmonitor_classify_path_gitdir_relative(rel); } +/* + * We try to combine small batches at the front of the batch-list to avoid + * having a long list. This hopefully makes it a little easier when we want + * to truncate and maintain the list. However, we don't want the paths array + * to just keep growing and growing with realloc, so we insert an arbitrary + * limit. + */ +#define MY_COMBINE_LIMIT (1024) + +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names) +{ + if (!batch && !cookie_names->nr) + return; + + pthread_mutex_lock(&state->main_lock); + + if (batch) { + struct fsmonitor_batch *head; + + head = state->current_token_data->batch_head; + if (!head) { + BUG("token does not have batch"); + } else if (head->pinned_time) { + /* + * We cannot alter the current batch list + * because: + * + * [a] it is being transmitted to at least one + * client and the handle_client() thread has a + * ref-count, but not a lock on the batch list + * starting with this item. + * + * [b] it has been transmitted in the past to + * at least one client such that future + * requests are relative to this head batch. + * + * So, we can only prepend a new batch onto + * the front of the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else if (!head->batch_seq_nr) { + /* + * Batch 0 is unpinned. See the note in + * `fsmonitor_new_token_data()` about why we + * don't need to accumulate these paths. + */ + fsmonitor_batch__pop(batch); + } else if (head->nr + batch->nr > MY_COMBINE_LIMIT) { + /* + * The head batch in the list has never been + * transmitted to a client, but folding the + * contents of the new batch onto it would + * exceed our arbitrary limit, so just prepend + * the new batch onto the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else { + /* + * We are free to append the paths in the given + * batch onto the end of the current head batch. + */ + fsmonitor_batch__combine(head, batch); + fsmonitor_batch__pop(batch); + } + } + + pthread_mutex_unlock(&state->main_lock); +} + static void *fsmonitor_fs_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -328,6 +551,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state) fsmonitor_fs_listen__loop(state); + pthread_mutex_lock(&state->main_lock); + if (state->current_token_data && + state->current_token_data->client_ref_count == 0) + fsmonitor_free_token_data(state->current_token_data); + state->current_token_data = NULL; + pthread_mutex_unlock(&state->main_lock); + trace2_thread_exit(); return NULL; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 7bbb3a27a1ce..89a9ef20b24b 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -12,6 +12,27 @@ struct fsmonitor_batch; struct fsmonitor_token_data; +/* + * Create a new batch of path(s). The returned batch is considered + * private and not linked into the fsmonitor daemon state. The caller + * should fill this batch with one or more paths and then publish it. + */ +struct fsmonitor_batch *fsmonitor_batch__new(void); + +/* + * Free this batch and return the value of the batch->next field. + */ +struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch); + +/* + * Add this path to this batch of modified files. + * + * The batch should be private and NOT (yet) linked into the fsmonitor + * daemon state and therefore not yet visible to worker threads and so + * no locking is required. + */ +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); + struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ struct fsmonitor_daemon_state { @@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( struct fsmonitor_daemon_state *state, const char *path); +/* + * Prepend the this batch of path(s) onto the list of batches associated + * with the current token. This makes the batch visible to worker threads. + * + * The caller no longer owns the batch and must not free it. + * + * Wake up the client threads waiting on these cookies. + */ +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names); + +/* + * If the platform-specific layer loses sync with the filesystem, + * it should call this to invalidate cached data and abort waiting + * threads. + */ +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Sat May 22 13:56:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F359C4707E for ; Sat, 22 May 2021 13:57:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F3563610A6 for ; Sat, 22 May 2021 13:57:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231316AbhEVN7C (ORCPT ); Sat, 22 May 2021 09:59:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231264AbhEVN6s (ORCPT ); Sat, 22 May 2021 09:58:48 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C26A4C061574 for ; Sat, 22 May 2021 06:57:22 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id u4-20020a05600c00c4b02901774b80945cso8907948wmm.3 for ; Sat, 22 May 2021 06:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2qfjyHGIj/ecIh92BpHHOx2mYJyFmHMDkPjRilrpnY0=; b=WfxxEXMDqQ0kYuy2MUmtp+HJTkSnB4X+7WazK/MC0s/iXFuofiTK/UVG9zdEq79Tbf OzQ+omrUj5ws9HyntHX3dt2Vu1TgSx2F7l6hcLUvDfL07JtjKZPiyjY+3Cc827hnfxFY sQ6i5NiW3oLIo66Koj8/B7cj/N+BxvXU1mZb7awfyGGcrS+R0kwYZV9jBnz8Fwnz8ULH IZ+3qC4L26CeQ//vIOnD1tsbjYtNKiOepD1mm/ugt57nfMXzkhn2fJ5eaFOMSlPs0kXL xL4FyJCgZBZa47hllYW8x/4sDPg3mJzjEnAsk9g+T669jxeuzszazg0JFKcVVeW66lU5 w46w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2qfjyHGIj/ecIh92BpHHOx2mYJyFmHMDkPjRilrpnY0=; b=DsAcbdX2werlRgjfgCQIyF0puCrcrxEEzSKTF0varKJuaTtZmRCmqDGBrPgCGoN684 3nHVs8JtXnhqhzz8+oLxH/qYaEUUYA8U0/SlyoNHpw50jPvQ6BZ9EaMRxavEZaM8lodE kCNoaL6beqhgaVnzBK16BAhnPe23YwI0X6L/wIhC0jIIX/+6MrxBfihnEBJP4r99KgD/ QIsEEhfDZxFJdXoYxZFulYzrp0NEfxVoUk31YNW3u/G88DtqEbtCDJgXwqgvkll76w7a jgcOPJJMhSJeLTuCiyCUIxToNJ3N7iKhuv5WcIwMaCP0kWUla7blYnzGpVj0BpUX83yg 8tMQ== X-Gm-Message-State: AOAM5319EL+WBO5nGmk76RYrgO2TPNyttwiBDos/5DU/eDw/ZSxAqQL4 pOTmq5QWU5+rBcguZtBD0eHQ0u1dPQo= X-Google-Smtp-Source: ABdhPJz9REhYd8PLYkKvU1Y1psbcFkVuhX5BwHAzheW3qBII28NmcPX/KXEoDGk5sNAsej/r/M2Nsg== X-Received: by 2002:a7b:cc83:: with SMTP id p3mr13301046wma.169.1621691841309; Sat, 22 May 2021 06:57:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o129sm2620892wmo.22.2021.05.22.06.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:20 -0700 (PDT) Message-Id: <9ea4b04b8215b1e397623ea4c63e6af3655c74d3.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:56 +0000 Subject: [PATCH v2 17/28] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the win32 backend to register a watch on the working tree root directory (recursively). Also watch the if it is not inside the working tree. And to collect path change notifications into batches and publish. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-win32.c | 532 +++++++++++++++++++ 1 file changed, 532 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c index 880446b49e35..ba087b292df6 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -2,20 +2,552 @@ #include "config.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +/* + * The documentation of ReadDirectoryChangesW() states that the maximum + * buffer size is 64K when the monitored directory is remote. + * + * Larger buffers may be used when the monitored directory is local and + * will help us receive events faster from the kernel and avoid dropped + * events. + * + * So we try to use a very large buffer and silently fallback to 64K if + * we get an error. + */ +#define MAX_RDCW_BUF_FALLBACK (65536) +#define MAX_RDCW_BUF (65536 * 8) + +struct one_watch +{ + char buffer[MAX_RDCW_BUF]; + DWORD buf_len; + DWORD count; + + struct strbuf path; + HANDLE hDir; + HANDLE hEvent; + OVERLAPPED overlapped; + + /* + * Is there an active ReadDirectoryChangesW() call pending. If so, we + * need to later call GetOverlappedResult() and possibly CancelIoEx(). + */ + BOOL is_active; +}; + +struct fsmonitor_daemon_backend_data +{ + struct one_watch *watch_worktree; + struct one_watch *watch_gitdir; + + HANDLE hEventShutdown; + + HANDLE hListener[3]; /* we don't own these handles */ +#define LISTENER_SHUTDOWN 0 +#define LISTENER_HAVE_DATA_WORKTREE 1 +#define LISTENER_HAVE_DATA_GITDIR 2 + int nr_listener_handles; +}; + +/* + * Convert the WCHAR path from the notification into UTF8 and + * then normalize it. + */ +static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, + struct strbuf *normalized_path) +{ + int reserve; + int len = 0; + + strbuf_reset(normalized_path); + if (!info->FileNameLength) + goto normalize; + + /* + * Pre-reserve enough space in the UTF8 buffer for + * each Unicode WCHAR character to be mapped into a + * sequence of 2 UTF8 characters. That should let us + * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. + */ + reserve = info->FileNameLength + 1; + strbuf_grow(normalized_path, reserve); + + for (;;) { + len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, + info->FileNameLength / sizeof(WCHAR), + normalized_path->buf, + strbuf_avail(normalized_path) - 1, + NULL, NULL); + if (len > 0) + goto normalize; + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + error("[GLE %ld] could not convert path to UTF-8: '%.*ls'", + GetLastError(), + (int)(info->FileNameLength / sizeof(WCHAR)), + info->FileName); + return -1; + } + + strbuf_grow(normalized_path, + strbuf_avail(normalized_path) + reserve); + } + +normalize: + strbuf_setlen(normalized_path, len); + return strbuf_normalize_path(normalized_path); +} void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) { + SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); +} + +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, + const char *path) +{ + struct one_watch *watch = NULL; + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileA(path, + desired_access, share_mode, NULL, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, + NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] could not watch '%s'"), + GetLastError(), path); + return NULL; + } + + CALLOC_ARRAY(watch, 1); + + watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ + + strbuf_init(&watch->path, 0); + strbuf_addstr(&watch->path, path); + + watch->hDir = hDir; + watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); + + return watch; +} + +static void destroy_watch(struct one_watch *watch) +{ + if (!watch) + return; + + strbuf_release(&watch->path); + if (watch->hDir != INVALID_HANDLE_VALUE) + CloseHandle(watch->hDir); + if (watch->hEvent != INVALID_HANDLE_VALUE) + CloseHandle(watch->hEvent); + + free(watch); +} + +static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, + struct one_watch *watch) +{ + DWORD dwNotifyFilter = + FILE_NOTIFY_CHANGE_FILE_NAME | + FILE_NOTIFY_CHANGE_DIR_NAME | + FILE_NOTIFY_CHANGE_ATTRIBUTES | + FILE_NOTIFY_CHANGE_SIZE | + FILE_NOTIFY_CHANGE_LAST_WRITE | + FILE_NOTIFY_CHANGE_CREATION; + + ResetEvent(watch->hEvent); + + memset(&watch->overlapped, 0, sizeof(watch->overlapped)); + watch->overlapped.hEvent = watch->hEvent; + +start_watch: + /* + * Queue an async call using Overlapped IO. This returns immediately. + * Our event handle will be signalled when the real result is available. + * + * The return value here just means that we successfully queued it. + * We won't know if the Read...() actually produces data until later. + */ + watch->is_active = ReadDirectoryChangesW( + watch->hDir, watch->buffer, watch->buf_len, TRUE, + dwNotifyFilter, &watch->count, &watch->overlapped, NULL); + + /* + * The kernel throws an invalid parameter error when our buffer + * is too big and we are pointed at a remote directory (and possibly + * for other reasons). Quietly set it down and try again. + * + * See note about MAX_RDCW_BUF at the top. + */ + if (!watch->is_active && + GetLastError() == ERROR_INVALID_PARAMETER && + watch->buf_len > MAX_RDCW_BUF_FALLBACK) { + watch->buf_len = MAX_RDCW_BUF_FALLBACK; + goto start_watch; + } + + if (watch->is_active) + return 0; + + error("ReadDirectoryChangedW failed on '%s' [GLE %ld]", + watch->path.buf, GetLastError()); + return -1; +} + +static int recv_rdcw_watch(struct one_watch *watch) +{ + watch->is_active = FALSE; + + /* + * The overlapped result is ready. If the Read...() was successful + * we finally receive the actual result into our buffer. + */ + if (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count, + TRUE)) + return 0; + + /* + * NEEDSWORK: If an external is deleted, the above + * returns an error. I'm not sure that there's anything that + * we can do here other than failing -- the /.git + * link file would be broken anyway. We might try to check + * for that and return a better error message, but I'm not + * sure it is worth it. + */ + + error("GetOverlappedResult failed on '%s' [GLE %ld]", + watch->path.buf, GetLastError()); + return -1; +} + +static void cancel_rdcw_watch(struct one_watch *watch) +{ + DWORD count; + + if (!watch || !watch->is_active) + return; + + /* + * The calls to ReadDirectoryChangesW() and GetOverlappedResult() + * form a "pair" (my term) where we queue an IO and promise to + * hang around and wait for the kernel to give us the result. + * + * If for some reason after we queue the IO, we have to quit + * or otherwise not stick around for the second half, we must + * tell the kernel to abort the IO. This prevents the kernel + * from writing to our buffer and/or signalling our event + * after we free them. + * + * (Ask me how much fun it was to track that one down). + */ + CancelIoEx(watch->hDir, &watch->overlapped); + GetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE); + watch->is_active = FALSE; +} + +/* + * Process filesystem events that happen anywhere (recursively) under the + * root directory. For a normal working directory, this includes + * both version controlled files and the contents of the .git/ directory. + * + * If /.git is a file, then we only see events for the file + * itself. + */ +static int process_worktree_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_worktree; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + struct fsmonitor_batch *batch = NULL; + const char *p = watch->buffer; + + /* + * If the kernel gets more events than will fit in the kernel + * buffer associated with our RDCW handle, it drops them and + * returns a count of zero. + * + * Yes, the call returns WITHOUT error and with length zero. + * + * (The "overflow" case is not ambiguous with the "no data" case + * because we did an INFINITE wait.) + * + * This means we have a gap in coverage. Tell the daemon layer + * to resync. + */ + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_WORKTREE; + } + + /* + * On Windows, `info` contains an "array" of paths that are + * relative to the root of whichever directory handle received + * the event. + */ + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_workdir_relative(path.buf); + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info->Action == FILE_ACTION_REMOVED) || + (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, path.buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, batch, &cookie_list); + batch = NULL; + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_WORKTREE; + +force_shutdown: + fsmonitor_batch__pop(batch); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_SHUTDOWN; +} + +/* + * Process filesystem events that happened anywhere (recursively) under the + * external (such as non-primary worktrees or submodules). + * We only care about cookie files that our client threads created here. + * + * Note that we DO NOT get filesystem events on the external + * itself (it is not inside something that we are watching). In particular, + * we do not get an event if the external is deleted. + */ +static int process_gitdir_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_gitdir; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *p = watch->buffer; + + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_GITDIR; + } + + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_gitdir_relative(path.buf); + + trace_printf_key(&trace_fsmonitor, "BBB: %s", path.buf); + + switch (t) { + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_GITDIR: + goto skip_this_path; + + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, NULL, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_GITDIR; } void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data = state->backend_data; + DWORD dwWait; + + state->error_code = 0; + + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + + if (data->watch_gitdir && + start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + + for (;;) { + dwWait = WaitForMultipleObjects(data->nr_listener_handles, + data->hListener, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) { + if (recv_rdcw_watch(data->watch_worktree) == -1) + goto force_error_stop; + if (process_worktree_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) { + if (recv_rdcw_watch(data->watch_gitdir) == -1) + goto force_error_stop; + if (process_gitdir_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN) + goto clean_shutdown; + + error(_("could not read directory changes [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->error_code = -1; + +force_shutdown: + /* + * Tell the IPC thead pool to stop (which completes the await + * in the main thread (which will also signal this thread (if + * we are still alive))). + */ + ipc_server_stop_async(state->ipc_server_data); + +clean_shutdown: + cancel_rdcw_watch(data->watch_worktree); + cancel_rdcw_watch(data->watch_gitdir); } int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->watch_worktree = create_watch(state, + state->path_worktree_watch.buf); + if (!data->watch_worktree) + goto failed; + + if (state->nr_paths_watching > 1) { + data->watch_gitdir = create_watch(state, + state->path_gitdir_watch.buf); + if (!data->watch_gitdir) + goto failed; + } + + data->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown; + data->nr_listener_handles++; + + data->hListener[LISTENER_HAVE_DATA_WORKTREE] = + data->watch_worktree->hEvent; + data->nr_listener_handles++; + + if (data->watch_gitdir) { + data->hListener[LISTENER_HAVE_DATA_GITDIR] = + data->watch_gitdir->hEvent; + data->nr_listener_handles++; + } + + state->backend_data = data; + return 0; + +failed: + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + return -1; } void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + + FREE_AND_NULL(state->backend_data); } From patchwork Sat May 22 13:56:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D7FBC2B9FB for ; Sat, 22 May 2021 13:57:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E6337610A6 for ; Sat, 22 May 2021 13:57:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231431AbhEVN7N (ORCPT ); Sat, 22 May 2021 09:59:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231267AbhEVN6t (ORCPT ); Sat, 22 May 2021 09:58:49 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32C7AC06174A for ; Sat, 22 May 2021 06:57:23 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id p7so20002719wru.10 for ; Sat, 22 May 2021 06:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=edBI/UfAdUANph40tPYHwXApM7BGoIz/01tZ6bP76xw=; b=nWEc6Rme/FytYr0E0k2NBghf5mLwqj+pasb/Q7/Y/7oAfw1ZGONynb4nC/AxNxvOiO bV/u2kgW7Q58ZwkFRQ+Wsx5VMEtkhokZDApy9m1+iIL58x8j3b/YqR0w/F9dyWHK5EsN 6X3iIXU8zBkonNNE0EL+yHSEB8b0pNedE9aRO2UahewM4v5fL29/nk1TBa3wtuG2av6z 2odB5k3+nmuqE0uPDODvHC5yRnn7c4jnYzEltXm/zKVFIar2sDQOVliESBFxstASZPvo D4P588IGNAN6giIuPGUzjGCcs+TyQk2S9NqF0fK0j5BOWUUG6MYwku1qVULQSgpDGEbm tcAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=edBI/UfAdUANph40tPYHwXApM7BGoIz/01tZ6bP76xw=; b=t9VR0pbc4WB2RBxDf/wlLvHWI7D3lG5y2uO+uwhVV1kcHSOq6A9ufPo3REudQ3ndbD SUzZI1OnGUqW9HJnNID+d0aeecyFkj+toFmDMjJTwn7q0kbhyul5ddSJ8H71r/DJlBo8 yUvKWO1gfnEooKvIf8iuuxA3+DuX19GvlaOXGrYLPaLD+DqYbhuz3HhH3XPvZxOgXe9g nbKNr0qQeHj85TEDqCFXviX0RhGhKysQ+CtxArGbfkqA1x+r0f4BsBuKtu8U8iFXjQL3 aolLhwDDYF4fBntJdGtowSE8FyYvvH1SvwXA3mKBUEGtp10mppgCIsTxgMxzqvnTPvGH vDOg== X-Gm-Message-State: AOAM5338E3qoLhQsV1//msXBCDSkL/+5y+L748VgulE07sQi3f8Qx+G6 jYmkqa1h+xZsXf+XFMEXeujYrroX+xE= X-Google-Smtp-Source: ABdhPJy39b5vmFCST/VFiXtTwduhKSyDZ/KYRRJlunj1ULvnS7nI7/UXDRCvzZkVTvLt7nF3DkRRUw== X-Received: by 2002:a05:6000:188d:: with SMTP id a13mr14334939wri.61.1621691841849; Sat, 22 May 2021 06:57:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r2sm5817069wrv.39.2021.05.22.06.57.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:21 -0700 (PDT) Message-Id: <21b2b4f941b20ff50fd7de0d777fa89d27729d6d.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:57 +0000 Subject: [PATCH v2 18/28] fsmonitor-fs-listen-macos: add macos header files for FSEvent Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Include MacOS system declarations to allow us to use FSEvent and CoreFoundation APIs. We need GCC and clang versions because of compiler and header file conflicts. While it is quite possible to #include Apple's CoreServices.h when compiling C source code with clang, trying to build it with GCC currently fails with this error: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45, /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope 193 | char bytes[kAuthorizationExternalFormLength]; | ^~~~~ The underlying reason is that GCC (rightfully) objects that an `enum` value such as `kAuthorizationExternalFormLength` is not a constant (because it is not, the preprocessor has no knowledge of it, only the actual C compiler does) and can therefore not be used to define the size of a C array. This is a known problem and tracked in GCC's bug tracker: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 In the meantime, let's not block things and go the slightly ugly route of declaring/defining the FSEvents constants, data structures and functions that we need, so that we can avoid above-mentioned issue. Let's do this _only_ for GCC, though, so that the CI/PR builds (which build both with clang and with GCC) can guarantee that we _are_ using the correct data types. Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c index b91058d1c4f8..bec5130d9e1d 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -1,3 +1,99 @@ +#if defined(__GNUC__) +/* + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling + * with clang, but not with GCC as of time of writing. + * + * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details. + */ +typedef unsigned int FSEventStreamCreateFlags; +#define kFSEventStreamEventFlagNone 0x00000000 +#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001 +#define kFSEventStreamEventFlagUserDropped 0x00000002 +#define kFSEventStreamEventFlagKernelDropped 0x00000004 +#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008 +#define kFSEventStreamEventFlagHistoryDone 0x00000010 +#define kFSEventStreamEventFlagRootChanged 0x00000020 +#define kFSEventStreamEventFlagMount 0x00000040 +#define kFSEventStreamEventFlagUnmount 0x00000080 +#define kFSEventStreamEventFlagItemCreated 0x00000100 +#define kFSEventStreamEventFlagItemRemoved 0x00000200 +#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400 +#define kFSEventStreamEventFlagItemRenamed 0x00000800 +#define kFSEventStreamEventFlagItemModified 0x00001000 +#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000 +#define kFSEventStreamEventFlagItemChangeOwner 0x00004000 +#define kFSEventStreamEventFlagItemXattrMod 0x00008000 +#define kFSEventStreamEventFlagItemIsFile 0x00010000 +#define kFSEventStreamEventFlagItemIsDir 0x00020000 +#define kFSEventStreamEventFlagItemIsSymlink 0x00040000 +#define kFSEventStreamEventFlagOwnEvent 0x00080000 +#define kFSEventStreamEventFlagItemIsHardlink 0x00100000 +#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000 +#define kFSEventStreamEventFlagItemCloned 0x00400000 + +typedef struct __FSEventStream *FSEventStreamRef; +typedef const FSEventStreamRef ConstFSEventStreamRef; + +typedef unsigned int CFStringEncoding; +#define kCFStringEncodingUTF8 0x08000100 + +typedef const struct __CFString *CFStringRef; +typedef const struct __CFArray *CFArrayRef; +typedef const struct __CFRunLoop *CFRunLoopRef; + +struct FSEventStreamContext { + long long version; + void *cb_data, *retain, *release, *copy_description; +}; + +typedef struct FSEventStreamContext FSEventStreamContext; +typedef unsigned int FSEventStreamEventFlags; +#define kFSEventStreamCreateFlagNoDefer 0x02 +#define kFSEventStreamCreateFlagWatchRoot 0x04 +#define kFSEventStreamCreateFlagFileEvents 0x10 + +typedef unsigned long long FSEventStreamEventId; +#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL + +typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef, + void *context, + __SIZE_TYPE__ num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]); +typedef double CFTimeInterval; +FSEventStreamRef FSEventStreamCreate(void *allocator, + FSEventStreamCallback callback, + FSEventStreamContext *context, + CFArrayRef paths_to_watch, + FSEventStreamEventId since_when, + CFTimeInterval latency, + FSEventStreamCreateFlags flags); +CFStringRef CFStringCreateWithCString(void *allocator, const char *string, + CFStringEncoding encoding); +CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count, + void *callbacks); +void CFRunLoopRun(void); +void CFRunLoopStop(CFRunLoopRef run_loop); +CFRunLoopRef CFRunLoopGetCurrent(void); +extern CFStringRef kCFRunLoopDefaultMode; +void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream, + CFRunLoopRef run_loop, + CFStringRef run_loop_mode); +unsigned char FSEventStreamStart(FSEventStreamRef stream); +void FSEventStreamStop(FSEventStreamRef stream); +void FSEventStreamInvalidate(FSEventStreamRef stream); +void FSEventStreamRelease(FSEventStreamRef stream); +#else +/* + * Let Apple's headers declare `isalnum()` first, before + * Git's headers override it via a constant + */ +#include +#include +#include +#endif + #include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" From patchwork Sat May 22 13:56:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EE91C2B9F2 for ; Sat, 22 May 2021 13:57:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 015B86115C for ; Sat, 22 May 2021 13:57:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231325AbhEVN7L (ORCPT ); Sat, 22 May 2021 09:59:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231150AbhEVN6t (ORCPT ); Sat, 22 May 2021 09:58:49 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3555C06138B for ; Sat, 22 May 2021 06:57:23 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id s5-20020a7bc0c50000b0290147d0c21c51so8616579wmh.4 for ; Sat, 22 May 2021 06:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NpG7cYalCLFK2akjzoh3dbkDBeYIApnMnViNCeeq0t8=; b=utn2lvX5uSUSztMeOvVyq3IHBw1DpSivpuXnRfR5usgMy4tt7pDqSuX0EZRsFrE3gk WsKlgq8r057vRV89mL+ZJI8UhOOWQp9JJzxRDi3N6lYT5201o9+PtV+ME/vdY0b6o0DG uRcr0a5bl4XFe6dpFX748JF+4Ktsf4MEguAyt75PHHX3IqfajcWzS/P0UdU48qlUfaOH DEWk3FiIrbvlTQBszGk5LNfflJy/QwkpPlBoP3KXJp6NRXqkJocZMZry0+eSRjNjMbJQ EZ/MB7WfhwjShzpJKf+CT5BDDgdHvMBm9G5AR7i9qlsbqPBNUGXDvcuVgCedEuQMbjci so0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NpG7cYalCLFK2akjzoh3dbkDBeYIApnMnViNCeeq0t8=; b=MPQZKzwDve57yk2mWXXFfCzZCmQo/ufmbOx+IryCZ/+1+u75g/nVPp3bFPdnn/JGfy 9/PeRQ9q2pT+FDicYjh958bSw/4O120Ryy/lv0aSeZyKt67t7lD8cW1p6jnnc8PxDcRF k5OMTe6nWpgYS+E49VoaDlQO4T1qcfOGXG6ieVIk64G2rxEgsKGbRfsnvzQIHBiptbEc qgXCezTwqfeHS0CmHZHifI490oFoGfd3ZZiysyA1faus321Vn3SolD581bCSKcSoE80E MlWeKLnED1UfTGK2cpuhd4SbvjBFPe69Iz8Gichd0c/GChbpecPv9Z21KHjHm5Y40Yns bTRg== X-Gm-Message-State: AOAM532GsXd6DD7eV8HfF2hcwLx0G1K48f2m/euCDmt006Bfr+M1qJ3H Lcl8aDS7O92grT1pUhFjujyQiZkaZ0I= X-Google-Smtp-Source: ABdhPJwLS+o0fLb8Gp6jDg3PpmTrlZVjBUQTDSYrHqmve24hlmg4h4kkPg+ME9EfkwTE9GAEOv+F6g== X-Received: by 2002:a1c:7515:: with SMTP id o21mr13038844wmc.90.1621691842490; Sat, 22 May 2021 06:57:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h10sm5462319wrt.3.2021.05.22.06.57.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:22 -0700 (PDT) Message-Id: <08474bad8303b1d9b1315af3fe97393190abb261.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:56:58 +0000 Subject: [PATCH v2 19/28] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement file system event listener on MacOS using FSEvent, CoreFoundation, and CoreServices. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 381 +++++++++++++++++++ 1 file changed, 381 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c index bec5130d9e1d..02f89de216e9 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -97,20 +97,401 @@ void FSEventStreamRelease(FSEventStreamRef stream); #include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +struct fsmonitor_daemon_backend_data +{ + CFStringRef cfsr_worktree_path; + CFStringRef cfsr_gitdir_path; + + CFArrayRef cfar_paths_to_watch; + int nr_paths_watching; + + FSEventStreamRef stream; + + CFRunLoopRef rl; + + enum shutdown_style { + SHUTDOWN_EVENT = 0, + FORCE_SHUTDOWN, + FORCE_ERROR_STOP, + } shutdown_style; + + unsigned int stream_scheduled:1; + unsigned int stream_started:1; +}; + +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) +{ + struct strbuf msg = STRBUF_INIT; + + if (flag & kFSEventStreamEventFlagMustScanSubDirs) + strbuf_addstr(&msg, "MustScanSubDirs|"); + if (flag & kFSEventStreamEventFlagUserDropped) + strbuf_addstr(&msg, "UserDropped|"); + if (flag & kFSEventStreamEventFlagKernelDropped) + strbuf_addstr(&msg, "KernelDropped|"); + if (flag & kFSEventStreamEventFlagEventIdsWrapped) + strbuf_addstr(&msg, "EventIdsWrapped|"); + if (flag & kFSEventStreamEventFlagHistoryDone) + strbuf_addstr(&msg, "HistoryDone|"); + if (flag & kFSEventStreamEventFlagRootChanged) + strbuf_addstr(&msg, "RootChanged|"); + if (flag & kFSEventStreamEventFlagMount) + strbuf_addstr(&msg, "Mount|"); + if (flag & kFSEventStreamEventFlagUnmount) + strbuf_addstr(&msg, "Unmount|"); + if (flag & kFSEventStreamEventFlagItemChangeOwner) + strbuf_addstr(&msg, "ItemChangeOwner|"); + if (flag & kFSEventStreamEventFlagItemCreated) + strbuf_addstr(&msg, "ItemCreated|"); + if (flag & kFSEventStreamEventFlagItemFinderInfoMod) + strbuf_addstr(&msg, "ItemFinderInfoMod|"); + if (flag & kFSEventStreamEventFlagItemInodeMetaMod) + strbuf_addstr(&msg, "ItemInodeMetaMod|"); + if (flag & kFSEventStreamEventFlagItemIsDir) + strbuf_addstr(&msg, "ItemIsDir|"); + if (flag & kFSEventStreamEventFlagItemIsFile) + strbuf_addstr(&msg, "ItemIsFile|"); + if (flag & kFSEventStreamEventFlagItemIsHardlink) + strbuf_addstr(&msg, "ItemIsHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsLastHardlink) + strbuf_addstr(&msg, "ItemIsLastHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsSymlink) + strbuf_addstr(&msg, "ItemIsSymlink|"); + if (flag & kFSEventStreamEventFlagItemModified) + strbuf_addstr(&msg, "ItemModified|"); + if (flag & kFSEventStreamEventFlagItemRemoved) + strbuf_addstr(&msg, "ItemRemoved|"); + if (flag & kFSEventStreamEventFlagItemRenamed) + strbuf_addstr(&msg, "ItemRenamed|"); + if (flag & kFSEventStreamEventFlagItemXattrMod) + strbuf_addstr(&msg, "ItemXattrMod|"); + if (flag & kFSEventStreamEventFlagOwnEvent) + strbuf_addstr(&msg, "OwnEvent|"); + if (flag & kFSEventStreamEventFlagItemCloned) + strbuf_addstr(&msg, "ItemCloned|"); + + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + path, flag, msg.buf); + + strbuf_release(&msg); +} + +static int ef_is_root_delete(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRemoved); +} + +static int ef_is_root_renamed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRenamed); +} + +static int ef_is_dropped(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagKernelDropped || + ef & kFSEventStreamEventFlagUserDropped); +} + +static void fsevent_callback(ConstFSEventStreamRef streamRef, + void *ctx, + size_t num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]) +{ + struct fsmonitor_daemon_state *state = ctx; + struct fsmonitor_daemon_backend_data *data = state->backend_data; + char **paths = (char **)event_paths; + struct fsmonitor_batch *batch = NULL; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *path_k; + const char *slash; + int k; + struct strbuf tmp = STRBUF_INIT; + + /* + * Build a list of all filesystem changes into a private/local + * list and without holding any locks. + */ + for (k = 0; k < num_of_events; k++) { + /* + * On Mac, we receive an array of absolute paths. + */ + path_k = paths[k]; + + /* + * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. + * Please don't log them to Trace2. + * + * trace_printf_key(&trace_fsmonitor, "Path: '%s'", path_k); + */ + + /* + * If event[k] is marked as dropped, we assume that we have + * lost sync with the filesystem and should flush our cached + * data. We need to: + * + * [1] Abort/wake any client threads waiting for a cookie and + * flush the cached state data (the current token), and + * create a new token. + * + * [2] Discard the batch that we were locally building (since + * they are conceptually relative to the just flushed + * token). + */ + if (ef_is_dropped(event_flags[k])) { + /* + * see also kFSEventStreamEventFlagMustScanSubDirs + */ + trace_printf_key(&trace_fsmonitor, "event: dropped"); + + fsmonitor_force_resync(state); + fsmonitor_batch__pop(batch); + string_list_clear(&cookie_list, 0); + + /* + * We assume that any events that we received + * in this callback after this dropped event + * may still be valid, so we continue rather + * than break. (And just in case there is a + * delete of ".git" hiding in there.) + */ + continue; + } + + switch (fsmonitor_classify_path_absolute(state, path_k)) { + + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within .git or gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path_k); + string_list_append(&cookie_list, + slash ? slash + 1 : path_k); + break; + + case IS_INSIDE_DOT_GIT: + case IS_INSIDE_GITDIR: + /* ignore all other paths inside of .git or gitdir */ + break; + + case IS_DOT_GIT: + case IS_GITDIR: + /* + * If .git directory is deleted or renamed away, + * we have to quit. + */ + if (ef_is_root_delete(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "event: gitdir removed"); + goto force_shutdown; + } + if (ef_is_root_renamed(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "event: gitdir renamed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* try to queue normal pathnames */ + + if (trace_pass_fl(&trace_fsmonitor)) + log_flags_set(path_k, event_flags[k]); + + /* + * Because of the implicit "binning" (the + * kernel calls us at a given frequency) and + * de-duping (the kernel is free to combine + * multiple events for a given pathname), an + * individual fsevent could be marked as both + * a file and directory. Add it to the queue + * with both spellings so that the client will + * know how much to invalidate/refresh. + */ + + if (event_flags[k] & kFSEventStreamEventFlagItemIsFile) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, rel); + } + + if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + strbuf_reset(&tmp); + strbuf_addstr(&tmp, rel); + strbuf_addch(&tmp, '/'); + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, tmp.buf); + } + + break; + + case IS_OUTSIDE_CONE: + default: + trace_printf_key(&trace_fsmonitor, + "ignoring '%s'", path_k); + break; + } + } + + fsmonitor_publish(state, batch, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&tmp); + return; + +force_shutdown: + fsmonitor_batch__pop(batch); + string_list_clear(&cookie_list, 0); + + data->shutdown_style = FORCE_SHUTDOWN; + CFRunLoopStop(data->rl); + strbuf_release(&tmp); + return; +} + +/* + * NEEDSWORK: Investigate the proper value for the `latency` argument + * in the call to `FSEventStreamCreate()`. I'm not sure that this + * needs to be a config setting or just something that we tune after + * some testing. + * + * With a latency of 0.1, I was seeing lots of dropped events during + * the "touch 100000" files test within t/perf/p7519, but with a + * latency of 0.001 I did not see any dropped events. So the + * "correct" value may be somewhere in between. + * + * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate + */ int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { + FSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer | + kFSEventStreamCreateFlagWatchRoot | + kFSEventStreamCreateFlagFileEvents; + FSEventStreamContext ctx = { + 0, + state, + NULL, + NULL, + NULL + }; + struct fsmonitor_daemon_backend_data *data; + const void *dir_array[2]; + + CALLOC_ARRAY(data, 1); + state->backend_data = data; + + data->cfsr_worktree_path = CFStringCreateWithCString( + NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_worktree_path; + + if (state->nr_paths_watching > 1) { + data->cfsr_gitdir_path = CFStringCreateWithCString( + NULL, state->path_gitdir_watch.buf, + kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path; + } + + data->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array, + data->nr_paths_watching, + NULL); + data->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx, + data->cfar_paths_to_watch, + kFSEventStreamEventIdSinceNow, + 0.001, flags); + if (data->stream == NULL) + goto failed; + + /* + * `data->rl` needs to be set inside the listener thread. + */ + + return 0; + +failed: + error("Unable to create FSEventStream."); + + FREE_AND_NULL(state->backend_data); return -1; } void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + if (data->stream) { + if (data->stream_started) + FSEventStreamStop(data->stream); + if (data->stream_scheduled) + FSEventStreamInvalidate(data->stream); + FSEventStreamRelease(data->stream); + } + + FREE_AND_NULL(state->backend_data); } void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + data->shutdown_style = SHUTDOWN_EVENT; + + CFRunLoopStop(data->rl); } void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + + data->rl = CFRunLoopGetCurrent(); + + FSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode); + data->stream_scheduled = 1; + + if (!FSEventStreamStart(data->stream)) { + error("Failed to start the FSEventStream"); + goto force_error_stop_without_loop; + } + data->stream_started = 1; + + CFRunLoopRun(); + + switch (data->shutdown_style) { + case FORCE_ERROR_STOP: + state->error_code = -1; + /* fall thru */ + case FORCE_SHUTDOWN: + ipc_server_stop_async(state->ipc_server_data); + /* fall thru */ + case SHUTDOWN_EVENT: + default: + break; + } + return; + +force_error_stop_without_loop: + state->error_code = -1; + ipc_server_stop_async(state->ipc_server_data); + return; } From patchwork Sat May 22 13:56:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79846C2B9FB for ; Sat, 22 May 2021 13:57:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 581A86115C for ; Sat, 22 May 2021 13:57:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231405AbhEVN7H (ORCPT ); Sat, 22 May 2021 09:59:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231279AbhEVN6u (ORCPT ); Sat, 22 May 2021 09:58:50 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85678C06138C for ; Sat, 22 May 2021 06:57:24 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id n2so23816201wrm.0 for ; Sat, 22 May 2021 06:57:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8w2tffOGt/kmaTT7ylsf+xJSswz23K2L+q3i3VMMYS8=; b=oIIz3TvBsBiipZ+RrUkDThvvmRM5aHB0mpda+COwfKY/A7cTXfKFBPG/38FwO0kEpS nONbSOBWKjByeVDqE7oWW6g0ZjBnoYZKXUJavf/mBWrfPeENk/ywxqIJ1OKslVja4Cu3 a+5FWmU2hTkDhLJjvDyGfuG53V+g906szvPSXVxHomAnhVzELfdd94kf71O2DIZqBhrO eovMoKGbi/wPvm5PHOXJrryhdCq2NwviZhJX9wS6ANwFdb84vOLxVy6ZlamweSDM5FHe xOoFBPpNCulUQcMedq3jGlnQIZuNsUdXWsfwqYZYyj5KsMEAl9UZkO/do96QQkLbFQxk x6qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8w2tffOGt/kmaTT7ylsf+xJSswz23K2L+q3i3VMMYS8=; b=Qw8pmXs5qBUtdN/t0dylayV/O0wsN7EC4IT5fR/WTPRBnXrv/xvPP523kUiuBKpCR5 1/jFoPbZ5TMntIpy+PW10eBlUStGZ1bibcVfXs0fcYH9heb1f8EzgI7iduDTWsrF8Kis b3ig1Tn/OHoH+qGTgcZ1efXBhrB5e+x6nTcgOGaQE45XFRlcJEzjbhwSBleCPeLVbQtW g8zXT7AttrcZv8LqRw0xcaqX+OGvo24L/eZDTjlU+0lu2twNE+woPDmBoETQZJTbfIWk o5k1T7nbJzBxcVucQkKypVpBydeMpK7cpNDMUCVn4CdN3od/gaJ1yMj1rPSqJCJPuYkR 3Buw== X-Gm-Message-State: AOAM530FE/VEyYk/mK/IeniVORk9d5N+vC5AD5K4EStJiHCBUUXlU347 vXWr0s909ovBuSVkZbSLF+GdB5tevkc= X-Google-Smtp-Source: ABdhPJzv0xZoOr7/pDPSLdJ8VXp/PHeZxYD6jBrUpnjzQ5WazHRnOWsA1h4p2L0hKDYgUoQmW+ULhQ== X-Received: by 2002:a5d:4346:: with SMTP id u6mr14302585wrr.304.1621691843104; Sat, 22 May 2021 06:57:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm5412477wrt.97.2021.05.22.06.57.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:22 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:56:59 +0000 Subject: [PATCH v2 20/28] fsmonitor--daemon: implement handle_client callback Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to respond to IPC requests from client Git processes and respond with a list of modified pathnames relative to the provided token. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++- 1 file changed, 310 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 663fead0d66e..33b4f09c72ca 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,6 +7,7 @@ #include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" +#include "pkt-line.h" static const char * const builtin_fsmonitor__daemon_usage[] = { N_("git fsmonitor--daemon start []"), @@ -353,6 +354,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) pthread_mutex_unlock(&state->main_lock); } +/* + * Format an opaque token string to send to the client. + */ +static void with_lock__format_response_token( + struct strbuf *response_token, + const struct strbuf *response_token_id, + const struct fsmonitor_batch *batch) +{ + /* assert current thread holding state->main_lock */ + + strbuf_reset(response_token); + strbuf_addf(response_token, "builtin:%s:%"PRIu64, + response_token_id->buf, batch->batch_seq_nr); +} + +/* + * Parse an opaque token from the client. + * Returns -1 on error. + */ +static int fsmonitor_parse_client_token(const char *buf_token, + struct strbuf *requested_token_id, + uint64_t *seq_nr) +{ + const char *p; + char *p_end; + + strbuf_reset(requested_token_id); + *seq_nr = 0; + + if (!skip_prefix(buf_token, "builtin:", &p)) + return -1; + + while (*p && *p != ':') + strbuf_addch(requested_token_id, *p++); + if (!*p++) + return -1; + + *seq_nr = (uint64_t)strtoumax(p, &p_end, 10); + if (*p_end) + return -1; + + return 0; +} + +KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal); + +static int do_handle_client(struct fsmonitor_daemon_state *state, + const char *command, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + struct fsmonitor_token_data *token_data = NULL; + struct strbuf response_token = STRBUF_INIT; + struct strbuf requested_token_id = STRBUF_INIT; + struct strbuf payload = STRBUF_INIT; + uint64_t requested_oldest_seq_nr = 0; + uint64_t total_response_len = 0; + const char *p; + const struct fsmonitor_batch *batch_head; + const struct fsmonitor_batch *batch; + intmax_t count = 0, duplicates = 0; + kh_str_t *shown; + int hash_ret; + int do_trivial = 0; + int do_flush = 0; + + /* + * We expect `command` to be of the form: + * + * := quit NUL + * | flush NUL + * | NUL + * | NUL + */ + + if (!strcmp(command, "quit")) { + /* + * A client has requested over the socket/pipe that the + * daemon shutdown. + * + * Tell the IPC thread pool to shutdown (which completes + * the await in the main thread (which can stop the + * fsmonitor listener thread)). + * + * There is no reply to the client. + */ + return SIMPLE_IPC_QUIT; + + } else if (!strcmp(command, "flush")) { + /* + * Flush all of our cached data and generate a new token + * just like if we lost sync with the filesystem. + * + * Then send a trivial response using the new token. + */ + do_flush = 1; + do_trivial = 1; + + } else if (!skip_prefix(command, "builtin:", &p)) { + /* assume V1 timestamp or garbage */ + + char *p_end; + + strtoumax(command, &p_end, 10); + trace_printf_key(&trace_fsmonitor, + ((*p_end) ? + "fsmonitor: invalid command line '%s'" : + "fsmonitor: unsupported V1 protocol '%s'"), + command); + do_trivial = 1; + + } else { + /* We have "builtin:*" */ + if (fsmonitor_parse_client_token(command, &requested_token_id, + &requested_oldest_seq_nr)) { + trace_printf_key(&trace_fsmonitor, + "fsmonitor: invalid V2 protocol token '%s'", + command); + do_trivial = 1; + + } else { + /* + * We have a V2 valid token: + * "builtin::" + */ + } + } + + pthread_mutex_lock(&state->main_lock); + + if (!state->current_token_data) + BUG("fsmonitor state does not have a current token"); + + if (do_flush) + with_lock__do_force_resync(state); + + /* + * We mark the current head of the batch list as "pinned" so + * that the listener thread will treat this item as read-only + * (and prevent any more paths from being added to it) from + * now on. + */ + token_data = state->current_token_data; + batch_head = token_data->batch_head; + ((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL); + + /* + * FSMonitor Protocol V2 requires that we send a response header + * with a "new current token" and then all of the paths that changed + * since the "requested token". We send the seq_nr of the just-pinned + * head batch so that future requests from a client will be relative + * to it. + */ + with_lock__format_response_token(&response_token, + &token_data->token_id, batch_head); + + reply(reply_data, response_token.buf, response_token.len + 1); + total_response_len += response_token.len + 1; + + trace2_data_string("fsmonitor", the_repository, "response/token", + response_token.buf); + trace_printf_key(&trace_fsmonitor, "response token: %s", + response_token.buf); + + if (!do_trivial) { + if (strcmp(requested_token_id.buf, token_data->token_id.buf)) { + /* + * The client last spoke to a different daemon + * instance -OR- the daemon had to resync with + * the filesystem (and lost events), so reject. + */ + trace2_data_string("fsmonitor", the_repository, + "response/token", "different"); + do_trivial = 1; + + } else if (requested_oldest_seq_nr < + token_data->batch_tail->batch_seq_nr) { + /* + * The client wants older events than we have for + * this token_id. This means that the end of our + * batch list was truncated and we cannot give the + * client a complete snapshot relative to their + * request. + */ + trace_printf_key(&trace_fsmonitor, + "client requested truncated data"); + do_trivial = 1; + } + } + + if (do_trivial) { + pthread_mutex_unlock(&state->main_lock); + + reply(reply_data, "/", 2); + + trace2_data_intmax("fsmonitor", the_repository, + "response/trivial", 1); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + return 0; + } + + /* + * We're going to hold onto a pointer to the current + * token-data while we walk the list of batches of files. + * During this time, we will NOT be under the lock. + * So we ref-count it. + * + * This allows the listener thread to continue prepending + * new batches of items to the token-data (which we'll ignore). + * + * AND it allows the listener thread to do a token-reset + * (and install a new `current_token_data`). + */ + token_data->client_ref_count++; + + pthread_mutex_unlock(&state->main_lock); + + /* + * The client request is relative to the token that they sent, + * so walk the batch list backwards from the current head back + * to the batch (sequence number) they named. + * + * We use khash to de-dup the list of pathnames. + * + * NEEDSWORK: each batch contains a list of interned strings, + * so we only need to do pointer comparisons here to build the + * hash table. Currently, we're still comparing the string + * values. + */ + shown = kh_init_str(); + for (batch = batch_head; + batch && batch->batch_seq_nr > requested_oldest_seq_nr; + batch = batch->next) { + size_t k; + + for (k = 0; k < batch->nr; k++) { + const char *s = batch->interned_paths[k]; + size_t s_len; + + if (kh_get_str(shown, s) != kh_end(shown)) + duplicates++; + else { + kh_put_str(shown, s, &hash_ret); + + trace_printf_key(&trace_fsmonitor, + "send[%"PRIuMAX"]: %s", + count, s); + + /* Each path gets written with a trailing NUL */ + s_len = strlen(s) + 1; + + if (payload.len + s_len >= + LARGE_PACKET_DATA_MAX) { + reply(reply_data, payload.buf, + payload.len); + total_response_len += payload.len; + strbuf_reset(&payload); + } + + strbuf_add(&payload, s, s_len); + count++; + } + } + } + + if (payload.len) { + reply(reply_data, payload.buf, payload.len); + total_response_len += payload.len; + } + + kh_release_str(shown); + + pthread_mutex_lock(&state->main_lock); + + if (token_data->client_ref_count > 0) + token_data->client_ref_count--; + + if (token_data->client_ref_count == 0) { + if (token_data != state->current_token_data) { + /* + * The listener thread did a token-reset while we were + * walking the batch list. Therefore, this token is + * stale and can be discarded completely. If we are + * the last reader thread using this token, we own + * that work. + */ + fsmonitor_free_token_data(token_data); + } + } + + pthread_mutex_unlock(&state->main_lock); + + trace2_data_intmax("fsmonitor", the_repository, "response/length", total_response_len); + trace2_data_intmax("fsmonitor", the_repository, "response/count/files", count); + trace2_data_intmax("fsmonitor", the_repository, "response/count/duplicates", duplicates); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + strbuf_release(&payload); + + return 0; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -360,7 +666,7 @@ static int handle_client(void *data, ipc_server_reply_cb *reply, struct ipc_server_reply_data *reply_data) { - /* struct fsmonitor_daemon_state *state = data; */ + struct fsmonitor_daemon_state *state = data; int result; /* @@ -371,10 +677,12 @@ static int handle_client(void *data, if (command_len != strlen(command)) BUG("FSMonitor assumes text messages"); + trace_printf_key(&trace_fsmonitor, "requested token: %s", command); + trace2_region_enter("fsmonitor", "handle_client", the_repository); trace2_data_string("fsmonitor", the_repository, "request", command); - result = 0; /* TODO Do something here. */ + result = do_handle_client(state, command, reply, reply_data); trace2_region_leave("fsmonitor", "handle_client", the_repository); From patchwork Sat May 22 13:57:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0DFEC2B9F2 for ; Sat, 22 May 2021 13:57:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B3AA66115C for ; Sat, 22 May 2021 13:57:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231355AbhEVN7J (ORCPT ); Sat, 22 May 2021 09:59:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231193AbhEVN6v (ORCPT ); Sat, 22 May 2021 09:58:51 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0AFBC06138D for ; Sat, 22 May 2021 06:57:24 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id o127so12549199wmo.4 for ; Sat, 22 May 2021 06:57:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=C/2I6MzuLOLiUVsctx/0Xt+MswXfzfsVGvZCqKl/OD8=; b=iw/zf6AB88QNA/5xHva8w8wZvZO2mGvZS4YWXs8CNxYD+JsC08Q26Mh6lXreIsV4VW moVDSPSgpx/+sjbZTIJwbCWfWFa4VgxLVeO0C2HhPIxgjwU5Q7U6NGiGBVkfn8jHOMH1 jWgsaulgnVJSDwx8UO6ek7Z/n4GzTtAKuK68KvPiJ1j/LCjcT4SkYVztMemuZ+3rFKSw Pn+TDEv9z1zaeULPY3Hu/+0JdjZiBqwQvwARCglt5YGdaDHwmHrtxbWLvsgNQvGHGYx6 AonYqeRcAuCfrPxMjEAp892EfTq67iUqiFTeUFolQ7XFcy9xQkW7KUcB4Mj+x9V5rAR5 kFqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=C/2I6MzuLOLiUVsctx/0Xt+MswXfzfsVGvZCqKl/OD8=; b=gJKtKeQPrCYSc37nthHTnSphQvUSwTh4m1W+PTwzi4AqjYmstiNnJ1FEHK+QlSJbyG b5V7vv5tmIXebz6bGuAljtnGkMgppuLuVjbnh2X5otGvuxZk70uCDMarT37pfwv6vKEj k5U3pC506icqiJqOkB8Q170ZmlhR0/cokhejsViskf/gH2bJ7tyKSY8Id554fFvteegO mGtnjgs21AseftaNCvy8rmYzFc5Aae3vjVq0pbOcpsPxr/bfOwJc5kdChFshjs+Qb/Yq 9kLur7fP0VK+lpn1n2jW2lqnFS5Z6lBnZGFVRq56Iwjeosp9BOw4+uz4TRJWUgh5nhBb /j7w== X-Gm-Message-State: AOAM530+bHclU1/YzWjrkGB/jrQ8hp9c5fISkHGKZUrIGONO+gHbIJLB 8xNe1AuEjz7sTcVzecG7Eff5577KIv0= X-Google-Smtp-Source: ABdhPJzpoktzYaRVqGzyo2Ve24gOEXDqDLEuLMHMWFfqWRjM4y11rusy6ofYccxXt0NCzA2QGoxQjw== X-Received: by 2002:a7b:c185:: with SMTP id y5mr12769104wmi.63.1621691843629; Sat, 22 May 2021 06:57:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f14sm5409218wry.40.2021.05.22.06.57.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:23 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:57:00 +0000 Subject: [PATCH v2 21/28] fsmonitor--daemon: periodically truncate list of modified files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to periodically truncate the list of modified files to save some memory. Clients will ask for the set of changes relative to a token that they found in the FSMN index extension in the index. (This token is like a point in time, but different). Clients will then update the index to contain the response token (so that subsequent commands will be relative to this new token). Therefore, the daemon can gradually truncate the in-memory list of changed paths as they become obsolete (older than the previous token). Since we may have multiple clients making concurrent requests with a skew of tokens and clients may be racing to the talk to the daemon, we lazily truncate the list. We introduce a 5 minute delay and truncate batches 5 minutes after they are considered obsolete. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 80 +++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 33b4f09c72ca..e807aa8f6741 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -298,6 +298,77 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, batch_src->interned_paths[k]; } +/* + * To keep the batch list from growing unbounded in response to filesystem + * activity, we try to truncate old batches from the end of the list as + * they become irrelevant. + * + * We assume that the .git/index will be updated with the most recent token + * any time the index is updated. And future commands will only ask for + * recent changes *since* that new token. So as tokens advance into the + * future, older batch items will never be requested/needed. So we can + * truncate them without loss of functionality. + * + * However, multiple commands may be talking to the daemon concurrently + * or perform a slow command, so a little "token skew" is possible. + * Therefore, we want this to be a little bit lazy and have a generous + * delay. + * + * The current reader thread walked backwards in time from `token->batch_head` + * back to `batch_marker` somewhere in the middle of the batch list. + * + * Let's walk backwards in time from that marker an arbitrary delay + * and truncate the list there. Note that these timestamps are completely + * artificial (based on when we pinned the batch item) and not on any + * filesystem activity. + */ +#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */ + +static void with_lock__truncate_old_batches( + struct fsmonitor_daemon_state *state, + const struct fsmonitor_batch *batch_marker) +{ + /* assert current thread holding state->main_lock */ + + const struct fsmonitor_batch *batch; + struct fsmonitor_batch *rest; + struct fsmonitor_batch *p; + + if (!batch_marker) + return; + + trace_printf_key(&trace_fsmonitor, "Truncate: mark (%"PRIu64",%"PRIu64")", + batch_marker->batch_seq_nr, + (uint64_t)batch_marker->pinned_time); + + for (batch = batch_marker; batch; batch = batch->next) { + time_t t; + + if (!batch->pinned_time) /* an overflow batch */ + continue; + + t = batch->pinned_time + MY_TIME_DELAY_SECONDS; + if (t > batch_marker->pinned_time) /* too close to marker */ + continue; + + goto truncate_past_here; + } + + return; + +truncate_past_here: + state->current_token_data->batch_tail = (struct fsmonitor_batch *)batch; + + rest = ((struct fsmonitor_batch *)batch)->next; + ((struct fsmonitor_batch *)batch)->next = NULL; + + for (p = rest; p; p = fsmonitor_batch__pop(p)) { + trace_printf_key(&trace_fsmonitor, + "Truncate: kill (%"PRIu64",%"PRIu64")", + p->batch_seq_nr, (uint64_t)p->pinned_time); + } +} + static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) { struct fsmonitor_batch *p; @@ -643,6 +714,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, * that work. */ fsmonitor_free_token_data(token_data); + } else if (batch) { + /* + * This batch is the first item in the list + * that is older than the requested sequence + * number and might be considered to be + * obsolete. See if we can truncate the list + * and save some memory. + */ + with_lock__truncate_old_batches(state, batch); } } From patchwork Sat May 22 13:57:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04C38C2B9FB for ; Sat, 22 May 2021 13:57:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA7A161164 for ; Sat, 22 May 2021 13:57:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231372AbhEVN7R (ORCPT ); Sat, 22 May 2021 09:59:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231304AbhEVN6w (ORCPT ); Sat, 22 May 2021 09:58:52 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5236C06138E for ; Sat, 22 May 2021 06:57:25 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id f20-20020a05600c4e94b0290181f6edda88so2004200wmq.2 for ; Sat, 22 May 2021 06:57:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NKyTYcyaVKCOZRQ2A0UdZY/nTIrMMGzVkqVeo5wXvFU=; b=VQS4OG2l1nDavaJkEz2DhJzvK3HDYVceuCTWpMATp0qwqdC4loKFc3Qg6g/+pvqQtS b/sNDmC/8ZFEKri9az8/roX3jpTWu+ZIGzcqJKA29BQklwVdJ3BENHFAa25fLbX5TL7o ZFEEutwxWEa4zUfLSK9Wn3LkAxAhtbs2bwUE+Xy8mKbVIeAzRu2+fxv0epLNvA9TTC99 ji4fOQtg4L5EA9o51o3Ix2IxDY2vwN1UD1hbChUs7IS0DCuBt5fw3DGJmqxiYPW2J+N/ 8YiYYyxJdmlURrZ2PtX6x5oZftbOuo+ylZSYL9IJY4AVP5NUB2c1qBL1dHHt6DYfgpZ8 4AHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NKyTYcyaVKCOZRQ2A0UdZY/nTIrMMGzVkqVeo5wXvFU=; b=jmltM9ITRruKMW4m025X+YF3IXEJhSdaZ/Yp2vHIIACb/kBcqIUdovgtKxE7FpYkok K8/22bhTtNQKE0EYXVMYVOjG85FDqsKxHTQEPTMfwDZZUlAa7KdVgnde08MT+4L2KDpS 9jW4YfVC8Wbo5rjsgkORt5rc3eD8HyYxf9GSpEKyZ+eCqaemZ5WkgVgArg/G5QOx1+TX h8WaHQ2Rx1UQS/MY4mLeKlVuvMKaIBsJNM0RsHq2+cLHGRv1GLU+vuJyDNoQ1hqaYjJw IAbrV/QcDiq4hnN+qi4EEuynWlC0f3GV5zTgwz9wq2wDbrrdD6G9tTKCnmpMyYuLj4Cb /alw== X-Gm-Message-State: AOAM530ouBO2zJeMdLrXeG/yLyIX2d8SzhBqWGcJPaBywvyJHl85XS6m lk7o6XyumNby4KP1G97MYQdHKhnrQN4= X-Google-Smtp-Source: ABdhPJwJTo93Z21IGsGZCjsoPYB+mGAN6ySt8cUfvVp9cWc28cPBrfFI/HJ/2c6QfY28f+b80lBxMg== X-Received: by 2002:a1c:3945:: with SMTP id g66mr13001198wma.70.1621691844416; Sat, 22 May 2021 06:57:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g10sm5670602wrq.12.2021.05.22.06.57.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:24 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:57:01 +0000 Subject: [PATCH v2 22/28] fsmonitor--daemon: use a cookie file to sync with file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon client threads to create a cookie file inside the .git directory and then wait until FS events for the cookie are observed by the FS listener thread. This helps address the racy nature of file system events by blocking the client response until the kernel has drained any event backlog. This is especially important on MacOS where kernel events are only issued with a limited frequency. See the `latency` argument of `FSeventStreamCreate()`. The kernel only signals every `latency` seconds, but does not guarantee that the kernel queue is completely drained, so we may have to wait more than one interval. If we increase the frequency, the system is more likely to drop events. We avoid these issues by having each client thread create a unique cookie file and then wait until it is seen in the event stream. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 210 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 5 + 2 files changed, 214 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index e807aa8f6741..985a82cf39e0 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -92,6 +92,149 @@ static int do_as_client__status(void) } } +enum fsmonitor_cookie_item_result { + FCIR_ERROR = -1, /* could not create cookie file ? */ + FCIR_INIT = 0, + FCIR_SEEN, + FCIR_ABORT, +}; + +struct fsmonitor_cookie_item { + struct hashmap_entry entry; + const char *name; + enum fsmonitor_cookie_item_result result; +}; + +static int cookies_cmp(const void *data, const struct hashmap_entry *he1, + const struct hashmap_entry *he2, const void *keydata) +{ + const struct fsmonitor_cookie_item *a = + container_of(he1, const struct fsmonitor_cookie_item, entry); + const struct fsmonitor_cookie_item *b = + container_of(he2, const struct fsmonitor_cookie_item, entry); + + return strcmp(a->name, keydata ? keydata : b->name); +} + +static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie( + struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + int fd; + struct fsmonitor_cookie_item *cookie; + struct strbuf cookie_pathname = STRBUF_INIT; + struct strbuf cookie_filename = STRBUF_INIT; + enum fsmonitor_cookie_item_result result; + int my_cookie_seq; + + CALLOC_ARRAY(cookie, 1); + + my_cookie_seq = state->cookie_seq++; + + strbuf_addf(&cookie_filename, "%i-%i", getpid(), my_cookie_seq); + + strbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix); + strbuf_addbuf(&cookie_pathname, &cookie_filename); + + cookie->name = strbuf_detach(&cookie_filename, NULL); + cookie->result = FCIR_INIT; + hashmap_entry_init(&cookie->entry, strhash(cookie->name)); + + hashmap_add(&state->cookies, &cookie->entry); + + trace_printf_key(&trace_fsmonitor, "cookie-wait: '%s' '%s'", + cookie->name, cookie_pathname.buf); + + /* + * Create the cookie file on disk and then wait for a notification + * that the listener thread has seen it. + */ + fd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600); + if (fd >= 0) { + close(fd); + unlink(cookie_pathname.buf); + + /* + * NEEDSWORK: This is an infinite wait (well, unless another + * thread sends us an abort). I'd like to change this to + * use `pthread_cond_timedwait()` and return an error/timeout + * and let the caller do the trivial response thing. + */ + while (cookie->result == FCIR_INIT) + pthread_cond_wait(&state->cookies_cond, + &state->main_lock); + } else { + error_errno(_("could not create fsmonitor cookie '%s'"), + cookie->name); + + cookie->result = FCIR_ERROR; + } + + hashmap_remove(&state->cookies, &cookie->entry, NULL); + + result = cookie->result; + + free((char*)cookie->name); + free(cookie); + strbuf_release(&cookie_pathname); + + return result; +} + +/* + * Mark these cookies as _SEEN and wake up the corresponding client threads. + */ +static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state, + const struct string_list *cookie_names) +{ + /* assert current thread holding state->main_lock */ + + int k; + int nr_seen = 0; + + for (k = 0; k < cookie_names->nr; k++) { + struct fsmonitor_cookie_item key; + struct fsmonitor_cookie_item *cookie; + + key.name = cookie_names->items[k].string; + hashmap_entry_init(&key.entry, strhash(key.name)); + + cookie = hashmap_get_entry(&state->cookies, &key, entry, NULL); + if (cookie) { + trace_printf_key(&trace_fsmonitor, "cookie-seen: '%s'", + cookie->name); + cookie->result = FCIR_SEEN; + nr_seen++; + } + } + + if (nr_seen) + pthread_cond_broadcast(&state->cookies_cond); +} + +/* + * Set _ABORT on all pending cookies and wake up all client threads. + */ +static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + struct hashmap_iter iter; + struct fsmonitor_cookie_item *cookie; + int nr_aborted = 0; + + hashmap_for_each_entry(&state->cookies, &iter, cookie, entry) { + trace_printf_key(&trace_fsmonitor, "cookie-abort: '%s'", + cookie->name); + cookie->result = FCIR_ABORT; + nr_aborted++; + } + + if (nr_aborted) + pthread_cond_broadcast(&state->cookies_cond); +} + /* * Requests to and from a FSMonitor Protocol V2 provider use an opaque * "token" as a virtual timestamp. Clients can request a summary of all @@ -395,6 +538,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) * We should create a new token and start fresh (as if we just * booted up). * + * [2] Some of those lost events may have been for cookie files. We + * should assume the worst and abort them rather letting them starve. + * * If there are no concurrent threads readering the current token data * series, we can free it now. Otherwise, let the last reader free * it. @@ -416,6 +562,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) state->current_token_data = new_one; fsmonitor_free_token_data(free_me); + + with_lock__abort_all_cookies(state); } void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) @@ -490,6 +638,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, int hash_ret; int do_trivial = 0; int do_flush = 0; + int do_cookie = 0; + enum fsmonitor_cookie_item_result cookie_result; /* * We expect `command` to be of the form: @@ -522,6 +672,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, */ do_flush = 1; do_trivial = 1; + do_cookie = 1; } else if (!skip_prefix(command, "builtin:", &p)) { /* assume V1 timestamp or garbage */ @@ -535,6 +686,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, "fsmonitor: unsupported V1 protocol '%s'"), command); do_trivial = 1; + do_cookie = 1; } else { /* We have "builtin:*" */ @@ -544,12 +696,14 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, "fsmonitor: invalid V2 protocol token '%s'", command); do_trivial = 1; + do_cookie = 1; } else { /* * We have a V2 valid token: * "builtin::" */ + do_cookie = 1; } } @@ -558,6 +712,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, if (!state->current_token_data) BUG("fsmonitor state does not have a current token"); + /* + * Write a cookie file inside the directory being watched in + * an effort to flush out existing filesystem events that we + * actually care about. Suspend this client thread until we + * see the filesystem events for this cookie file. + * + * Creating the cookie lets us guarantee that our FS listener + * thread has drained the kernel queue and we are caught up + * with the kernel. + * + * If we cannot create the cookie (or otherwise guarantee that + * we are caught up), we send a trivial response. We have to + * assume that there might be some very, very recent activity + * on the FS still in flight. + */ + if (do_cookie) { + cookie_result = with_lock__wait_for_cookie(state); + if (cookie_result != FCIR_SEEN) { + error(_("fsmonitor: cookie_result '%d' != SEEN"), + cookie_result); + do_trivial = 1; + } + } + if (do_flush) with_lock__do_force_resync(state); @@ -769,7 +947,9 @@ static int handle_client(void *data, return result; } -#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" +#define FSMONITOR_DIR "fsmonitor--daemon" +#define FSMONITOR_COOKIE_DIR "cookies" +#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR "/" FSMONITOR_COOKIE_DIR "/") enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( const char *rel) @@ -922,6 +1102,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, } } + if (cookie_names->nr) + with_lock__mark_cookies_seen(state, cookie_names); + pthread_mutex_unlock(&state->main_lock); } @@ -1011,7 +1194,9 @@ static int fsmonitor_run_daemon(void) memset(&state, 0, sizeof(state)); + hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); + pthread_cond_init(&state.cookies_cond, NULL); state.error_code = 0; state.current_token_data = fsmonitor_new_token_data(); @@ -1035,6 +1220,23 @@ static int fsmonitor_run_daemon(void) state.nr_paths_watching = 2; } + /* + * We will write filesystem syncing cookie files into + * ///-. + */ + strbuf_init(&state.path_cookie_prefix, 0); + strbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch); + + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR); + mkdir(state.path_cookie_prefix.buf, 0777); + + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR); + mkdir(state.path_cookie_prefix.buf, 0777); + + strbuf_addch(&state.path_cookie_prefix, '/'); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1047,6 +1249,7 @@ static int fsmonitor_run_daemon(void) err = fsmonitor_run_daemon_1(&state); done: + pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsmonitor_fs_listen__dtor(&state); @@ -1054,6 +1257,11 @@ static int fsmonitor_run_daemon(void) strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); + strbuf_release(&state.path_cookie_prefix); + + /* + * NEEDSWORK: Consider "rm -rf /" + */ return err; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 89a9ef20b24b..e9fc099bae9c 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -45,6 +45,11 @@ struct fsmonitor_daemon_state { struct fsmonitor_token_data *current_token_data; + struct strbuf path_cookie_prefix; + pthread_cond_t cookies_cond; + int cookie_seq; + struct hashmap cookies; + int error_code; struct fsmonitor_daemon_backend_data *backend_data; From patchwork Sat May 22 13:57:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C216C2B9F2 for ; Sat, 22 May 2021 13:57:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E5C91610A6 for ; Sat, 22 May 2021 13:57:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231445AbhEVN7Q (ORCPT ); Sat, 22 May 2021 09:59:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231309AbhEVN6w (ORCPT ); Sat, 22 May 2021 09:58:52 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5395AC06138F for ; Sat, 22 May 2021 06:57:26 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id d11so23701783wrw.8 for ; Sat, 22 May 2021 06:57:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GEJvB6DkpLBq+oWPB2TvhNaTQQmMiQL+xy7JCCvEvbs=; b=MpDddXo/shbtTsjsnfLeKD8Tve1iGhhOrCDv8Fpp5coXEkaTzWtLuppW7YF4GVRNpd h+r835JPnHLvNCe2eu2Oku+gupqQo30dVkFJcsoYSN70PsgCLsGTg4+CAZHsOKqm3RaV CVO72jR4JetjkS4ZpjGXNLGabPDnYaLdbhyyfN6eRcnaWDbYqHQIj1wFCyt9/cL1UeyD u9buFPY1l4umJft0+H0LoJvKOY/9EiyF25sLwkeLRBRA1Pr68u3jhoX0QZVm07gRO6wQ et77N0A/Nje5t3A7IxL1BJ1UwZxHwPOnh+pjhjrXHdql5dWFI6ftX/nuJRJ06ORp6QV+ n+fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GEJvB6DkpLBq+oWPB2TvhNaTQQmMiQL+xy7JCCvEvbs=; b=UUEij8UFLayer4LIF+i5baMZO9ugWTFvC6t+hLtgJ3I7Q5qIW1ma7/7bwa2sTBHvq9 Cn7TVUVDDg9yX7jFF3/7YBfiriM+YgGEvIPAhnc8dShrUTZCQVOiKdNL3hTmzgmqTGZX 6U6iXFyxXKLVMM0igIppng7w9ujF2lF6Jl2WPl6PNelo4gwPU+6OXb4aLueYd5CEKbMx G4xlL/1fCrO812iYQ72qu+usOlawFQVRr8uVbI9iZz6xQA8zPRzTJPpKAOUy01roZk44 KbiUMtsFajniDCH9n/0AyDSa6wXTZPCmAkAhTqmzSSNDLgtF884g8xTdxm3o5z1xl4Bp I+1Q== X-Gm-Message-State: AOAM533gALtI1nWQMYSamitTUVJSPis0PzmfVSJ+mXr74JDd1+J4ARJ+ XXK/0kjt9Lv23SFofzYfrbC2zLhvuAo= X-Google-Smtp-Source: ABdhPJxqUc85HixHqNrC0KWyyO/CabpCaP0A/LlOsOu3amA37jCfAApNFsrcl08swk1zR/EGICWhLA== X-Received: by 2002:a5d:4c91:: with SMTP id z17mr14497860wrs.349.1621691844983; Sat, 22 May 2021 06:57:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k6sm2576137wmi.42.2021.05.22.06.57.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:24 -0700 (PDT) Message-Id: <102e17cbc87533ab7e95d32c1c258404b73252e5.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:57:02 +0000 Subject: [PATCH v2 23/28] fsmonitor: enhance existing comments Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- fsmonitor.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index c6d3c34ad78e..29adf3e53ef3 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -344,9 +344,25 @@ void refresh_fsmonitor(struct index_state *istate) } apply_results: - /* a fsmonitor process can return '/' to indicate all entries are invalid */ + /* + * The response from FSMonitor (excluding the header token) is + * either: + * + * [a] a (possibly empty) list of NUL delimited relative + * pathnames of changed paths. This list can contain + * files and directories. Directories have a trailing + * slash. + * + * [b] a single '/' to indicate the provider had no + * information and that we should consider everything + * invalid. We call this a trivial response. + */ if (query_success && query_result.buf[bol] != '/') { - /* Mark all entries returned by the monitor as dirty */ + /* + * Mark all pathnames returned by the monitor as dirty. + * + * This updates both the cache-entries and the untracked-cache. + */ buf = query_result.buf; for (i = bol; i < query_result.len; i++) { if (buf[i] != '\0') @@ -361,11 +377,15 @@ void refresh_fsmonitor(struct index_state *istate) if (istate->untracked) istate->untracked->use_fsmonitor = 1; } else { - - /* We only want to run the post index changed hook if we've actually changed entries, so keep track - * if we actually changed entries or not */ + /* + * We received a trivial response, so invalidate everything. + * + * We only want to run the post index changed hook if + * we've actually changed entries, so keep track if we + * actually changed entries or not. + */ int is_cache_changed = 0; - /* Mark all entries invalid */ + for (i = 0; i < istate->cache_nr; i++) { if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) { is_cache_changed = 1; @@ -373,7 +393,10 @@ void refresh_fsmonitor(struct index_state *istate) } } - /* If we're going to check every file, ensure we save the results */ + /* + * If we're going to check every file, ensure we save + * the results. + */ if (is_cache_changed) istate->cache_changed |= FSMONITOR_CHANGED; From patchwork Sat May 22 13:57:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98A91C2B9FB for ; Sat, 22 May 2021 13:57:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7FCA361164 for ; Sat, 22 May 2021 13:57:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231342AbhEVN7G (ORCPT ); Sat, 22 May 2021 09:59:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231296AbhEVN6v (ORCPT ); Sat, 22 May 2021 09:58:51 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0389C0613ED for ; Sat, 22 May 2021 06:57:26 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id d11so23701797wrw.8 for ; Sat, 22 May 2021 06:57:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=nljdf+Z23/6i2jJYLWQb++EzMtltI8k+RIHGIumkbeE=; b=XhKI1fUcgPtJzUk9unto01nw0Hvx6TDZi+47oKl0W9/uwOXv4c9pyfqhR0UM9awyHm Iwv2amXWchxoYlIGGMNIScCsxgzK7q4d42MjieHApE8UegxmAkodYdemGfMOdyxwSaFc nKF1S+dvBEKZ2FGCNqMyYgTvgq4SrR76KsLE1VZM/wTQ6lc+RRWBkknv0ER+CoiDyMlw 7RoYHtC7yUSbW4Qj28MxUFXZaTx7FWNha9OKKNofLjdVSKVFoM52hxgXYDIrfJDKG6V6 fkbFSYCHwFwN1xGLoZ4iC9MgKFe/jnzrz6SDHAplzHbznSl0GY0CgrHmKRIWGAe2U7Hm /+zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=nljdf+Z23/6i2jJYLWQb++EzMtltI8k+RIHGIumkbeE=; b=RE2WVyimNjwMnubF+J4J2CU7x3SBP9F+2Qy7sE1Y6bXkGJgtqxYHkTR2gWLln2mn9g K4h4caCYJL+bNiM+k8hOZmZFBTJXEy6S5KEXoa3i0kmiptguBzMx81Oiu9VT0GQd1DLg 2KrYJ6Ld1vF+Cnr819vNHDQg8y4jRV8bkIGIT2pqFvN/z97z258f1H9pQpUyY3OSMMCh HWKEd2hCDvXA+0UoGa5+V0mIKWn1f2b+l+9l2KigXpKnjymh3KWfS4p3bj9zywP92Cxk UyxGF6pyTMv4o8P8c7iiHE+hdXvYmc1Ta8AVEER3Il4KF6h1UEp3Co903uHT/Qmneznp VxUw== X-Gm-Message-State: AOAM532Lc/t5henm3WkT5LzEufkNjwIDI7HM0MFcjQNwVQscq8OpUY9u 5L2sKpC0eFVi+Z/1XmYfm3BLvOQdfl8= X-Google-Smtp-Source: ABdhPJwYMpRRs4s8GaHArQY75rSWkQkhrAIcA/QrdBh3SF6pBVSPJ/o9gZCZ4lBzr64nKCKqo6jSSQ== X-Received: by 2002:a5d:64e4:: with SMTP id g4mr14231726wri.366.1621691845530; Sat, 22 May 2021 06:57:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v10sm6671149wrq.0.2021.05.22.06.57.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:25 -0700 (PDT) Message-Id: <11ea2f97def686ca7f73e6c1a3eabba341bea145.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:57:03 +0000 Subject: [PATCH v2 24/28] fsmonitor: force update index after large responses Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Set the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when FSMonitor returns a very large repsonse to ensure that the index is written to disk. Normally, when the FSMonitor response includes a tracked file, the index is always updated. Similarly, the index might be updated when the response alters the untracked-cache (when enabled). However, in cases where neither of those cause the index to be considered changed, the FSMonitor response is wasted. Subsequent Git commands will make requests with the same token and receive the same response. If that response is very large, performance may suffer. It would be more efficient to force update the index now (and the token in the index extension) in order to reduce the size of the response received by future commands. This was observed on Windows after a large checkout. On Windows, the kernel emits events for the files that are changed as they are changed. However, it might delay events for the containing directories until the system is more idle (or someone scans the directory (so it seems)). The first status following a checkout would get the list of files. The subsequent status commands would get the list of directories as the events trickled out. But they would never catch up because the token was not advanced because the index wasn't updated. This list of directories caused `wt_status_collect_untracked()` to unnecessarily spend time actually scanning them during each command. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/fsmonitor.c b/fsmonitor.c index 29adf3e53ef3..22623fd228fc 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -230,6 +230,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) untracked_cache_invalidate_path(istate, name, 0); } +/* + * The number of pathnames that we need to receive from FSMonitor + * before we force the index to be updated. + * + * Note that any pathname within the set of received paths MAY cause + * cache-entry or istate flag bits to be updated and thus cause the + * index to be updated on disk. + * + * However, the response may contain many paths (such as ignored + * paths) that will not update any flag bits. And thus not force the + * index to be updated. (This is fine and normal.) It also means + * that the token will not be updated in the FSMonitor index + * extension. So the next Git command will find the same token in the + * index, make the same token-relative request, and receive the same + * response (plus any newly changed paths). If this response is large + * (and continues to grow), performance could be impacted. + * + * For example, if the user runs a build and it writes 100K object + * files but doesn't modify any source files, the index would not need + * to be updated. The FSMonitor response (after the build and + * relative to a pre-build token) might be 5MB. Each subsequent Git + * command will receive that same 100K/5MB response until something + * causes the index to be updated. And `refresh_fsmonitor()` will + * have to iterate over those 100K paths each time. + * + * Performance could be improved if we optionally force update the + * index after a very large response and get an updated token into + * the FSMonitor index extension. This should allow subsequent + * commands to get smaller and more current responses. + * + * The value chosen here does not need to be precise. The index + * will be updated automatically the first time the user touches + * a tracked file and causes a command like `git status` to + * update an mtime to be updated and/or set a flag bit. + * + * NEEDSWORK: Does this need to be a config value? + */ +static int fsmonitor_force_update_threshold = 100; + void refresh_fsmonitor(struct index_state *istate) { struct repository *r = istate->repo ? istate->repo : the_repository; @@ -363,19 +402,28 @@ void refresh_fsmonitor(struct index_state *istate) * * This updates both the cache-entries and the untracked-cache. */ + int count = 0; + buf = query_result.buf; for (i = bol; i < query_result.len; i++) { if (buf[i] != '\0') continue; fsmonitor_refresh_callback(istate, buf + bol); bol = i + 1; + count++; } - if (bol < query_result.len) + if (bol < query_result.len) { fsmonitor_refresh_callback(istate, buf + bol); + count++; + } /* Now mark the untracked cache for fsmonitor usage */ if (istate->untracked) istate->untracked->use_fsmonitor = 1; + + if (count > fsmonitor_force_update_threshold) + istate->cache_changed |= FSMONITOR_CHANGED; + } else { /* * We received a trivial response, so invalidate everything. From patchwork Sat May 22 13:57:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D09D5C2B9F2 for ; Sat, 22 May 2021 13:57:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B6FB961244 for ; Sat, 22 May 2021 13:57:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231453AbhEVN7S (ORCPT ); Sat, 22 May 2021 09:59:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231332AbhEVN6x (ORCPT ); Sat, 22 May 2021 09:58:53 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C193C06138A for ; Sat, 22 May 2021 06:57:27 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id x8so23716160wrq.9 for ; Sat, 22 May 2021 06:57:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JplwVMIpZLNbxwiKWEP1U/Bop5s9vlGWwhemhWOBynM=; b=c4aDLHHXXbg1xKlUTc1eXUm34eiXJUkRZ1xDvz4MlNPPm1AokTwTOuDZm44BZJA9bH lVPtqYs18XoCpgMzXTFrYdJjkP4ply0oAl3CNjJ96rZz5kxpxiBG1PxwLeB9itDyHBjn AVEpizMq6y3BG8CZFCPOKG6Xf0kDegdED7FQ2rKh/gQu1os7L/83lT9zpi0Jlp9Ha4Gk Q7uOvcgM2HEQ+D/ZixPKXPFbwZZMrMiPFsvpCH6CfcEYeIS02Mjyy4d0NcBY25kuiu9g XJ6T/v9iYpeDkm/ZOzllSR6WSKZQ8RomuMV6iqybqdrEV1oVtbDWBR5v8meE7xAqHBce mrng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JplwVMIpZLNbxwiKWEP1U/Bop5s9vlGWwhemhWOBynM=; b=S2fzpuvaYl3GvmGqEkHg9hh4drgEPmeVqhApEiDMsLtEPhhrWLm61t8+YT9crXGxlN pcT9lqV+m+hoSMzgt/igHxUT6h9LoiE/RMuEf79S2sIRuFYmkIzS5Gi42oQ2PQSDYVvI dHWCh6nGZQDqSgT04IT3mt3Gy7TP0NO/N4d/fhdx3K3Whc0uoqTJwWl2KObzwhEAP4YN XTGDD0j6MCoTBtb2e/EmUIWCXsTH4RaXkdmxOFpvfc0nmgi1avpB9yhMDqVHRbx4zuv4 imYqQ94lHHTPGcKkUYUACccjPaxylCHltCLCTEtoNoaGV6rkx51r3HOD5XeMXyYKZYRB 3SDg== X-Gm-Message-State: AOAM533ujQXZeRID0xShsTMJMw4j2VlXnCUQMp6G3YsjyArQbcOXgMBj cycSliaP3FcmNwvwCcNRKWQ1Gb1qVDk= X-Google-Smtp-Source: ABdhPJzjxfJI7PLHeYjPSUOl5qTCgKXlSrDFv4cailwuDqum4fSrwuekX9tk7nPtDU+CBvi0VEYT/g== X-Received: by 2002:a5d:6ac2:: with SMTP id u2mr14001364wrw.272.1621691846157; Sat, 22 May 2021 06:57:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g11sm5569585wri.59.2021.05.22.06.57.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:25 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 22 May 2021 13:57:04 +0000 Subject: [PATCH v2 25/28] t7527: create test for fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 475 +++++++++++++++++++++++++++++++++++ 1 file changed, 475 insertions(+) create mode 100755 t/t7527-builtin-fsmonitor.sh diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh new file mode 100755 index 000000000000..eaed44ebad63 --- /dev/null +++ b/t/t7527-builtin-fsmonitor.sh @@ -0,0 +1,475 @@ +#!/bin/sh + +test_description='built-in file system watcher' + +. ./test-lib.sh + +git version --build-options | grep "feature:" | grep "fsmonitor--daemon" || { + skip_all="The built-in FSMonitor is not supported on this platform" + test_done +} + +kill_repo () { + r=$1 + git -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null + rm -rf $1 + return 0 +} + +start_daemon () { + case "$#" in + 1) r="-C $1";; + *) r=""; + esac + + git $r fsmonitor--daemon start || return $? + git $r fsmonitor--daemon status || return $? + + return 0 +} + +test_expect_success 'explicit daemon start and stop' ' + test_when_finished "kill_repo test_explicit" && + + git init test_explicit && + start_daemon test_explicit && + + git -C test_explicit fsmonitor--daemon stop && + test_must_fail git -C test_explicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon start' ' + test_when_finished "kill_repo test_implicit" && + + git init test_implicit && + test_must_fail git -C test_implicit fsmonitor--daemon status && + + # query will implicitly start the daemon. + # + # for test-script simplicity, we send a V1 timestamp rather than + # a V2 token. either way, the daemon response to any query contains + # a new V2 token. (the daemon may complain that we sent a V1 request, + # but this test case is only concerned with whether the daemon was + # implicitly started.) + + GIT_TRACE2_EVENT="$PWD/.git/trace" \ + test-tool -C test_implicit fsmonitor-client query --token 0 >actual && + nul_to_q actual.filtered && + grep "builtin:" actual.filtered && + + # confirm that a daemon was started in the background. + # + # since the mechanism for starting the background daemon is platform + # dependent, just confirm that the foreground command received a + # response from the daemon. + + grep :\"query/response-length\" .git/trace && + + git -C test_implicit fsmonitor--daemon status && + git -C test_implicit fsmonitor--daemon stop && + test_must_fail git -C test_implicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (delete .git)' ' + test_when_finished "kill_repo test_implicit_1" && + + git init test_implicit_1 && + + start_daemon test_implicit_1 && + + # deleting the .git directory will implicitly stop the daemon. + rm -rf test_implicit_1/.git && + + # Create an empty .git directory so that the following Git command + # will stay relative to the `-C` directory. Without this, the Git + # command will (override the requested -C argument) and crawl out + # to the containing Git source tree. This would make the test + # result dependent upon whether we were using fsmonitor on our + # development worktree. + + sleep 1 && + mkdir test_implicit_1/.git && + + test_must_fail git -C test_implicit_1 fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (rename .git)' ' + test_when_finished "kill_repo test_implicit_2" && + + git init test_implicit_2 && + + start_daemon test_implicit_2 && + + # renaming the .git directory will implicitly stop the daemon. + mv test_implicit_2/.git test_implicit_2/.xxx && + + # Create an empty .git directory so that the following Git command + # will stay relative to the `-C` directory. Without this, the Git + # command will (override the requested -C argument) and crawl out + # to the containing Git source tree. This would make the test + # result dependent upon whether we were using fsmonitor on our + # development worktree. + + sleep 1 && + mkdir test_implicit_2/.git && + + test_must_fail git -C test_implicit_2 fsmonitor--daemon status +' + +test_expect_success 'cannot start multiple daemons' ' + test_when_finished "kill_repo test_multiple" && + + git init test_multiple && + + start_daemon test_multiple && + + test_must_fail git -C test_multiple fsmonitor--daemon start 2>actual && + grep "fsmonitor--daemon is already running" actual && + + git -C test_multiple fsmonitor--daemon stop && + test_must_fail git -C test_multiple fsmonitor--daemon status +' + +test_expect_success 'setup' ' + >tracked && + >modified && + >delete && + >rename && + mkdir dir1 && + >dir1/tracked && + >dir1/modified && + >dir1/delete && + >dir1/rename && + mkdir dir2 && + >dir2/tracked && + >dir2/modified && + >dir2/delete && + >dir2/rename && + mkdir dirtorename && + >dirtorename/a && + >dirtorename/b && + + cat >.gitignore <<-\EOF && + .gitignore + expect* + actual* + EOF + + git -c core.useBuiltinFSMonitor= add . && + test_tick && + git -c core.useBuiltinFSMonitor= commit -m initial && + + git config core.useBuiltinFSMonitor true +' + +test_expect_success 'update-index implicitly starts daemon' ' + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_1" \ + git update-index --fsmonitor && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_1 +' + +test_expect_success 'status implicitly starts daemon' ' + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_2" \ + git status >actual && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_2 +' + +edit_files() { + echo 1 >modified + echo 2 >dir1/modified + echo 3 >dir2/modified + >dir1/untracked +} + +delete_files() { + rm -f delete + rm -f dir1/delete + rm -f dir2/delete +} + +create_files() { + echo 1 >new + echo 2 >dir1/new + echo 3 >dir2/new +} + +rename_files() { + mv rename renamed + mv dir1/rename dir1/renamed + mv dir2/rename dir2/renamed +} + +file_to_directory() { + rm -f delete + mkdir delete + echo 1 >delete/new +} + +directory_to_file() { + rm -rf dir1 + echo 1 >dir1 +} + +verify_status() { + git status >actual && + GIT_INDEX_FILE=.git/fresh-index git read-tree master && + GIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect && + test_cmp expect actual && + echo HELLO AFTER && + cat .git/trace && + echo HELLO AFTER +} + +# The next few test cases confirm that our fsmonitor daemon sees each type +# of OS filesystem notification that we care about. At this layer we just +# ensure we are getting the OS notifications and do not try to confirm what +# is reported by `git status`. +# +# We run a simple query after modifying the filesystem just to introduce +# a bit of a delay so that the trace logging from the daemon has time to +# get flushed to disk. +# +# We `reset` and `clean` at the bottom of each test (and before stopping the +# daemon) because these commands might implicitly restart the daemon. + +clean_up_repo_and_stop_daemon () { + git reset --hard HEAD + git clean -fd + git fsmonitor--daemon stop + rm -f .git/trace +} + +test_expect_success 'edit some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + edit_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/modified$" .git/trace && + grep "^event: dir2/modified$" .git/trace && + grep "^event: modified$" .git/trace && + grep "^event: dir1/untracked$" .git/trace +' + +test_expect_success 'create some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + create_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/new$" .git/trace && + grep "^event: dir2/new$" .git/trace && + grep "^event: new$" .git/trace +' + +test_expect_success 'delete some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + delete_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/delete$" .git/trace && + grep "^event: dir2/delete$" .git/trace && + grep "^event: delete$" .git/trace +' + +test_expect_success 'rename some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + rename_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/rename$" .git/trace && + grep "^event: dir2/rename$" .git/trace && + grep "^event: rename$" .git/trace && + grep "^event: dir1/renamed$" .git/trace && + grep "^event: dir2/renamed$" .git/trace && + grep "^event: renamed$" .git/trace +' + +test_expect_success 'rename directory' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + mv dirtorename dirrenamed && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace +' + +test_expect_success 'file changes to directory' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + file_to_directory && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: delete$" .git/trace && + grep "^event: delete/new$" .git/trace +' + +test_expect_success 'directory changes to a file' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + directory_to_file && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1$" .git/trace +' + +# The next few test cases exercise the token-resync code. When filesystem +# drops events (because of filesystem velocity or because the daemon isn't +# polling fast enough), we need to discard the cached data (relative to the +# current token) and start collecting events under a new token. +# +# the 'test-tool fsmonitor-client flush' command can be used to send a +# "flush" message to a running daemon and ask it to do a flush/resync. + +test_expect_success 'flush cached data' ' + test_when_finished "kill_repo test_flush" && + + git init test_flush && + + ( + GIT_TEST_FSMONITOR_TOKEN=true && + export GIT_TEST_FSMONITOR_TOKEN && + + GIT_TRACE_FSMONITOR="$PWD/.git/trace_daemon" && + export GIT_TRACE_FSMONITOR && + + start_daemon test_flush + ) && + + # The daemon should have an initial token with no events in _0 and + # then a few (probably platform-specific number of) events in _1. + # These should both have the same . + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_0 && + nul_to_q actual_q0 && + + touch test_flush/file_1 && + touch test_flush/file_2 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_1 && + nul_to_q actual_q1 && + + grep "file_1" actual_q1 && + + # Force a flush. This will change the , reset the , and + # flush the file data. Then create some events and ensure that the file + # again appears in the cache. It should have the new . + + test-tool -C test_flush fsmonitor-client flush >flush_0 && + nul_to_q flush_q0 && + grep "^builtin:test_00000002:0Q/Q$" flush_q0 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && + nul_to_q actual_q2 && + + grep "^builtin:test_00000002:0Q$" actual_q2 && + + touch test_flush/file_3 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_3 && + nul_to_q actual_q3 && + + grep "file_3" actual_q3 +' + +# The next few test cases create repos where the .git directory is NOT +# inside the one of the working directory. That is, where .git is a file +# that points to a directory elsewhere. This happens for submodules and +# non-primary worktrees. + +test_expect_success 'setup worktree base' ' + git init wt-base && + echo 1 >wt-base/file1 && + git -C wt-base add file1 && + git -C wt-base commit -m "c1" +' + +test_expect_success 'worktree with .git file' ' + git -C wt-base worktree add ../wt-secondary && + + ( + GIT_TRACE2_PERF="$PWD/trace2_wt_secondary" && + export GIT_TRACE2_PERF && + + GIT_TRACE_FSMONITOR="$PWD/trace_wt_secondary" && + export GIT_TRACE_FSMONITOR && + + start_daemon wt-secondary + ) && + + git -C wt-secondary fsmonitor--daemon stop && + test_must_fail git -C wt-secondary fsmonitor--daemon status +' + +test_done From patchwork Sat May 22 13:57:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8590C2B9FB for ; Sat, 22 May 2021 13:57:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A14E161261 for ; Sat, 22 May 2021 13:57:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231464AbhEVN7U (ORCPT ); Sat, 22 May 2021 09:59:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231336AbhEVN6y (ORCPT ); Sat, 22 May 2021 09:58:54 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18A6BC0613ED for ; Sat, 22 May 2021 06:57:28 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id a4so23772188wrr.2 for ; Sat, 22 May 2021 06:57:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=juZn1iwAaJab6sNAP3JmCQ0mmV46nXh55l2jz5p/6IE=; b=dnd/aFeE6pzuqfox1lapuq/RCBbOi6BnNuyC1uI0WgV94peENPhdJ0DUuCiy7wH3rx BgtVcCBhyqns4g8amzH5dxnZlHSXOXyR5inTx1cZGD9wSXMWEcxWj5KrzP7f34KPeYR2 6Mi+Bm6Vy2RrjtU38dt0oCku/hAjKs4zzo/a9Gk3OjlvSYv8lLAMDTw4ZMlxIr0IOoTb 7l/B48oKzZtMnOcHAt04RpxKbhJE+linAyLtRvIJW/Aa3V9Vc0DsChDcyl4LXf8dj5gU qsERW+cAhLIqFuXBl+sfQ2xkgPK+YPKL6dCKaP9PQuM3JGsBpvg82PtTE/Ws7915azZn XVgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=juZn1iwAaJab6sNAP3JmCQ0mmV46nXh55l2jz5p/6IE=; b=SQlqyTdJ/C12f9cSvStCZyViJX/sVP2UL6cNPCou20SDwk8fQCy8RQQqOqvs5ckDkW bHMmFA+gU3xRx1IrkpSrrArJIHdo3TvzAd2J9ThQ9mFUtrCKGwqGhMDU71KCs1vKxATa 2mvICkHMotcpxbWnpz9JLwtCZSxnJjt77MGyfqF+DYMn7G7N9UzKGyY4emVbd/Yp8WRg ORSOPG6m9CCHPfq70SlP+qFJ+H1VeVyjAMlOHb+r20FZdmW/7TcoDeo0Vuu8Y1eXe21O /8+Wh3WcFqLoa12a6oLTDCr4XY7NIjvKo+YOZ+1OC2NZYmE6N0OOtdwRSkKipq8key+M uRIg== X-Gm-Message-State: AOAM532p9Kdil6CQYnaLXN1JMUFxWESjBOASDb+E1Qxu96KxY+g9cr6v ae3phIhRqKQjrZXIv1l4m81YbYOIups= X-Google-Smtp-Source: ABdhPJyYS10QNuy0Vr2wz1rl7/3SmgSR0GyQ8KphL1SIDGidmV3Ibs/LfAIo0VbV8C8l/F6YOl0M2g== X-Received: by 2002:a5d:48ce:: with SMTP id p14mr14601289wrs.170.1621691846724; Sat, 22 May 2021 06:57:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u132sm726970wmg.31.2021.05.22.06.57.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:26 -0700 (PDT) Message-Id: <5b035c6e0d60157cae5da515a0b1ae366fc38e30.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:57:05 +0000 Subject: [PATCH v2 26/28] p7519: add fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and the "Simple IPC" interface. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 42 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 5eb5044a103c..542ae61c99d2 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -24,7 +24,8 @@ test_description="Test core.fsmonitor" # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an # absolute path to an integration. May be a space delimited list of -# absolute paths to integrations. +# absolute paths to integrations. (This hook or list of hooks does not +# include the built-in fsmonitor--daemon.) # # The big win for using fsmonitor is the elimination of the need to scan the # working directory looking for changed and untracked files. If the file @@ -135,10 +136,16 @@ test_expect_success "one time repo setup" ' setup_for_fsmonitor() { # set INTEGRATION_SCRIPT depending on the environment - if test -n "$INTEGRATION_PATH" + if test -n "$USE_FSMONITOR_DAEMON" then + git config core.useBuiltinFSMonitor true && + INTEGRATION_SCRIPT=false + elif test -n "$INTEGRATION_PATH" + then + git config core.useBuiltinFSMonitor false && INTEGRATION_SCRIPT="$INTEGRATION_PATH" else + git config core.useBuiltinFSMonitor false && # # Choose integration script based on existence of Watchman. # Fall back to an empty integration script. @@ -174,7 +181,10 @@ test_perf_w_drop_caches () { } test_fsmonitor_suite() { - if test -n "$INTEGRATION_SCRIPT"; then + if test -n "$USE_FSMONITOR_DAEMON" + then + DESC="builtin fsmonitor--daemon" + elif test -n "$INTEGRATION_SCRIPT"; then DESC="fsmonitor=$(basename $INTEGRATION_SCRIPT)" else DESC="fsmonitor=disabled" @@ -285,4 +295,30 @@ test_expect_success "setup without fsmonitor" ' test_fsmonitor_suite trace_stop +# +# Run a full set of perf tests using the built-in fsmonitor--daemon. +# It does not use the Hook API, so it has a different setup. +# Explicitly start the daemon here and before we start client commands +# so that we can later add custom tracing. +# + +test_lazy_prereq HAVE_FSMONITOR_DAEMON ' + git version --build-options | grep "feature:" | grep "fsmonitor--daemon" +' + +if test_have_prereq HAVE_FSMONITOR_DAEMON +then + USE_FSMONITOR_DAEMON=t + + trace_start fsmonitor--daemon--server + git fsmonitor--daemon start + + trace_start fsmonitor--daemon--client + test_expect_success "setup for fsmonitor--daemon" 'setup_for_fsmonitor' + test_fsmonitor_suite + + git fsmonitor--daemon stop + trace_stop +fi + test_done From patchwork Sat May 22 13:57:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42443C4707E for ; Sat, 22 May 2021 13:57:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 28C8661244 for ; Sat, 22 May 2021 13:57:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231468AbhEVN7V (ORCPT ); Sat, 22 May 2021 09:59:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231341AbhEVN6y (ORCPT ); Sat, 22 May 2021 09:58:54 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A67B0C061343 for ; Sat, 22 May 2021 06:57:28 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id n2so23816324wrm.0 for ; Sat, 22 May 2021 06:57:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wqD1wAVn4t0gvnt0JUw+dLSMFpMCzqYvNsIe2Nw5PRU=; b=MbvSFlskADutqq/lVXHWTfU1+N5rnPq5HBb0tmaPzeI58kruKFZignhJoZWoIbFVoN bolgdon25U59niXgQd6d9LzHJTwOviv1yKmtYVGyqwDW5aqmsHw5hmqSAGojHOOJBExs 0zoDOesRcU6L2PVA7ehHuNzN7t0MCVErr4E6d84X+jG/BRYIH9t9wjK68w9TivlOzpe7 0++lR0zhbFdd48JlGsQx9y4V/szni0jpvp0kUezEs4rs4tI03WB0ZivdRWNTjAE5vwBU C2pPndOzwT9z3boqkx0z3N/AO32+3KOi23yyo73yyhfMbVkenCUJxM4jJZ2hAFLsOaG7 obSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wqD1wAVn4t0gvnt0JUw+dLSMFpMCzqYvNsIe2Nw5PRU=; b=GBmgChdrVqk6gnoX+q60xs0+oizhetpAA6xbaj7uFuUTaFDjiLMjJ5vOfw7r3BCn94 O5b+ATWn2gpkXBOifTeceZ686OIZBa+TGz80DKC3jVWOhxkbhbqV6VffYCkkKWyBqaFe Ws1svIOoa1RCTqUqYnCNeDgzuGpkGTRjtVC0Sd/dYwNrqsmvT5hY8kus04tVIvWft0Mk ztXVd5e8l1zdVuVeHQIHh15qEeDP1Kqx4ua/dOSxea05wUPLzXRa6Cv0kcUccwexzIKQ WnyNNOVAc5/aToZnNdCvhYkTTAVDhVZMA/H8RR8OW6Xr0eQ4pvsHty6SOL2Dn0XmkZnL aqIw== X-Gm-Message-State: AOAM532pArFCJWMkzz7U3XdezhHyMcfZVpR3j6emFW2xWLPqRFiwnb+d B0LgayMAT4lxmi2lrR693e8YFmVDnbk= X-Google-Smtp-Source: ABdhPJw7N4/TjDt21XbAGFdNL3ajruvpOzW8hqA7KNYWYJS/Y0wMQixu4SnqK6iUWwiejFhyiTgekQ== X-Received: by 2002:adf:9d48:: with SMTP id o8mr14308714wre.183.1621691847263; Sat, 22 May 2021 06:57:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u14sm2568628wmc.41.2021.05.22.06.57.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:26 -0700 (PDT) Message-Id: <1483c68855cbdf2adfa27e1a429e1693ce3ba6ff.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:57:06 +0000 Subject: [PATCH v2 27/28] t7527: test status with untracked-cache and fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon features and a series of edits and verify that status output is identical. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 97 ++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index eaed44ebad63..106adc2a7ee0 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -153,6 +153,8 @@ test_expect_success 'setup' ' .gitignore expect* actual* + flush* + trace* EOF git -c core.useBuiltinFSMonitor= add . && @@ -472,4 +474,99 @@ test_expect_success 'worktree with .git file' ' test_must_fail git -C wt-secondary fsmonitor--daemon status ' +# TODO Repeat one of the "edit" tests on wt-secondary and confirm that +# TODO we get the same events and behavior -- that is, that fsmonitor--daemon +# TODO correctly listens to events on both the working directory and to the +# TODO referenced GITDIR. + +test_expect_success 'cleanup worktrees' ' + kill_repo wt-secondary && + kill_repo wt-base +' + +# The next few tests perform arbitrary/contrived file operations and +# confirm that status is correct. That is, that the data (or lack of +# data) from fsmonitor doesn't cause incorrect results. And doesn't +# cause incorrect results when the untracked-cache is enabled. + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop +' + +matrix_clean_up_repo () { + git reset --hard HEAD + git clean -fd +} + +matrix_try () { + uc=$1 + fsm=$2 + fn=$3 + + test_expect_success "Matrix[uc:$uc][fsm:$fsm] $fn" ' + matrix_clean_up_repo && + $fn && + if test $uc = false -a $fsm = false + then + git status --porcelain=v1 >.git/expect.$fn + else + git status --porcelain=v1 >.git/actual.$fn + test_cmp .git/expect.$fn .git/actual.$fn + fi + ' + + return $? +} + +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" +for uc_val in $uc_values +do + if test $uc_val = false + then + test_expect_success "Matrix[uc:$uc_val] disable untracked cache" ' + git config core.untrackedcache false && + git update-index --no-untracked-cache + ' + else + test_expect_success "Matrix[uc:$uc_val] enable untracked cache" ' + git config core.untrackedcache true && + git update-index --untracked-cache + ' + fi + + fsm_values="false true" + for fsm_val in $fsm_values + do + if test $fsm_val = false + then + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor" ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop 2>/dev/null + ' + else + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor" ' + git config core.useBuiltinFSMonitor true && + git fsmonitor--daemon start && + git update-index --fsmonitor + ' + fi + + matrix_try $uc_val $fsm_val edit_files + matrix_try $uc_val $fsm_val delete_files + matrix_try $uc_val $fsm_val create_files + matrix_try $uc_val $fsm_val rename_files + matrix_try $uc_val $fsm_val file_to_directory + matrix_try $uc_val $fsm_val directory_to_file + done +done + test_done From patchwork Sat May 22 13:57:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12274601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A41BC2B9F2 for ; Sat, 22 May 2021 13:57:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E388261244 for ; Sat, 22 May 2021 13:57:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231474AbhEVN7W (ORCPT ); Sat, 22 May 2021 09:59:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbhEVN6y (ORCPT ); Sat, 22 May 2021 09:58:54 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E43CC061344 for ; Sat, 22 May 2021 06:57:29 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id j14so21974180wrq.5 for ; Sat, 22 May 2021 06:57:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wQlllqGY5z+42eIeW/Ax1KavTjRNbCneblIipm+P3nI=; b=FvtNrW1t4PcMPIvuSOazSI7FUMeiF6YpjBpjq45YEcRM5+jPst+0MsGMAm0domthwn IrpSgmJQa+/rRhyrxrv8ZcabvoGH4G3HevLk0Lls6+rbNQJDsPYRhnQZXcbCpUti6VbH /O2HMBnN3WtQ6K3TAwbvnONbICa9TiSK3Gnzrij60TCQmVB6jaDB+t2HLgem06K5udTV dE1PxnDufZ0hluMAu7qsBiHrwbmK6zqn2WUDnEGuiQkOB2VNzOSbm9QacOBVXORy0pDG gVaT/lIwA7peUG95LElw+EnTnO8Ig/ppPSOIqT7RlxQeW57ovCpZ+F7vy7cB92yfBY0U xZCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wQlllqGY5z+42eIeW/Ax1KavTjRNbCneblIipm+P3nI=; b=bQ1T2C0++OiUayFRgAw/ob2a6gFSMZWJBKjeYgM8TcWhODTUKsiA+kOcqX+UgHMjnF 9M0rDLZ5M+LLhSztqy72YAAETNjkHsAOfkx8VkARHkUw9oaeFDfKRsuTvcTcemwKQKv1 g5/nFoVIJ9EP1D6YbUE7gg226DvzqMPMlaYQCLZq2AVoydFv/qCVQFhpdSkDQcsk5bgC zvYxRTYQ0TJHJpruzfFqrdEdzo672UqZdb39KK7wIagPrNIWel67RYeOM4ZsHEA1rsHC X+fs+wvNeN3069z7cbAq2Ernil8IysjWhMCEKZbM7XQrUIuTCzddYhDdV5GG/cnuEmPf b56A== X-Gm-Message-State: AOAM5313QLCfxsMGw+RmmKePrtDhhG/8kfxyaeWmNF9vpPsd7tK6oA1Z RADs1O6tPk74Ne5jE3rdbAQWhT8TF5Q= X-Google-Smtp-Source: ABdhPJyODAuuDTpypU5pQSgyOGxtWS/zva+Eq7OmOb42CsrdmMkgEDu1deijI9GFS4vjAL4soRMsgQ== X-Received: by 2002:adf:ea82:: with SMTP id s2mr14167289wrm.397.1621691847869; Sat, 22 May 2021 06:57:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h8sm5582440wrw.85.2021.05.22.06.57.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 May 2021 06:57:27 -0700 (PDT) Message-Id: <96a3eab819f455e56643158cd135f03530b66be5.1621691828.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 22 May 2021 13:57:07 +0000 Subject: [PATCH v2 28/28] t/perf: avoid copying builtin fsmonitor files into test repo Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Do not try to copy a fsmonitor--daemon socket from the current development directory into the test trash directory. When we run the perf suite without an explicit source repo set, we copy of the current $GIT_DIR into the test trash directory. Unix domain sockets cannot be copied in that manner, so the test setup fails. Additionally, omit any other fsmonitor--daemon temp files inside the $GIT_DIR directory. Signed-off-by: Jeff Hostetler --- t/perf/perf-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 601d9f67ddb0..3b97e3fc0f27 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -74,7 +74,7 @@ test_perf_copy_repo_contents () { for stuff in "$1"/* do case "$stuff" in - */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees) + */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*) ;; *) cp -R "$stuff" "$repo/.git/" || exit 1