From patchwork Thu Jul 1 14:47:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BD23C11F69 for ; Thu, 1 Jul 2021 14:47:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3464B61414 for ; Thu, 1 Jul 2021 14:47:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232518AbhGAOuU (ORCPT ); Thu, 1 Jul 2021 10:50:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232458AbhGAOuT (ORCPT ); Thu, 1 Jul 2021 10:50:19 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1701C061764 for ; Thu, 1 Jul 2021 07:47:47 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id i8so8665381wrc.0 for ; Thu, 01 Jul 2021 07:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8Ey5pVclP/nMS6WKkIL4uf44AVx+0z97PoZxSVXx7Vk=; b=VrGR0oKrpIWVSwyaXqbCt3yUaJvGy1G0+CajzCN85t3+eE5x9NlIQokSJCLOqJ26G4 YW9ADb2peo3WbUxmPkW92D19q1ShJTt4ohxl03knpusjlHaMquqwg55P6sIWLcC9/5dn 0V7niL9bQPdvMddh7doI2wKiATBQi8bz2WldQmRsFsIilzcvKWwVEYhkqVR55LSjuluE lmlBlS3oAyYR/xdeszrL0DKAKvGai0YWp+aM0XY9SRnzfhg7pjR1twE3jaBL265FVdwR jpzBMT5e3WVM2y/Vx77ul9At0+BVi3mGNwGAc5+0LEEl44i8reBRLKIU+jKDOZ4x0Y69 lfCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8Ey5pVclP/nMS6WKkIL4uf44AVx+0z97PoZxSVXx7Vk=; b=OIxpcbr7FTclUFCTshWTc52Dxl7k2ltrhTBRCqSUaQ2dis3uSDUckMJOk2Rpy3ZAw+ y7CEvvDCLZ8DHptPK5qVfAJwtP3mWhLHflf+HWv+Z/5py3DVk11QlxWCnRMdIKINdb27 GVGi52iau81JRAY4NUezDsbM3H0Zs9fYhN9Ml5Y2itnIFgaPtPoHu9qTBm6HMaJKNVPc +tgK4fEO4Vn2yElh9QDsdz4RzKpOZKqJJb5+JUaIzqeatRnI6xQBNWu8cE43H1lu2pyr j6h3pQDjDtNQBpR/JzgatdLd7Pbc2tMVp1et2d/qgfjh6Ex9Qez5sLcTsUgB09j/Zl4w GVoQ== X-Gm-Message-State: AOAM530d9ObiXE9ox3Gc2tlf+sqdADrq9Ep+Lqj45ohvuIk6OlMM1+fD vhTT1C4yel0BOHsp+wJXwG3JM2JKyp4= X-Google-Smtp-Source: ABdhPJwnXdRupQo+gq4R9+CKEDw3IRTMRRDm80OkIcDjE/YaFiiet2HyVYMW4EDFmQK+1pMBhTW+Pg== X-Received: by 2002:adf:facf:: with SMTP id a15mr20008wrs.308.1625150866416; Thu, 01 Jul 2021 07:47:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s8sm215924wmh.36.2021.07.01.07.47.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:46 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:11 +0000 Subject: [PATCH v3 01/34] simple-ipc: preparations for supporting binary messages. Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add `command_len` argument to the Simple IPC API. In my original Simple IPC API, I assumed that the request would always be a null-terminated string of text characters. The command arg was just a `const char *`. I found a caller that would like to pass a binary command to the daemon, so I want to ammend the Simple IPC API to take `const char *command, size_t command_len` and pass that to the daemon. (Really, the first arg should just be a `void *` or `const unsigned byte *` to make that clearer.) Note, the response side has always been a `struct strbuf` which includes the buffer and length, so we already support returning a binary answer. (Yes, it feels a little weird returning a binary buffer in a `strbuf`, but it works.) Signed-off-by: Jeff Hostetler --- compat/simple-ipc/ipc-unix-socket.c | 14 +++++++----- compat/simple-ipc/ipc-win32.c | 14 +++++++----- simple-ipc.h | 7 ++++-- t/helper/test-simple-ipc.c | 34 +++++++++++++++++++---------- 4 files changed, 46 insertions(+), 23 deletions(-) diff --git a/compat/simple-ipc/ipc-unix-socket.c b/compat/simple-ipc/ipc-unix-socket.c index 1927e6ef4bc..4e28857a0a1 100644 --- a/compat/simple-ipc/ipc-unix-socket.c +++ b/compat/simple-ipc/ipc-unix-socket.c @@ -168,7 +168,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection) int ipc_client_send_command_to_connection( struct ipc_client_connection *connection, - const char *message, struct strbuf *answer) + const char *message, size_t message_len, + struct strbuf *answer) { int ret = 0; @@ -176,7 +177,7 @@ int ipc_client_send_command_to_connection( trace2_region_enter("ipc-client", "send-command", NULL); - if (write_packetized_from_buf_no_flush(message, strlen(message), + if (write_packetized_from_buf_no_flush(message, message_len, connection->fd) < 0 || packet_flush_gently(connection->fd) < 0) { ret = error(_("could not send IPC command")); @@ -197,7 +198,8 @@ done: int ipc_client_send_command(const char *path, const struct ipc_client_connect_options *options, - const char *message, struct strbuf *answer) + const char *message, size_t message_len, + struct strbuf *answer) { int ret = -1; enum ipc_active_state state; @@ -208,7 +210,9 @@ int ipc_client_send_command(const char *path, if (state != IPC_STATE__LISTENING) return ret; - ret = ipc_client_send_command_to_connection(connection, message, answer); + ret = ipc_client_send_command_to_connection(connection, + message, message_len, + answer); ipc_client_close_connection(connection); @@ -503,7 +507,7 @@ static int worker_thread__do_io( if (ret >= 0) { ret = worker_thread_data->server_data->application_cb( worker_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data); packet_flush_gently(reply_data.fd); } diff --git a/compat/simple-ipc/ipc-win32.c b/compat/simple-ipc/ipc-win32.c index 8dc7bda087d..8e889d6a506 100644 --- a/compat/simple-ipc/ipc-win32.c +++ b/compat/simple-ipc/ipc-win32.c @@ -208,7 +208,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection) int ipc_client_send_command_to_connection( struct ipc_client_connection *connection, - const char *message, struct strbuf *answer) + const char *message, size_t message_len, + struct strbuf *answer) { int ret = 0; @@ -216,7 +217,7 @@ int ipc_client_send_command_to_connection( trace2_region_enter("ipc-client", "send-command", NULL); - if (write_packetized_from_buf_no_flush(message, strlen(message), + if (write_packetized_from_buf_no_flush(message, message_len, connection->fd) < 0 || packet_flush_gently(connection->fd) < 0) { ret = error(_("could not send IPC command")); @@ -239,7 +240,8 @@ done: int ipc_client_send_command(const char *path, const struct ipc_client_connect_options *options, - const char *message, struct strbuf *response) + const char *message, size_t message_len, + struct strbuf *response) { int ret = -1; enum ipc_active_state state; @@ -250,7 +252,9 @@ int ipc_client_send_command(const char *path, if (state != IPC_STATE__LISTENING) return ret; - ret = ipc_client_send_command_to_connection(connection, message, response); + ret = ipc_client_send_command_to_connection(connection, + message, message_len, + response); ipc_client_close_connection(connection); @@ -458,7 +462,7 @@ static int do_io(struct ipc_server_thread_data *server_thread_data) if (ret >= 0) { ret = server_thread_data->server_data->application_cb( server_thread_data->server_data->application_data, - buf.buf, do_io_reply_callback, &reply_data); + buf.buf, buf.len, do_io_reply_callback, &reply_data); packet_flush_gently(reply_data.fd); diff --git a/simple-ipc.h b/simple-ipc.h index 2c48a5ee004..9c7330fcda0 100644 --- a/simple-ipc.h +++ b/simple-ipc.h @@ -107,7 +107,8 @@ void ipc_client_close_connection(struct ipc_client_connection *connection); */ int ipc_client_send_command_to_connection( struct ipc_client_connection *connection, - const char *message, struct strbuf *answer); + const char *message, size_t message_len, + struct strbuf *answer); /* * Used by the client to synchronously connect and send and receive a @@ -119,7 +120,8 @@ int ipc_client_send_command_to_connection( */ int ipc_client_send_command(const char *path, const struct ipc_client_connect_options *options, - const char *message, struct strbuf *answer); + const char *message, size_t message_len, + struct strbuf *answer); /* * Simple IPC Server Side API. @@ -144,6 +146,7 @@ typedef int (ipc_server_reply_cb)(struct ipc_server_reply_data *, */ typedef int (ipc_server_application_cb)(void *application_data, const char *request, + size_t request_len, ipc_server_reply_cb *reply_cb, struct ipc_server_reply_data *reply_data); diff --git a/t/helper/test-simple-ipc.c b/t/helper/test-simple-ipc.c index 42040ef81b1..91345180750 100644 --- a/t/helper/test-simple-ipc.c +++ b/t/helper/test-simple-ipc.c @@ -112,7 +112,7 @@ static int app__slow_command(ipc_server_reply_cb *reply_cb, /* * The client sent a command followed by a (possibly very) large buffer. */ -static int app__sendbytes_command(const char *received, +static int app__sendbytes_command(const char *received, size_t received_len, ipc_server_reply_cb *reply_cb, struct ipc_server_reply_data *reply_data) { @@ -123,6 +123,13 @@ static int app__sendbytes_command(const char *received, int errs = 0; int ret; + /* + * The test is setup to send: + * "sendbytes" SP + */ + if (received_len < strlen("sendbytes ")) + BUG("received_len is short in app__sendbytes_command"); + if (skip_prefix(received, "sendbytes ", &p)) len_ballast = strlen(p); @@ -160,7 +167,7 @@ static ipc_server_application_cb test_app_cb; * by this application. */ static int test_app_cb(void *application_data, - const char *command, + const char *command, size_t command_len, ipc_server_reply_cb *reply_cb, struct ipc_server_reply_data *reply_data) { @@ -173,7 +180,7 @@ static int test_app_cb(void *application_data, if (application_data != (void*)&my_app_data) BUG("application_cb: application_data pointer wrong"); - if (!strcmp(command, "quit")) { + if (command_len == 4 && !strncmp(command, "quit", 4)) { /* * The client sent a "quit" command. This is an async * request for the server to shutdown. @@ -193,22 +200,23 @@ static int test_app_cb(void *application_data, return SIMPLE_IPC_QUIT; } - if (!strcmp(command, "ping")) { + if (command_len == 4 && !strncmp(command, "ping", 4)) { const char *answer = "pong"; return reply_cb(reply_data, answer, strlen(answer)); } - if (!strcmp(command, "big")) + if (command_len == 3 && !strncmp(command, "big", 3)) return app__big_command(reply_cb, reply_data); - if (!strcmp(command, "chunk")) + if (command_len == 5 && !strncmp(command, "chunk", 5)) return app__chunk_command(reply_cb, reply_data); - if (!strcmp(command, "slow")) + if (command_len == 4 && !strncmp(command, "slow", 4)) return app__slow_command(reply_cb, reply_data); - if (starts_with(command, "sendbytes ")) - return app__sendbytes_command(command, reply_cb, reply_data); + if (command_len >= 10 && starts_with(command, "sendbytes ")) + return app__sendbytes_command(command, command_len, + reply_cb, reply_data); return app__unhandled_command(command, reply_cb, reply_data); } @@ -488,7 +496,9 @@ static int client__send_ipc(void) options.wait_if_busy = 1; options.wait_if_not_found = 0; - if (!ipc_client_send_command(cl_args.path, &options, command, &buf)) { + if (!ipc_client_send_command(cl_args.path, &options, + command, strlen(command), + &buf)) { if (buf.len) { printf("%s\n", buf.buf); fflush(stdout); @@ -556,7 +566,9 @@ static int do_sendbytes(int bytecount, char byte, const char *path, strbuf_addstr(&buf_send, "sendbytes "); strbuf_addchars(&buf_send, byte, bytecount); - if (!ipc_client_send_command(path, options, buf_send.buf, &buf_resp)) { + if (!ipc_client_send_command(path, options, + buf_send.buf, buf_send.len, + &buf_resp)) { strbuf_rtrim(&buf_resp); printf("sent:%c%08d %s\n", byte, bytecount, buf_resp.buf); fflush(stdout); From patchwork Thu Jul 1 14:47:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20A2EC11F64 for ; Thu, 1 Jul 2021 14:47:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0282C6141C for ; Thu, 1 Jul 2021 14:47:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232506AbhGAOuT (ORCPT ); Thu, 1 Jul 2021 10:50:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229900AbhGAOuS (ORCPT ); Thu, 1 Jul 2021 10:50:18 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C7FBC061765 for ; Thu, 1 Jul 2021 07:47:48 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id a5-20020a7bc1c50000b02901e3bbe0939bso4295634wmj.0 for ; Thu, 01 Jul 2021 07:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qLUO8zfdh0xE1VlzgNb2Oce9n0gRtddKjmmxp+0Cu60=; b=MHtUXQf/Iz9bsRw8C3lvOKKbIWftuZgGkV8W9viNBmmiJ49hAXOj4uN6d4FI5MGGS9 ncxPEw3JkQz0yHQHPOVobIp2yyTItQOejRlX2g70Er9biBIf/XrPm4bMlubxJaMTUDrQ SQ9BmfrG16Qag9RNnm5yQQRwOrfTejJkWWgv0kT/4gJHRtDZz7921rRsJf4pO4ZpgLAA 36lZ3JNVwmuCniWpiDGnxu/ZJfQ48q8MXyGsKCuZoMpBpVCN6/Enbl/iPL2NKNsbq5nq 9t2QtOi34uDulIYEsHMyk1tMMT2o8aXtcoR1apJKE39L3ZONQZswknknbf4ArXtdLiVI m72Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qLUO8zfdh0xE1VlzgNb2Oce9n0gRtddKjmmxp+0Cu60=; b=S3O2SsnFNfvqDofGGHuYfXBexW3ZeVQWBRnpRwRiTim76J9vDmIfpVN/dSWYLuZgUU /aX/lT4e/UMcnn5iMIWLHiTnriI7A2NRWHY41RBwSkFIBYPJ2/Ju1GP2PIoKTzhHgQpW pwRyei5aSDkKu+VVVXgJ+SJ0cxJFCLhrIXkihnKzwnkjG1QS2A9+poVsQmD64j8ScgyS KAogACWnMafROFrwL3Tjo7JYEIpusy/xOnvOMMjQMTB4q95l5aYlMwo1O/O0QPF6NU/i ZQ0zG/VIQChNuIArF1ZSXlhyQxbYdQKQKVVHHIJzfKrJ+p6mGwwxNp2+HMCBRNTh4Esg TrlQ== X-Gm-Message-State: AOAM531zIlCgHKfdozuRKK8+n/brSjvFMJTefbhXlFifAdbqFjMyzPey bbu4EFZsI7TwNyQD+121xzbvTwiwav8= X-Google-Smtp-Source: ABdhPJx1Tv3b3iAYFlG/LhQP9griRMYWmAB7C15GGCGi2S+4TpYfjqjZPVGLqD7+7dcn6VQMSFF8OA== X-Received: by 2002:a05:600c:4b99:: with SMTP id e25mr11042394wmp.176.1625150866926; Thu, 01 Jul 2021 07:47:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f13sm121086wrt.86.2021.07.01.07.47.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:46 -0700 (PDT) Message-Id: <5db2c0390a657d5790cf24404201505c4ec3a829.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:12 +0000 Subject: [PATCH v3 02/34] fsmonitor--daemon: man page Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a manual page describing the `git fsmonitor--daemon` feature. Signed-off-by: Jeff Hostetler --- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 Documentation/git-fsmonitor--daemon.txt diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt new file mode 100644 index 00000000000..154e7684daa --- /dev/null +++ b/Documentation/git-fsmonitor--daemon.txt @@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.useBuiltinFSMonitor` is set to `true` (see +linkgit:git-config[1]) commands, such as `git status`, will ask the +daemon for changes and automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does +not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite From patchwork Thu Jul 1 14:47:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4170C11F64 for ; Thu, 1 Jul 2021 14:47:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7FC366140C for ; Thu, 1 Jul 2021 14:47:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232629AbhGAOuZ (ORCPT ); Thu, 1 Jul 2021 10:50:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232549AbhGAOuV (ORCPT ); Thu, 1 Jul 2021 10:50:21 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00681C061765 for ; Thu, 1 Jul 2021 07:47:48 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id o22so4731040wms.0 for ; Thu, 01 Jul 2021 07:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3XmhVkrJ3wLF1aw9bykaW4kn1GzQb5tVCihnsJrD+2g=; b=css+3PtrUhHJVMHlE/QGWh/kbkR/BiLazi0QvSywMn2jgSCD5xzWySuE+RkJFkYIiV mRHCDolx+RBsWV403gWpIgi+P5WU8P/ETxITvG3zvZcIxgRfgpTLAlZnS3mLo7KyEMUP q4wLwpRgTrHwUZm8+xlXAuvf/BH9WZ5teMuBCbPEJ6nUZ3BzBZcARWQTuL0uix5Yk/B9 lLpr8wyr0FWINfwhBLUx4pTuhGAPeQfjF7MWCkQ9589pBTD5sS5sqZn+exMCY/Rklrc8 2OyDzt80C6IHqWxyqawpnzH5j6aekqWB7KXmCGCE9rwTZAwZjhrdm9aMLyPDl9SrvYXz XgfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3XmhVkrJ3wLF1aw9bykaW4kn1GzQb5tVCihnsJrD+2g=; b=NcjXonCuU1rs4moMFcOV7vIPUIVar53yvQtLh4O1CR/qhqsfs3YpcDxs5egJkK3B5v Ol5X7ng+eQtqB5xq5XRcxszw/biqbbXTgcJdCezpZOeCe781DfSslHNOovYiOkgeodVr ql4JK3cxaS2Tr/3jk9xbAkLHX0F/ZpYNOBb+Ve+Ls+RbpN3WBAsPkTHvRZdIz0DD86wG 1ODuzqlfdCFbBKODK07onXm+6czYpZWPMU392vYqFZXVVex1sAMsBXSe3oydz4wLW8FL MhXfRB97eSPG1aA/b+6Ll8Z3JiRdrGXL7yA7H7bHRhqX4xXMbVpdHl8bHqDLTY+7ay2u abhQ== X-Gm-Message-State: AOAM530EUZSL9KCwkJT+SX4+59oSiveWehTrH8fvNHqdkFG8xFQv6oJQ WTe01RFr9mH/c+SHdS9CSiyKTEVeNyA= X-Google-Smtp-Source: ABdhPJz5Kduz/brY/xwzG7zaP/tnaYQsE6XeY9tNjLl0pLP+lDkGtt6XrPXxdmcUT+n7XLcf3yNQzg== X-Received: by 2002:a05:600c:3ba9:: with SMTP id n41mr132727wms.28.1625150867494; Thu, 01 Jul 2021 07:47:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w8sm126712wre.70.2021.07.01.07.47.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:47 -0700 (PDT) Message-Id: <86413bfe34722dcd5c846cf7326340473471773c.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:13 +0000 Subject: [PATCH v3 03/34] fsmonitor--daemon: update fsmonitor documentation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and pointers to `Watchman` to mention the new built-in `fsmonitor--daemon`. Signed-off-by: Jeff Hostetler --- Documentation/config/core.txt | 56 ++++++++++++++++++++++-------- Documentation/git-update-index.txt | 27 +++++++------- Documentation/githooks.txt | 3 +- 3 files changed, 59 insertions(+), 27 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..4f6e519bc02 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -62,22 +62,50 @@ core.protectNTFS:: Defaults to `true` on Windows, and `false` elsewhere. core.fsmonitor:: - If set, the value of this variable is used as a command which - will identify all files that may have changed since the - requested date/time. This information is used to speed up git by - avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. + If set, this variable contains the pathname of the "fsmonitor" + hook command. ++ +This hook command is used to identify all files that may have changed +since the requested date/time. This information is used to speed up +git by avoiding unnecessary scanning of files that have not changed. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the protocol version to be used when invoking the + "fsmonitor" hook. ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like Watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. ++ +Note: The value of this config setting is ignored if the +built-in file system monitor is enabled (see `core.useBuiltinFSMonitor`). + +core.useBuiltinFSMonitor:: + If set to true, enable the built-in file system monitor + daemon for this working directory (linkgit:git-fsmonitor--daemon[1]). ++ +Like hook-based file system monitors, the built-in file system monitor +can speed up Git commands that need to refresh the Git index +(e.g. `git status`) in a working directory with many files. The +built-in monitor eliminates the need to install and maintain an +external third-party tool. ++ +The built-in file system monitor is currently available only on a +limited set of supported platforms. Currently, this includes Windows +and MacOS. ++ +Note: if this config setting is set to `true`, the values of +`core.fsmonitor` and `core.fsmonitorHookVersion` are ignored. core.trustctime:: If false, the ctime differences between the index and the diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt index 2853f168d97..c7c31b3fcf9 100644 --- a/Documentation/git-update-index.txt +++ b/Documentation/git-update-index.txt @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files. @@ -508,17 +510,18 @@ performance by avoiding the cost of scanning the entire working directory looking for new files. If you want to enable (or disable) this feature, it is easier to use -the `core.fsmonitor` configuration variable (see -linkgit:git-config[1]) than using the `--fsmonitor` option to -`git update-index` in each repository, especially if you want to do so -across all repositories you use, because you can set the configuration -variable in your `$HOME/.gitconfig` just once and have it affect all -repositories you touch. - -When the `core.fsmonitor` configuration variable is changed, the -file system monitor is added to or removed from the index the next time -a command reads the index. When `--[no-]fsmonitor` are used, the file -system monitor is immediately added to or removed from the index. +the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable (see linkgit:git-config[1]) than using the `--fsmonitor` +option to `git update-index` in each repository, especially if you +want to do so across all repositories you use, because you can set the +configuration variable in your `$HOME/.gitconfig` just once and have +it affect all repositories you touch. + +When the `core.fsmonitor` or `core.useBuiltinFSMonitor` configuration +variable is changed, the file system monitor is added to or removed +from the index the next time a command reads the index. When +`--[no-]fsmonitor` are used, the file system monitor is immediately +added to or removed from the index. CONFIGURATION ------------- diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt index b51959ff941..b7d5e926f7b 100644 --- a/Documentation/githooks.txt +++ b/Documentation/githooks.txt @@ -593,7 +593,8 @@ fsmonitor-watchman This hook is invoked when the configuration option `core.fsmonitor` is set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2` -depending on the version of the hook to use. +depending on the version of the hook to use, unless overridden via +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]). Version 1 takes two arguments, a version (1) and the time in elapsed nanoseconds since midnight, January 1, 1970. From patchwork Thu Jul 1 14:47:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21BD5C11F64 for ; Thu, 1 Jul 2021 14:47:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 02BBA6141C for ; Thu, 1 Jul 2021 14:47:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232821AbhGAOuW (ORCPT ); Thu, 1 Jul 2021 10:50:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232609AbhGAOuV (ORCPT ); Thu, 1 Jul 2021 10:50:21 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E3D0C061764 for ; Thu, 1 Jul 2021 07:47:49 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id j1so8559785wrn.9 for ; Thu, 01 Jul 2021 07:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=q/IG4tajktxIQa8aqHJi11ZrCyXntesR5uZmvPlgdZ8=; b=KP9ABWulKLo8Q+G/LRSmLP7Sk9chgfyL4Bm3WX3U77zbfIbdOsyryyHoXMGqo6rNzb IZtbIL+X7NHieILVIKOPKYvho3BveB39x+1KZZz6IfuI6U6q+ZLGc4sQOQYu4TQV1jHx XFMHbI92J+jd7rIjra+d8NvXFQE/YOLKN6ESqpIW9IMFNRLoCVX+N4JtkcjUJkG6PESa 7huslGm8sQ6KFYis/c9oJ/HnBaR/KnTEDXKI+ORb/wEE+TtWaCuZdV7YH4ulxsTkKFD4 u7NvoDa74kde7iN48plwNQ46hhykvV0nAaTaCVW51/rEd9OgRNXhWhZHooHOfX1/1A6e o+mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=q/IG4tajktxIQa8aqHJi11ZrCyXntesR5uZmvPlgdZ8=; b=bR1Hiw1TyOChbssS5IzXh6FA6xPRR2GgyOIxlsEP9BfpnYeLJJeNoLV/5EYjrYzAOs ROnyAHaHKKbsYCzpkkSy/0rRCjv/MWPy781SRJhfucrc1Fe/G4aH1dbOLd+Cp89MB9D9 crB5sva2BnYiUxWn7cf6s//Vepe3F0cr/YfYnrV/CoKlK/2hVuR8WpMytwQWOK4kqjqO Vxr9t2BZM1vL+HYNBiTdqJtcpuLUzpkOmDb964h5rxob3dU/Dq3liM6fX1iJA8R9mLZd aRSUZU4H/Ha7nSdCZVUd7DjI94Na+XI5q2zJ2cCj6oz/SgdVHAdau6EXRtcBopb5/bnK i1/Q== X-Gm-Message-State: AOAM531ubyfo3cnDHAW6fYKkQxMVOmbsF1d9LhQz2AJvbg+OM16YqJKV lxaz1XOMqF3qOzI+utMEUxOdZdV5pao= X-Google-Smtp-Source: ABdhPJzzJ+4xN/tyloeLIw4FVqyxdkgmwHuRSlHjdX0qmbSkVPYcUaW2JBabj7Y1SepktbbaC+hpjg== X-Received: by 2002:a05:6000:1889:: with SMTP id a9mr44580371wri.141.1625150868146; Thu, 01 Jul 2021 07:47:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n7sm9081442wmq.37.2021.07.01.07.47.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:47 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:14 +0000 Subject: [PATCH v3 04/34] fsmonitor-ipc: create client routines for git-fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create fsmonitor_ipc__*() client routines to spawn the built-in file system monitor daemon and send it an IPC request using the `Simple IPC` API. Stub in empty fsmonitor_ipc__*() functions for unsupported platforms. Signed-off-by: Jeff Hostetler --- Makefile | 1 + fsmonitor-ipc.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++ fsmonitor-ipc.h | 48 +++++++++++++ 3 files changed, 228 insertions(+) create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h diff --git a/Makefile b/Makefile index c3565fc0f8f..209c97aa22d 100644 --- a/Makefile +++ b/Makefile @@ -893,6 +893,7 @@ LIB_OBJS += fetch-pack.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o +LIB_OBJS += fsmonitor-ipc.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c new file mode 100644 index 00000000000..41706972520 --- /dev/null +++ b/fsmonitor-ipc.c @@ -0,0 +1,179 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "simple-ipc.h" +#include "fsmonitor-ipc.h" +#include "run-command.h" +#include "strbuf.h" +#include "trace2.h" + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int fsmonitor_ipc__is_supported(void) +{ + return 1; +} + +GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc") + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return ipc_get_active_state(fsmonitor_ipc__get_path()); +} + +static int spawn_daemon(void) +{ + const char *args[] = { "fsmonitor--daemon", "start", NULL }; + + return run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD, + "fsmonitor"); +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + int ret = -1; + int tried_to_spawn = 0; + enum ipc_active_state state = IPC_STATE__OTHER_ERROR; + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + const char *tok = since_token ? since_token : ""; + size_t tok_len = since_token ? strlen(since_token) : 0; + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + trace2_region_enter("fsm_client", "query", NULL); + trace2_data_string("fsm_client", NULL, "query/command", tok); + +try_again: + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + + switch (state) { + case IPC_STATE__LISTENING: + ret = ipc_client_send_command_to_connection( + connection, tok, tok_len, answer); + ipc_client_close_connection(connection); + + trace2_data_intmax("fsm_client", NULL, + "query/response-length", answer->len); + + if (fsmonitor_is_trivial_response(answer)) + trace2_data_intmax("fsm_client", NULL, + "query/trivial-response", 1); + + goto done; + + case IPC_STATE__NOT_LISTENING: + ret = error(_("fsmonitor_ipc__send_query: daemon not available")); + goto done; + + case IPC_STATE__PATH_NOT_FOUND: + if (tried_to_spawn) + goto done; + + tried_to_spawn++; + if (spawn_daemon()) + goto done; + + /* + * Try again, but this time give the daemon a chance to + * actually create the pipe/socket. + * + * Granted, the daemon just started so it can't possibly have + * any FS cached yet, so we'll always get a trivial answer. + * BUT the answer should include a new token that can serve + * as the basis for subsequent requests. + */ + options.wait_if_not_found = 1; + goto try_again; + + case IPC_STATE__INVALID_PATH: + ret = error(_("fsmonitor_ipc__send_query: invalid path '%s'"), + fsmonitor_ipc__get_path()); + goto done; + + case IPC_STATE__OTHER_ERROR: + default: + ret = error(_("fsmonitor_ipc__send_query: unspecified error on '%s'"), + fsmonitor_ipc__get_path()); + goto done; + } + +done: + trace2_region_leave("fsm_client", "query", NULL); + + return ret; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + int ret; + enum ipc_active_state state; + const char *c = command ? command : ""; + size_t c_len = command ? strlen(command) : 0; + + strbuf_reset(answer); + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + if (state != IPC_STATE__LISTENING) { + die("fsmonitor--daemon is not running"); + return -1; + } + + ret = ipc_client_send_command_to_connection(connection, c, c_len, + answer); + ipc_client_close_connection(connection); + + if (ret == -1) { + die("could not send '%s' command to fsmonitor--daemon", c); + return -1; + } + + return 0; +} + +#else + +/* + * A trivial implementation of the fsmonitor_ipc__ API for unsupported + * platforms. + */ + +int fsmonitor_ipc__is_supported(void) +{ + return 0; +} + +const char *fsmonitor_ipc__get_path(void) +{ + return NULL; +} + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return IPC_STATE__OTHER_ERROR; +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + return -1; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + return -1; +} + +#endif diff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h new file mode 100644 index 00000000000..837c5e5b64a --- /dev/null +++ b/fsmonitor-ipc.h @@ -0,0 +1,48 @@ +#ifndef FSMONITOR_IPC_H +#define FSMONITOR_IPC_H + +/* + * Returns true if built-in file system monitor daemon is defined + * for this platform. + */ +int fsmonitor_ipc__is_supported(void); + +/* + * Returns the pathname to the IPC named pipe or Unix domain socket + * where a `git-fsmonitor--daemon` process will listen. This is a + * per-worktree value. + * + * Returns NULL if the daemon is not supported on this platform. + */ +const char *fsmonitor_ipc__get_path(void); + +/* + * Try to determine whether there is a `git-fsmonitor--daemon` process + * listening on the IPC pipe/socket. + */ +enum ipc_active_state fsmonitor_ipc__get_state(void); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc + * and ask for the set of changed files since the given token. + * + * This DOES NOT use the hook interface. + * + * Spawn a daemon process in the background if necessary. + * + * Returns -1 on error; 0 on success. + */ +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc and + * send a command verb. If no daemon is available, we DO NOT try to + * start one. + * + * Returns -1 on error; 0 on success. + */ +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer); + +#endif /* FSMONITOR_IPC_H */ From patchwork Thu Jul 1 14:47:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4285AC11F67 for ; Thu, 1 Jul 2021 14:47:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2BB5A61403 for ; Thu, 1 Jul 2021 14:47:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232854AbhGAOuX (ORCPT ); Thu, 1 Jul 2021 10:50:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232566AbhGAOuV (ORCPT ); Thu, 1 Jul 2021 10:50:21 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BE12C061762 for ; Thu, 1 Jul 2021 07:47:50 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id l8so8543341wry.13 for ; Thu, 01 Jul 2021 07:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=E+4P/UqyriY5nC/kbmhLD9YFCbmINoM2rwj8oEowo4w=; b=rMZqLeCNUGs7Fga70+QBs6WWy6nA+5E2hdkX7pEL40kxRj5jvmGI08vJv6be4lzQ1a HL+Bb8uRF2bHxyfB3b3fGPcaTuen8Q8oI2peCqzyEjQxkDOCNKk9Ggk+VHAOULrRx33T UmUEtoEw5Jw62J+8uLgB1NbRvkQeiXMF1Q6CBM1nqifWJmXtaVKYh/EbV+QEyt3fdWpr fxWsHPD12wOFqYNpfnBtMgM+ckCm9Vt7jZ4/eKgPZ+I+BjbgSoU89GyqBhMaxWyWJAVL MWgaMz1gGqN2vMw/Tg4u62TjhyOJdi6VTIa7mNNUbZNr9CKKOiM5ZTzYJsuGdY02IGLT Y5cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=E+4P/UqyriY5nC/kbmhLD9YFCbmINoM2rwj8oEowo4w=; b=jTngJz+LHj8nHr7UrCwXVjvRrkVc+03UabowlH71XCRErEnas/9/qmB127Jd3Rn5ks 2J09UrdYfBQhISCqqMBKPkwJePJPuzVu0hQcCUjHJD7tOmUh3JQlk/aE16detsqvNTjs hm6vjYO7xQvhXcCpnc4GtDUNSwrIPkG2R2K5gKYcfMeiDXTx7uPTuIRmpfTnF4lV9O3B XL/7dcdhzxnuPMY+5JDOkIiLIqE8HYr0wowQDRaTPLAVq3klaAw7w6uiJRkSip5xE7V8 xnEN/YSUahCIEuuBTdkMBYug5cu4/l1zTEcCGxeIp9eo9u+IfwgIp1Vg3OUZn+eTZ/Z8 pZmg== X-Gm-Message-State: AOAM532b+VFFApnbPprhrREt9PN17Zdp07b5nkzi9JdrJfe/vpQpvg4N fIxYyAIPy8BwVWLmwgNqfVNVPz0+M5E= X-Google-Smtp-Source: ABdhPJzzYz9Sa9ipMnb8g42LS1n9fCmk82x34fnxj77nkK8YIqMtIfcQToHloQYF7jTRpz2mqVDnbw== X-Received: by 2002:a5d:4f10:: with SMTP id c16mr5034wru.3.1625150868730; Thu, 01 Jul 2021 07:47:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t16sm10313880wmi.2.2021.07.01.07.47.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:48 -0700 (PDT) Message-Id: <0aaca2f93908256b8d6385d321144dbd5cca813e.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:15 +0000 Subject: [PATCH v3 05/34] help: include fsmonitor--daemon feature flag in version info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add the "feature: fsmonitor--daemon" message to the output of `git version --build-options`. This allows users to know if the built-in fsmonitor feature is supported on their platform. Signed-off-by: Jeff Hostetler --- help.c | 4 ++++ t/test-lib.sh | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/help.c b/help.c index 3c3bdec2135..e22ba1d246a 100644 --- a/help.c +++ b/help.c @@ -11,6 +11,7 @@ #include "version.h" #include "refs.h" #include "parse-options.h" +#include "fsmonitor-ipc.h" struct category_description { uint32_t category; @@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options) strbuf_addf(buf, "sizeof-size_t: %d\n", (int)sizeof(size_t)); strbuf_addf(buf, "shell-path: %s\n", SHELL_PATH); /* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */ + + if (fsmonitor_ipc__is_supported()) + strbuf_addstr(buf, "feature: fsmonitor--daemon\n"); } } diff --git a/t/test-lib.sh b/t/test-lib.sh index adaf03543e8..bfb7ff6ed17 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1674,3 +1674,9 @@ test_lazy_prereq REBASE_P ' # Tests that verify the scheduler integration must set this locally # to avoid errors. GIT_TEST_MAINT_SCHEDULER="none:exit 1" + +# Does this platform support `git fsmonitor--daemon` +# +test_lazy_prereq FSMONITOR_DAEMON ' + git version --build-options | grep "feature:" | grep "fsmonitor--daemon" +' From patchwork Thu Jul 1 14:47:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38EF7C11F6A for ; Thu, 1 Jul 2021 14:47:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1AEEC6140C for ; Thu, 1 Jul 2021 14:47:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232919AbhGAOu2 (ORCPT ); Thu, 1 Jul 2021 10:50:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232832AbhGAOuW (ORCPT ); Thu, 1 Jul 2021 10:50:22 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C364DC0613DB for ; Thu, 1 Jul 2021 07:47:50 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id m41-20020a05600c3b29b02901dcd3733f24so7034997wms.1 for ; Thu, 01 Jul 2021 07:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mMIzOiO//AaaC5WUT/3D5iEauks3LZySkNj2HRQPzwo=; b=rujT/M4R3LBhSiJXyxHsrWwgqZhZxV1uUlS+bShByIKKhpppvcUEbqYQoviUYrJKWM Yw/GXwk9tGKtJqizlRgY9OXWyOKdH1+6JJ0rzSQFKNcLJQmMwpaZ5MleUAx+L+z7tt2t Nbsu9U+KnygHoQLFowTnpr0hnVsgXuzfZ8zHPCUiEQFPnJ1BffbskXg4WWiQm1bj7xHi EHJ5sS40mW5Tuv+xFr2VeHeau3e1Cd14tOCKuArXCnjd0NwMZLLZV083Ay0Ip8yzJqX0 jKIKeizrPHx267gr7DHLCjxM8Cgrs4DWKT1PXZLeOFpfdYZpNzsssaRTGIfE8XyrdVCE S2bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mMIzOiO//AaaC5WUT/3D5iEauks3LZySkNj2HRQPzwo=; b=q4HRd7zM/GLLS76t4zNLcmpJHsdst1MPbeYnpyyYn+ir16Ahqbjshnq2cPTfRHZqMB UPuifRUhLvUV7zggFarevvahtdZQgWlVdXHx5VrLl3R/n3HvoMOeejiR6zmtSdnQHaDt BruHA4btfZ4q7yHW+MRHEBNNrdDYhWVrh9NBBMNh1Rkqec/7RFa87eToaxEm1r5NDplW ccTbHS+caQnk/9qjca+Kb1UvyPAzcLKFijNAgkK3+LLJ915vzbrXvUNcBQeanVAits5d 4ehDzEfGqNPbdSuotuJLrMGL9oMYoJiTKx0YUcv+JmbLYcShzT4aSA1upYpXFBKyqM0Y fEgQ== X-Gm-Message-State: AOAM5313YPuratcAUqywyHce5dPeKDo6rF2b39PGTyWWfOab5T8wEHC/ QCQD8MJBLEzcIU2kD4YgL4xxfwJEii8= X-Google-Smtp-Source: ABdhPJxEmm3hJMA7JMfMSdMLEuCvqwx9JpcCtDGtJmIkZuh5O8dWLZzGFaI/RI6uiEZvvFzlS4/Dhw== X-Received: by 2002:a1c:b306:: with SMTP id c6mr11048183wmf.37.1625150869306; Thu, 01 Jul 2021 07:47:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y16sm132886wrw.42.2021.07.01.07.47.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:49 -0700 (PDT) Message-Id: <8b64b7cd3674163adb0588d42eccf4873b30974b.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:16 +0000 Subject: [PATCH v3 06/34] fsmonitor: config settings are repository-specific Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Move FSMonitor config settings to `struct repo_settings`, get rid of the `core_fsmonitor` global variable, and add support for the new `core.useBuiltinFSMonitor` config setting. Move config code to lookup `core.fsmonitor` into `prepare_repo_settings()` with the rest of the setup code. The `core_fsmonitor` global variable was used to store the pathname to the FSMonitor hook and it was used as a boolean to see if FSMonitor was enabled. This dual usage will lead to confusion when we add support for a builtin FSMonitor based on IPC, since the builtin FSMonitor doesn't need the hook pathname. Replace the boolean usage with an `enum fsmonitor_mode` to represent the state of FSMonitor. And only set the pathname when in HOOK mode. Also, disable FSMonitor when the repository working directory is incompatible. For example, in bare repositories, since there aren't any files to watch. Signed-off-by: Jeff Hostetler --- builtin/update-index.c | 20 +++++++++--- cache.h | 1 - config.c | 14 -------- config.h | 1 - environment.c | 1 - fsmonitor.c | 65 +++++++++++++++++++++++-------------- fsmonitor.h | 14 ++++++-- repo-settings.c | 48 +++++++++++++++++++++++++++ repository.h | 11 +++++++ t/README | 4 +-- t/t7519-status-fsmonitor.sh | 22 +++++++++++++ 11 files changed, 150 insertions(+), 51 deletions(-) diff --git a/builtin/update-index.c b/builtin/update-index.c index f1f16f2de52..0141899caa3 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1216,14 +1216,26 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) } if (fsmonitor > 0) { - if (git_config_get_fsmonitor() == 0) + if (r->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE) + return error( + _("repository is incompatible with fsmonitor")); + + if (r->settings.fsmonitor_mode == FSMONITOR_MODE_DISABLED) { + warning(_("core.useBuiltinFSMonitor is unset; " + "set it if you really want to enable the " + "builtin fsmonitor")); warning(_("core.fsmonitor is unset; " - "set it if you really want to " - "enable fsmonitor")); + "set it if you really want to enable the " + "hook-based fsmonitor")); + } add_fsmonitor(&the_index); report(_("fsmonitor enabled")); } else if (!fsmonitor) { - if (git_config_get_fsmonitor() == 1) + if (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) + warning(_("core.useBuiltinFSMonitor is set; " + "remove it if you really want to " + "disable fsmonitor")); + if (r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK) warning(_("core.fsmonitor is set; " "remove it if you really want to " "disable fsmonitor")); diff --git a/cache.h b/cache.h index ba04ff8bd36..50463876852 100644 --- a/cache.h +++ b/cache.h @@ -981,7 +981,6 @@ extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; extern int protect_ntfs; -extern const char *core_fsmonitor; extern int core_apply_sparse_checkout; extern int core_sparse_checkout_cone; diff --git a/config.c b/config.c index f9c400ad306..ec3b88b4639 100644 --- a/config.c +++ b/config.c @@ -2516,20 +2516,6 @@ int git_config_get_max_percent_split_change(void) return -1; /* default value */ } -int git_config_get_fsmonitor(void) -{ - if (git_config_get_pathname("core.fsmonitor", &core_fsmonitor)) - core_fsmonitor = getenv("GIT_TEST_FSMONITOR"); - - if (core_fsmonitor && !*core_fsmonitor) - core_fsmonitor = NULL; - - if (core_fsmonitor) - return 1; - - return 0; -} - int git_config_get_index_threads(int *dest) { int is_bool, val; diff --git a/config.h b/config.h index 9038538ffdc..b8d6b75d4fe 100644 --- a/config.h +++ b/config.h @@ -609,7 +609,6 @@ int git_config_get_index_threads(int *dest); int git_config_get_untracked_cache(void); int git_config_get_split_index(void); int git_config_get_max_percent_split_change(void); -int git_config_get_fsmonitor(void); /* This dies if the configured or default date is in the future */ int git_config_get_expiry(const char *key, const char **output); diff --git a/environment.c b/environment.c index 2f27008424a..7b5e8ff78da 100644 --- a/environment.c +++ b/environment.c @@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT; #define PROTECT_NTFS_DEFAULT 1 #endif int protect_ntfs = PROTECT_NTFS_DEFAULT; -const char *core_fsmonitor; /* * The character that begins a commented line in user-editable file diff --git a/fsmonitor.c b/fsmonitor.c index ab9bfc60b34..374189be7d9 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -3,6 +3,7 @@ #include "dir.h" #include "ewah/ewok.h" #include "fsmonitor.h" +#include "fsmonitor-ipc.h" #include "run-command.h" #include "strbuf.h" @@ -148,15 +149,21 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) /* * Call the query-fsmonitor hook passing the last update token of the saved results. */ -static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result) +static int query_fsmonitor_hook(struct repository *r, + int version, + const char *last_update, + struct strbuf *query_result) { struct child_process cp = CHILD_PROCESS_INIT; int result; - if (!core_fsmonitor) + if (r->settings.fsmonitor_mode != FSMONITOR_MODE_HOOK) return -1; - strvec_push(&cp.args, core_fsmonitor); + assert(r->settings.fsmonitor_hook_path); + assert(*r->settings.fsmonitor_hook_path); + + strvec_push(&cp.args, r->settings.fsmonitor_hook_path); strvec_pushf(&cp.args, "%d", version); strvec_pushf(&cp.args, "%s", last_update); cp.use_shell = 1; @@ -238,17 +245,27 @@ void refresh_fsmonitor(struct index_state *istate) struct strbuf last_update_token = STRBUF_INIT; char *buf; unsigned int i; + struct repository *r = istate->repo ? istate->repo : the_repository; - if (!core_fsmonitor || istate->fsmonitor_has_run_once) + if (r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED || + istate->fsmonitor_has_run_once) return; - hook_version = fsmonitor_hook_version(); - istate->fsmonitor_has_run_once = 1; trace_printf_key(&trace_fsmonitor, "refresh fsmonitor"); + + if (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) { + /* TODO */ + return; + } + + assert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK); + + hook_version = fsmonitor_hook_version(); + /* - * This could be racy so save the date/time now and query_fsmonitor + * This could be racy so save the date/time now and query_fsmonitor_hook * should be inclusive to ensure we don't miss potential changes. */ last_update = getnanotime(); @@ -256,13 +273,14 @@ void refresh_fsmonitor(struct index_state *istate) strbuf_addf(&last_update_token, "%"PRIu64"", last_update); /* - * If we have a last update token, call query_fsmonitor for the set of + * If we have a last update token, call query_fsmonitor_hook for the set of * changes since that token, else assume everything is possibly dirty * and check it all. */ if (istate->fsmonitor_last_update) { if (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) { - query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2, + query_success = !query_fsmonitor_hook( + r, HOOK_INTERFACE_VERSION2, istate->fsmonitor_last_update, &query_result); if (query_success) { @@ -292,13 +310,17 @@ void refresh_fsmonitor(struct index_state *istate) } if (hook_version == HOOK_INTERFACE_VERSION1) { - query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1, + query_success = !query_fsmonitor_hook( + r, HOOK_INTERFACE_VERSION1, istate->fsmonitor_last_update, &query_result); } - trace_performance_since(last_update, "fsmonitor process '%s'", core_fsmonitor); - trace_printf_key(&trace_fsmonitor, "fsmonitor process '%s' returned %s", - core_fsmonitor, query_success ? "success" : "failure"); + trace_performance_since(last_update, "fsmonitor process '%s'", + r->settings.fsmonitor_hook_path); + trace_printf_key(&trace_fsmonitor, + "fsmonitor process '%s' returned %s", + r->settings.fsmonitor_hook_path, + query_success ? "success" : "failure"); } /* a fsmonitor process can return '/' to indicate all entries are invalid */ @@ -411,7 +433,8 @@ void remove_fsmonitor(struct index_state *istate) void tweak_fsmonitor(struct index_state *istate) { unsigned int i; - int fsmonitor_enabled = git_config_get_fsmonitor(); + struct repository *r = istate->repo ? istate->repo : the_repository; + int fsmonitor_enabled = r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED; if (istate->fsmonitor_dirty) { if (fsmonitor_enabled) { @@ -431,16 +454,8 @@ void tweak_fsmonitor(struct index_state *istate) istate->fsmonitor_dirty = NULL; } - switch (fsmonitor_enabled) { - case -1: /* keep: do nothing */ - break; - case 0: /* false */ - remove_fsmonitor(istate); - break; - case 1: /* true */ + if (fsmonitor_enabled) add_fsmonitor(istate); - break; - default: /* unknown value: do nothing */ - break; - } + else + remove_fsmonitor(istate); } diff --git a/fsmonitor.h b/fsmonitor.h index f20d72631d7..9cc14e05239 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -57,7 +57,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result); */ static inline int is_fsmonitor_refreshed(const struct index_state *istate) { - return !core_fsmonitor || istate->fsmonitor_has_run_once; + struct repository *r = istate->repo ? istate->repo : the_repository; + + return r->settings.fsmonitor_mode <= FSMONITOR_MODE_DISABLED || + istate->fsmonitor_has_run_once; } /* @@ -67,7 +70,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate) */ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce) { - if (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) { + struct repository *r = istate->repo ? istate->repo : the_repository; + + if (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED && + !(ce->ce_flags & CE_FSMONITOR_VALID)) { istate->cache_changed = 1; ce->ce_flags |= CE_FSMONITOR_VALID; trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name); @@ -83,7 +89,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache */ static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce) { - if (core_fsmonitor) { + struct repository *r = istate->repo ? istate->repo : the_repository; + + if (r->settings.fsmonitor_mode > FSMONITOR_MODE_DISABLED) { ce->ce_flags &= ~CE_FSMONITOR_VALID; untracked_cache_invalidate_path(istate, ce->name, 1); trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_invalid '%s'", ce->name); diff --git a/repo-settings.c b/repo-settings.c index 0cfe8b787db..faf197ff60a 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -5,10 +5,42 @@ #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0) +/* + * Return 1 if the repo/workdir is incompatible with FSMonitor. + */ +static int is_repo_incompatible_with_fsmonitor(struct repository *r) +{ + const char *const_strval; + + /* + * Bare repositories don't have a working directory and + * therefore, nothing to watch. + */ + if (!r->worktree) + return 1; + + /* + * GVFS (aka VFS for Git) is incompatible with FSMonitor. + * + * Granted, core Git does not know anything about GVFS and + * we shouldn't make assumptions about a downstream feature, + * but users can install both versions. And this can lead + * to incorrect results from core Git commands. So, without + * bringing in any of the GVFS code, do a simple config test + * for a published config setting. (We do not look at the + * various *_TEST_* environment variables.) + */ + if (!repo_config_get_value(r, "core.virtualfilesystem", &const_strval)) + return 1; + + return 0; +} + void prepare_repo_settings(struct repository *r) { int value; char *strval; + const char *const_strval; if (r->settings.initialized) return; @@ -26,6 +58,22 @@ void prepare_repo_settings(struct repository *r) UPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1); UPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1); + r->settings.fsmonitor_hook_path = NULL; + r->settings.fsmonitor_mode = FSMONITOR_MODE_DISABLED; + if (is_repo_incompatible_with_fsmonitor(r)) + r->settings.fsmonitor_mode = FSMONITOR_MODE_INCOMPATIBLE; + else if (!repo_config_get_bool(r, "core.usebuiltinfsmonitor", &value) + && value) + r->settings.fsmonitor_mode = FSMONITOR_MODE_IPC; + else { + if (repo_config_get_pathname(r, "core.fsmonitor", &const_strval)) + const_strval = getenv("GIT_TEST_FSMONITOR"); + if (const_strval && *const_strval) { + r->settings.fsmonitor_hook_path = strdup(const_strval); + r->settings.fsmonitor_mode = FSMONITOR_MODE_HOOK; + } + } + if (!repo_config_get_int(r, "index.version", &value)) r->settings.index_version = value; if (!repo_config_get_maybe_bool(r, "core.untrackedcache", &value)) { diff --git a/repository.h b/repository.h index a45f7520fd9..09154298ba1 100644 --- a/repository.h +++ b/repository.h @@ -26,6 +26,14 @@ enum fetch_negotiation_setting { FETCH_NEGOTIATION_NOOP = 3, }; +enum fsmonitor_mode { + FSMONITOR_MODE_INCOMPATIBLE = -2, + FSMONITOR_MODE_UNSET = -1, + FSMONITOR_MODE_DISABLED = 0, + FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor */ + FSMONITOR_MODE_IPC = 2, /* core.useBuiltinFSMonitor */ +}; + struct repo_settings { int initialized; @@ -34,6 +42,9 @@ struct repo_settings { int gc_write_commit_graph; int fetch_write_commit_graph; + enum fsmonitor_mode fsmonitor_mode; + char *fsmonitor_hook_path; + int index_version; enum untracked_cache_setting core_untracked_cache; diff --git a/t/README b/t/README index 1a2072b2c8a..852a4eae9da 100644 --- a/t/README +++ b/t/README @@ -398,8 +398,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was passed in. GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor -code path for utilizing a file system monitor to speed up detecting -new or changed files. +code path for utilizing a (hook based) file system monitor to speed up +detecting new or changed files. GIT_TEST_INDEX_VERSION= exercises the index read/write code path for the index version specified. Can be set to any valid version diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 637391c6ce4..02919c68ddd 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -383,4 +383,26 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Test that we detect and disallow repos that are incompatible with FSMonitor. +test_expect_success 'incompatible bare repo' ' + test_when_finished "rm -rf ./bare-clone" && + git clone --bare . ./bare-clone && + cat >expect <<-\EOF && + error: repository is incompatible with fsmonitor + EOF + test_must_fail git -C ./bare-clone update-index --fsmonitor 2>actual && + test_cmp expect actual +' + +test_expect_success 'incompatible core.virtualfilesystem' ' + test_when_finished "rm -rf ./fake-gvfs-clone" && + git clone . ./fake-gvfs-clone && + git -C ./fake-gvfs-clone config core.virtualfilesystem true && + cat >expect <<-\EOF && + error: repository is incompatible with fsmonitor + EOF + test_must_fail git -C ./fake-gvfs-clone update-index --fsmonitor 2>actual && + test_cmp expect actual +' + test_done From patchwork Thu Jul 1 14:47:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DCD5C11F67 for ; Thu, 1 Jul 2021 14:47:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6AA0A6141C for ; Thu, 1 Jul 2021 14:47:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232900AbhGAOu0 (ORCPT ); Thu, 1 Jul 2021 10:50:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232625AbhGAOuW (ORCPT ); Thu, 1 Jul 2021 10:50:22 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 243DAC0613DC for ; Thu, 1 Jul 2021 07:47:51 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id j1so8559894wrn.9 for ; Thu, 01 Jul 2021 07:47:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uUlEGZcJ8HWSol8QoU0XdH0dmL3aUVI9LugPaHK4haA=; b=mmVy1P9TNwy7CzeAmkSzq552r+pAYCYPDIsCu3bwW3vat1h+gdj18kcRhqm9f4MJpq 4GlW29cRWxDm8lP4qbueatobkExBOZCw/ayu/aWZHxsyBwEGeBPi9JmOXwn7Cku0jwON bF4R2MrM+HP0VbAyheUUzlMy3v8G/Oy8rPBB82yojSkKu6JcdDH1ITDNDgy/xpoHQ2ZN AjUfpZMH/R+2Z8C+M0VnKv5ChfSlmSHO/d78T67obMsXgPNs5BIExLLIqSERqQsBXk4Q LdQxbBwBbgHBvzOddokE54htBWzndaqlWVWkFflrkjF1jCRuv7cE9sCawzzaXk9nnKvK McsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uUlEGZcJ8HWSol8QoU0XdH0dmL3aUVI9LugPaHK4haA=; b=SAfoTnfE91a3Vr5aOQIsCTVBk9+zeP27lB1rK02TWjwd1eLnwhJ3PvOj5guZmIQJN+ RQHKz47OybL4txqqrkMx6IYHNNEE2grZTYLfiz4IZ3srb65l29FdnBvjTkK+EEj5jM8w QT34w36aTkhzl9VXSxUWkVAxj6I+uMSFvMdefq/bwjxAAHrp2+snucWHw0uMg2gNnLYX PXd/OH8itXA1FKRcjKCua1Wgse1GtITGiuET0XKQ5ylTr0v/GI5yyyt3jiZHia2LwFBP yVz/TiN9YyKwRWXxFw5PuoRDZ7xrzOi8/G48sernVtYRSjh9vUWWUOVSsCRyoNMw8bFB 2/8A== X-Gm-Message-State: AOAM532Yt0JsLJ1fiDalXEVe6IulhDZYicv0/4j8SXfmLpqrOBExlcTz ZIsqAp5Adl4FY0m2qY6B1pZtnywCV4w= X-Google-Smtp-Source: ABdhPJyF6CcmVmbQ66z9QQKR9qzlNXxxFT2v/lBaATvAgPYOYF20jd/STW6iGvjxmRg7+JcqXSVt+A== X-Received: by 2002:a05:6000:152:: with SMTP id r18mr46723512wrx.203.1625150869835; Thu, 01 Jul 2021 07:47:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p5sm175619wrd.25.2021.07.01.07.47.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:49 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:17 +0000 Subject: [PATCH v3 07/34] fsmonitor: use IPC to query the builtin FSMonitor daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Use simple IPC to directly communicate with the new builtin file system monitor daemon when `core.useBuiltinFSMonitor` is set. The `core.fsmonitor` setting has already been defined as a HOOK pathname. Historically, this has been set to a HOOK script that will talk with Watchman. For compatibility reasons, we do not want to overload that definition (and cause problems if users have multiple versions of Git installed). Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- fsmonitor.c | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 374189be7d9..3719ddfeec9 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -256,8 +256,44 @@ void refresh_fsmonitor(struct index_state *istate) trace_printf_key(&trace_fsmonitor, "refresh fsmonitor"); if (r->settings.fsmonitor_mode == FSMONITOR_MODE_IPC) { - /* TODO */ - return; + query_success = !fsmonitor_ipc__send_query( + istate->fsmonitor_last_update ? + istate->fsmonitor_last_update : "builtin:fake", + &query_result); + if (query_success) { + /* + * The response contains a series of nul terminated + * strings. The first is the new token. + * + * Use `char *buf` as an interlude to trick the CI + * static analysis to let us use `strbuf_addstr()` + * here (and only copy the token) rather than + * `strbuf_addbuf()`. + */ + buf = query_result.buf; + strbuf_addstr(&last_update_token, buf); + bol = last_update_token.len + 1; + } else { + /* + * The builtin daemon is not available on this + * platform -OR- we failed to get a response. + * + * Generate a fake token (rather than a V1 + * timestamp) for the index extension. (If + * they switch back to the hook API, we don't + * want ambiguous state.) + */ + strbuf_addstr(&last_update_token, "builtin:fake"); + } + + /* + * Regardless of whether we successfully talked to a + * fsmonitor daemon or not, we skip over and do not + * try to use the hook. The "core.useBuiltinFSMonitor" + * config setting ALWAYS overrides the "core.fsmonitor" + * hook setting. + */ + goto apply_results; } assert(r->settings.fsmonitor_mode == FSMONITOR_MODE_HOOK); @@ -323,6 +359,7 @@ void refresh_fsmonitor(struct index_state *istate) query_success ? "success" : "failure"); } +apply_results: /* a fsmonitor process can return '/' to indicate all entries are invalid */ if (query_success && query_result.buf[bol] != '/') { /* Mark all entries returned by the monitor as dirty */ From patchwork Thu Jul 1 14:47:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A668C11F69 for ; Thu, 1 Jul 2021 14:47:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4114561403 for ; Thu, 1 Jul 2021 14:47:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232884AbhGAOu1 (ORCPT ); Thu, 1 Jul 2021 10:50:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232817AbhGAOuW (ORCPT ); Thu, 1 Jul 2021 10:50:22 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0F92C061762 for ; Thu, 1 Jul 2021 07:47:51 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id i94so8602446wri.4 for ; Thu, 01 Jul 2021 07:47:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZoO136qbmKSNeu7dMlW5FdUmzzXdWr4uk4/pcU4Tq1I=; b=AQsOQ0Y3vIZJkdPKtaGBZgxTm6CY9aqCEFD45pO2IXAyl/PB2z7MtA0sOJ6N4VNe6S 7Leg1srYVaOMAElMIVYGnii/4l4FCU/5qqK4WZOlg/yMx+Ozi4s037AxM3p2LTTYPjHE JO3htjkmDrrsg79RhTNswI+fFGbO5Jh7bo+/jKOaeecTIsBgUMZZMnQSsf5dFuTvOA26 DVpIcUUJqG5ZtrClDS0khFmebq/4jdVBHL0kohlUu82stD3+Xf+U5mALCetadTgOEpSB dzGQpkcsA9BZOP+UEDQGmif8awbujY7WQihYI23KEpTRV3asnMqcjYXT5BIHsKzVZ1fo /MtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZoO136qbmKSNeu7dMlW5FdUmzzXdWr4uk4/pcU4Tq1I=; b=ejRRyiXkPN135SjXlpRr0IumFMF9DZLS89Ey611La1g6aJbtinbHIEbnwzu2y5SJY1 aa2xOMu70jI/oiEzPbjFOVIU6jcStdUPVof06hjMeKqBVfHR/N9B+l0ow7u/TZut8KAD wxSOUuncyQS0QgL4/URtCcFsqBtUawGinIqEviCTKqHL/IRHc3nV0ZKNam+Xx92DVXub PHBQynw4xmyvQQqtFUX06WyxBAyCOUVADZ6KIMehGP5orP+iHX2yG0RYvZopSo0qr4EE ncBQu/A/o/+YoiaPlU8JRdsJDjofKg0vezWZXKf0RtxLcuUUWi0D0babju18NgpQebxX +ruw== X-Gm-Message-State: AOAM53282spMxz18CHQLAsMYp9sXKCyE1YgB6xNoKbI0ZvE8wzrOBdtK QDzQCuzvCGpjNtGdF6tctMhPpTNoxj0= X-Google-Smtp-Source: ABdhPJwro/flhiVj5aWWr+VSMXak0HctkFnpTRotqd+kwLjxzB87hA3fQR/QZDp6VasrbjA9cjX08A== X-Received: by 2002:a5d:5742:: with SMTP id q2mr22260wrw.256.1625150870428; Thu, 01 Jul 2021 07:47:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d3sm156462wrx.28.2021.07.01.07.47.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:50 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:18 +0000 Subject: [PATCH v3 08/34] fsmonitor--daemon: add a built-in fsmonitor daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a built-in file system monitoring daemon that can be used by the existing `fsmonitor` feature (protocol API and index extension) to improve the performance of various Git commands, such as `status`. The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and provides an alternative to hook access to existing fsmonitors such as `watchman`. This commit merely adds the new command without any functionality. Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- .gitignore | 1 + Makefile | 1 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 53 +++++++++++++++++++++++++++++++++++++ git.c | 1 + 5 files changed, 57 insertions(+) create mode 100644 builtin/fsmonitor--daemon.c diff --git a/.gitignore b/.gitignore index 311841f9bed..4baba472aa8 100644 --- a/.gitignore +++ b/.gitignore @@ -72,6 +72,7 @@ /git-format-patch /git-fsck /git-fsck-objects +/git-fsmonitor--daemon /git-gc /git-get-tar-commit-id /git-grep diff --git a/Makefile b/Makefile index 209c97aa22d..8fe1e42a435 100644 --- a/Makefile +++ b/Makefile @@ -1097,6 +1097,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o +BUILTIN_OBJS += builtin/fsmonitor--daemon.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o BUILTIN_OBJS += builtin/grep.o diff --git a/builtin.h b/builtin.h index 16ecd5586f0..2470d1cd3a2 100644 --- a/builtin.h +++ b/builtin.h @@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix); int cmd_for_each_repo(int argc, const char **argv, const char *prefix); int cmd_format_patch(int argc, const char **argv, const char *prefix); int cmd_fsck(int argc, const char **argv, const char *prefix); +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix); int cmd_gc(int argc, const char **argv, const char *prefix); int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix); int cmd_grep(int argc, const char **argv, const char *prefix); diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c new file mode 100644 index 00000000000..df2bad53111 --- /dev/null +++ b/builtin/fsmonitor--daemon.c @@ -0,0 +1,53 @@ +#include "builtin.h" +#include "config.h" +#include "parse-options.h" +#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +#include "simple-ipc.h" +#include "khash.h" + +static const char * const builtin_fsmonitor__daemon_usage[] = { + NULL +}; + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + const char *subcmd; + + struct option options[] = { + OPT_END() + }; + + if (argc < 2) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + git_config(git_default_config, NULL); + + subcmd = argv[1]; + argv--; + argc++; + + argc = parse_options(argc, argv, prefix, options, + builtin_fsmonitor__daemon_usage, 0); + + die(_("Unhandled subcommand '%s'"), subcmd); +} + +#else +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + struct option options[] = { + OPT_END() + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + die(_("fsmonitor--daemon not supported on this platform")); +} +#endif diff --git a/git.c b/git.c index 18bed9a9964..c6160f4a886 100644 --- a/git.c +++ b/git.c @@ -533,6 +533,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY }, From patchwork Thu Jul 1 14:47:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F27BBC11F64 for ; Thu, 1 Jul 2021 14:47:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DFA5D6140C for ; Thu, 1 Jul 2021 14:47:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232920AbhGAOu3 (ORCPT ); Thu, 1 Jul 2021 10:50:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232566AbhGAOuX (ORCPT ); Thu, 1 Jul 2021 10:50:23 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BADEC061764 for ; Thu, 1 Jul 2021 07:47:52 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id m9-20020a05600c3b09b02901f246b43bbeso4264230wms.3 for ; Thu, 01 Jul 2021 07:47:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=W24eIgcVkPMjgPidGJzbRp5I7QCe7bRzj3qpbqX2Z20=; b=MMwlg8Pd5wGLdz5Oh1LGutY6Dod6tPegRL197inFCRZSZFFOK2pgQe67H0D2WG9d7x OdmoYwX/XchQBytprnHL4YCqiarJgxTWcIJv56XIN4jYGLwAKl/xAeZVkvrZNl0WOfqH hGwD7vRD8FRU72fbROwvh/EChCK2mgZKBklUsDJ0AWBfeN8EZbAecjQzqL9FPRFgWPco Adn8JxXkay2jK+EtdZGwFS3MlpgbBuHlzy116L06TrIR9E5LGCXliNuJS89LEcujvfeG 8Is66q5bhL5Qiv0hbI6qx++QWh3j1OBKf/eb+aZV0aaP5jkDQoVdjRaTz58k6RPGqpe5 bkgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=W24eIgcVkPMjgPidGJzbRp5I7QCe7bRzj3qpbqX2Z20=; b=dW7G0igjJ4hR3l0WP6DfWBvyt4y2Q+C2pzfFD5qM+exQSeATzLqiwv2Lr/ugapoIqa ZDPe5v5q759fYJUNKFXXpi9SiinmVkarhkfFV9O3BDlzXNp13Cu/USKRPQwuXQmOhGHi MCAihbfUp8OWaD6YhV4BpUgi/Un96K1EwBrIML3PqUAraXpoPjYRXpFYevngiuoC1UFn 5u4flOrPmvR0BsiL4+p8KgK8NjoZZAYu4qkTdqrGuSFOwiUhhBWNoK72yVF33sCvFwbd JTjjn2+iji/1cVpndm0MWEwW7Mz4YZSBr+PJkxInyhZ+CdxkUQgNdMF1tRGn9ZSsignL euFw== X-Gm-Message-State: AOAM5319IgdCqlyIcgOTlTAzHe7iVQSqeI1dj6OAJKgiMRNVfeWrNR7d YaQW/47sHt9rwkdkJRbTnkZsyQKY4fQ= X-Google-Smtp-Source: ABdhPJxa0L7SGet3oKE92LYjXJ9ao1KJlCKx1kss3Gs+IHkeQ0P+b7V9xPfzpkc59g6m5mWBx2YI1A== X-Received: by 2002:a1c:59c3:: with SMTP id n186mr11263534wmb.48.1625150870978; Thu, 01 Jul 2021 07:47:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z5sm159029wrt.11.2021.07.01.07.47.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:50 -0700 (PDT) Message-Id: <02e21384ef0ca4909e0bda2c78fa63c06be22a50.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:19 +0000 Subject: [PATCH v3 09/34] fsmonitor--daemon: implement 'stop' and 'status' commands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement `stop` and `status` client commands to control and query the status of a `fsmonitor--daemon` server process (and implicitly start a server process if necessary). Later commits will implement the actual server and monitor the file system. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index df2bad53111..62efd5ea787 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,10 +7,55 @@ #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon stop"), + N_("git fsmonitor--daemon status"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Acting as a CLIENT. + * + * Send a "quit" command to the `git-fsmonitor--daemon` (if running) + * and wait for it to shutdown. + */ +static int do_as_client__send_stop(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("quit", &answer); + + /* The quit command does not return any response data. */ + strbuf_release(&answer); + + if (ret) + return ret; + + trace2_region_enter("fsm_client", "polling-for-daemon-exit", NULL); + while (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + sleep_millisec(50); + trace2_region_leave("fsm_client", "polling-for-daemon-exit", NULL); + + return 0; +} + +static int do_as_client__status(void) +{ + enum ipc_active_state state = fsmonitor_ipc__get_state(); + + switch (state) { + case IPC_STATE__LISTENING: + printf(_("fsmonitor-daemon is watching '%s'\n"), + the_repository->worktree); + return 0; + + default: + printf(_("fsmonitor-daemon is not watching '%s'\n"), + the_repository->worktree); + return 1; + } +} int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { @@ -35,6 +80,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc = parse_options(argc, argv, prefix, options, builtin_fsmonitor__daemon_usage, 0); + if (!strcmp(subcmd, "stop")) + return !!do_as_client__send_stop(); + + if (!strcmp(subcmd, "status")) + return !!do_as_client__status(); + die(_("Unhandled subcommand '%s'"), subcmd); } From patchwork Thu Jul 1 14:47:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5FDEC11F6A for ; Thu, 1 Jul 2021 14:48:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C9D946140C for ; Thu, 1 Jul 2021 14:48:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232855AbhGAOuf (ORCPT ); Thu, 1 Jul 2021 10:50:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232866AbhGAOuY (ORCPT ); Thu, 1 Jul 2021 10:50:24 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E730BC061765 for ; Thu, 1 Jul 2021 07:47:52 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id f14so8218901wrs.6 for ; Thu, 01 Jul 2021 07:47:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=FPBKBRBBPCafHEVXnDZiFc4V2cPGyq12y/pnJ2IeXvY=; b=dVUYpjVt7yPovvdZ50QO9vtrewWbkpn3GZTPbnoFq3xivs8e3oA20weO+ltkh02hcL mEo6uGO7retVFWyVjd1naYZ1irmDI5wpRjroKCabc9b92STYkLTUC2/GP4SS2qggJOI+ qatMm9ULdoqkxdQinjH7MotnRQm5ahzSsFRyyLhPuzYRxZkKBtIeFRfee4bnMd2XHTYD FV68qXT1P24kJtzz1faQSZUDjN+OGtDGSoqOAkou3ZWj5k7NTi2BvMVqrvfjDKMe02ij daCsjREA8OkE0AzQmEUMA4ywAhk4/iWB9FpGqGp318VrT+F9snvVOdHb20QDVAWyCzMU o8Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=FPBKBRBBPCafHEVXnDZiFc4V2cPGyq12y/pnJ2IeXvY=; b=eaHt4Yf0hE2fgjeNTWerQvykz+qhWuvtr46E99nk07OkekMRutdG1R5IpCoXRFBZ+P AvQykhEQVTMxgTzRQXhN/bkQ+Z5Rl1VKoGLPBs7Tjg+XrmKmkTgPzlyMvPKlv7AAzdZo 4++OI6qOk1kGCyz7+VphwjPwa8MANbB7luYLSnTO/oBlrFsvUtCm7SY1CbRqQzosSs2V hNky39tY+M3/OnpshEGjOHGEb/BPxrUYY+UAYfYkZ+HKkgpmBq4nlfRZYDuCZoLhFBUX EnJra86vnw2KzMeosEdmbdH3vS0ZdnjfEE017LNjpkTfaz8OM3+xTsT1XBOW0rIUsXWm aoiw== X-Gm-Message-State: AOAM533KlgINCEyQURgQwTblEC683kqZ8P9g9AnK9jWc2mZyUjdk21A2 QdLQ1TwUxtZUtmwPEsy9YQC5T4U5rII= X-Google-Smtp-Source: ABdhPJwpM/xG1mImS8itmI5s7cOWePi4XgZPprIgF6Uz+mlBChur+F80Cmk0IsygBlEOsnepCYKvJA== X-Received: by 2002:adf:fc47:: with SMTP id e7mr21404wrs.284.1625150871578; Thu, 01 Jul 2021 07:47:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z3sm10069442wmi.29.2021.07.01.07.47.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:51 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:20 +0000 Subject: [PATCH v3 10/34] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create an IPC client to send query and flush commands to the daemon. Signed-off-by: Jeff Hostetler --- Makefile | 1 + t/helper/test-fsmonitor-client.c | 121 +++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 124 insertions(+) create mode 100644 t/helper/test-fsmonitor-client.c diff --git a/Makefile b/Makefile index 8fe1e42a435..c45caacf2c3 100644 --- a/Makefile +++ b/Makefile @@ -709,6 +709,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o TEST_BUILTINS_OBJS += test-hash-speed.o diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c new file mode 100644 index 00000000000..f7a5b3a32fa --- /dev/null +++ b/t/helper/test-fsmonitor-client.c @@ -0,0 +1,121 @@ +/* + * test-fsmonitor-client.c: client code to send commands/requests to + * a `git fsmonitor--daemon` daemon. + */ + +#include "test-tool.h" +#include "cache.h" +#include "parse-options.h" +#include "fsmonitor-ipc.h" + +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND +int cmd__fsmonitor_client(int argc, const char **argv) +{ + die("fsmonitor--daemon not available on this platform"); +} +#else + +/* + * Read the `.git/index` to get the last token written to the + * FSMonitor Index Extension. + */ +static const char *get_token_from_index(void) +{ + struct index_state *istate = the_repository->index; + + if (do_read_index(istate, the_repository->index_file, 0) < 0) + die("unable to read index file"); + if (!istate->fsmonitor_last_update) + die("index file does not have fsmonitor extension"); + + return istate->fsmonitor_last_update; +} + +/* + * Send an IPC query to a `git-fsmonitor--daemon` daemon and + * ask for the changes since the given token or from the last + * token in the index extension. + * + * This will implicitly start a daemon process if necessary. The + * daemon process will persist after we exit. + */ +static int do_send_query(const char *token) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + if (!token || !*token) + token = get_token_from_index(); + + ret = fsmonitor_ipc__send_query(token, &answer); + if (ret < 0) + die(_("could not query fsmonitor--daemon")); + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +/* + * Send a "flush" command to the `git-fsmonitor--daemon` (if running) + * and tell it to flush its cache. + * + * This feature is primarily used by the test suite to simulate a loss of + * sync with the filesystem where we miss kernel events. + */ +static int do_send_flush(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("flush", &answer); + if (ret) + return ret; + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +int cmd__fsmonitor_client(int argc, const char **argv) +{ + const char *subcmd; + const char *token = NULL; + + const char * const fsmonitor_client_usage[] = { + N_("test-helper fsmonitor-client query []"), + N_("test-helper fsmonitor-client flush"), + NULL, + }; + + struct option options[] = { + OPT_STRING(0, "token", &token, N_("token"), + N_("command token to send to the server")), + OPT_END() + }; + + if (argc < 2) + usage_with_options(fsmonitor_client_usage, options); + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(fsmonitor_client_usage, options); + + subcmd = argv[1]; + argv--; + argc++; + + argc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0); + + setup_git_directory(); + + if (!strcmp(subcmd, "query")) + return !!do_send_query(token); + + if (!strcmp(subcmd, "flush")) + return !!do_send_flush(); + + die("Unhandled subcommand: '%s'", subcmd); +} +#endif diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index c5bd0c6d4c7..af879e4a5d7 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -31,6 +31,7 @@ static struct test_cmd cmds[] = { { "dump-untracked-cache", cmd__dump_untracked_cache }, { "example-decorate", cmd__example_decorate }, { "fast-rebase", cmd__fast_rebase }, + { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, { "hashmap", cmd__hashmap }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index e8069a3b222..6c5134b46d9 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -21,6 +21,7 @@ int cmd__dump_split_index(int argc, const char **argv); int cmd__dump_untracked_cache(int argc, const char **argv); int cmd__example_decorate(int argc, const char **argv); int cmd__fast_rebase(int argc, const char **argv); +int cmd__fsmonitor_client(int argc, const char **argv); int cmd__genrandom(int argc, const char **argv); int cmd__genzeros(int argc, const char **argv); int cmd__hashmap(int argc, const char **argv); From patchwork Thu Jul 1 14:47:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04262C11F67 for ; Thu, 1 Jul 2021 14:48:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DDAFF61414 for ; Thu, 1 Jul 2021 14:48:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232968AbhGAOua (ORCPT ); Thu, 1 Jul 2021 10:50:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232855AbhGAOuY (ORCPT ); Thu, 1 Jul 2021 10:50:24 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8ACCBC0613DC for ; Thu, 1 Jul 2021 07:47:53 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id f14so8218939wrs.6 for ; Thu, 01 Jul 2021 07:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UgqoBQFqZt7+8MK/XFDvVVK3eW1rSdtiaAl9JpmaH1w=; b=gYozgeQhcdNuD+8Hx5lt0HJHJRiFKE5fkFzQQPNmdeZrdIEXwFjzRZEB6ByEDBqRTF MfirajIpEVwXuOZ3a/JjVlQVZvuF60Ola0LK8k1VkawiFQ4ypo4iM6Ks7FYks4I4TRpX 9TF/TljeQgiAxEM35gYw7d2EGUNnDHOeDR/flhRlyz7FdntbJZrgGlOopOGN+KjVKn1a wX8JOEeXlbZbUMeKymQP2M/Kf9lJQVk7mVlOKDh4jKe+ELobMyHr+gUa/os6I/Zl6uuS OylFrW1gFBiJtA0f1sDRpFBlKyvnOge9GeF7xM60MwYibIlBrz6fviFDII5czztUmgqk BSAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UgqoBQFqZt7+8MK/XFDvVVK3eW1rSdtiaAl9JpmaH1w=; b=RRQqo5zObOigAUN3xmZeZIkQ9ksIGvbg+X92of7pnSwioBeeP4OSLebEsQrxO3SmJy 4jpruW84n621W6GmCfAQt1zKrG9IU5h/J00nQJMPSbw5rqKNXQshj01otYClx5iPIh5w rbk9iRd4MdYxlysOvnP7/8jEP5AEbmeNAASjWAuILjV5H4tiLJ74U3qBwFydV/qN5BND TF0Hsh0kardbE9E0ftljHvq979vTcZqvXhBiSCEW9y6hFBHo1bY+cywFxys9v2DkUO14 XZN6CGCYrjm/SZRV33AnrmaTKDDeZLKaP78ZbXkC3k+FpR2JelKKWDxZi7xEeG8qqHvU cuyQ== X-Gm-Message-State: AOAM530anFfvOjzhMxgvX2lFRu04pn0j60lv2lnDJ0nhYipD1QaMtEcD 9i+Qea3m+hclRzWZUYGQl4V/cxDHMiY= X-Google-Smtp-Source: ABdhPJw2BzO9JrpWX1TibQljWgCy+ZKIX4sM+ZpuMkWZ9RMqANYUnJPTfdakc3E1S4WY6fVcHIyNhg== X-Received: by 2002:a5d:5405:: with SMTP id g5mr50155wrv.12.1625150872128; Thu, 01 Jul 2021 07:47:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c1sm120094wrp.94.2021.07.01.07.47.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:51 -0700 (PDT) Message-Id: <5a9bda7220356ebf0689bb6aaa9068520dc6e33b.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:21 +0000 Subject: [PATCH v3 11/34] fsmonitor-fs-listen-win32: stub in backend for Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty backend for fsmonitor--daemon on Windows. Signed-off-by: Jeff Hostetler --- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h diff --git a/Makefile b/Makefile index c45caacf2c3..a2a6e1f20f6 100644 --- a/Makefile +++ b/Makefile @@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "" of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1929,6 +1934,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2793,6 +2803,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c new file mode 100644 index 00000000000..880446b49e3 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -0,0 +1,21 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) +{ +} + +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsmonitor-fs-listen.h b/compat/fsmonitor/fsmonitor-fs-listen.h new file mode 100644 index 00000000000..c7b5776b3b6 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen.h @@ -0,0 +1,49 @@ +#ifndef FSMONITOR_FS_LISTEN_H +#define FSMONITOR_FS_LISTEN_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread PRIOR to staring the + * fsmonitor_fs_listener thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread AFTER joining the listener. + */ +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to watch for + * filesystem events. This will run in the fsmonitor_fs_listen thread. + * + * It should call `ipc_server_stop_async()` if the listener thread + * prematurely terminates (because of a filesystem error or if it + * detects that the .git directory has been deleted). (It should NOT + * do so if the listener thread receives a normal shutdown signal from + * the IPC layer.) + * + * It should set `state->error_code` to -1 if the daemon should exit + * with an error. + */ +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the fsmonitor listener thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_FS_LISTEN_H */ diff --git a/config.mak.uname b/config.mak.uname index cb443b4e023..fcd88b60b14 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows) # so we don't need this: # # SNPRINTF_RETURNS_BOGUS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) NO_STRTOUMAX = YesPlease NO_MKDTEMP = YesPlease NO_SVN_TESTS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index a87841340e6..1ab94eb284f 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -263,6 +263,11 @@ else() endif() endif() +if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +endif() + set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) #header checks From patchwork Thu Jul 1 14:47:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07726C11F67 for ; Thu, 1 Jul 2021 14:48:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E994F6141C for ; Thu, 1 Jul 2021 14:48:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233027AbhGAOue (ORCPT ); Thu, 1 Jul 2021 10:50:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232876AbhGAOuZ (ORCPT ); Thu, 1 Jul 2021 10:50:25 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B8F3C0613DD for ; Thu, 1 Jul 2021 07:47:54 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id u8-20020a7bcb080000b02901e44e9caa2aso4264355wmj.4 for ; Thu, 01 Jul 2021 07:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pIwm/Gno09dspEBpdqAIIGCyt0/sJ9C5U74NilrVyV4=; b=ku0veBi14TP2PhJ5xq0vct6YMnKsbn5mz5RbSa9UjPZifnUO6DwVBr1hgDRaK+JaNs Y99i1udoya2PywH1jCxG9w0QqFr/C4dEwc+poEKYfiQ9pzrXrpePIV4hyvu1+XfaKvGu 8Xh0Ixzfd+Q7DHMR7cW2VwpkVgq0m0eD8g6RuJsZpGPjuC31ZOaRcL0I1hcdX2x71AYT qNQIFVDUCpvv9sByxbT72z2FBTiH7a5s+cDS5MNWEFNEUwfcBR5e3BzBpVnO29RWhKCI DOY2/1Pks86Ms80fl/6VIy4KPfHlqz3hLpOsfYGPUl6CnAeDRqFdudi4jTmC947t7Q2L BWWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pIwm/Gno09dspEBpdqAIIGCyt0/sJ9C5U74NilrVyV4=; b=Jv+IHB+HCkc3BpW/cE0sq+hqsvGU7sGeb0ntnlQ64b1q/Enavc5WFj48gSJl6HBe4D l7KC/QAiqK+pfn/OYererduHeReJ4DbmmxIq3X1DNH2zzWIRpdOmlYuxlwLnqrr+DucC AHeDvUo7IFV06Mklx7bAvQ/ZuSr7hCQCI/qUE6i3XBVKf/aI8Ukikpop8a45YDn5lIEI WybiwrWq6pgn1xdmZcx4ZxtUbQVcRYH+/bfqcc5HXVfbGwkKt5ClQmxKgzPM6Qpz/uZC B8aX7xKDQ+9RdTvj1D0sDvZUXWI/EnZZ1/GGLMm2UpTps+DncCTPlx9pRCudZtv6cnSd c5mw== X-Gm-Message-State: AOAM533/Nwkgfuvdv1hfK9C7HR+gsilcq5eZAZF2eFmFlAdAv/zn7xQo qTETlo3W4+FqlfJ9qbTVkQjdYvOU7gQ= X-Google-Smtp-Source: ABdhPJz9bg5ihLnDgmE0EMtqJ0GHAXCDT7e5YwYWr5uLlPNSm1f82f0+QaGR9M65KJvHS3KTh6eU7w== X-Received: by 2002:a7b:c0d6:: with SMTP id s22mr11149922wmh.52.1625150872730; Thu, 01 Jul 2021 07:47:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n29sm9382812wms.34.2021.07.01.07.47.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:52 -0700 (PDT) Message-Id: <587580489473a7a2ad665bdf3c482ea5d2c54f61.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:22 +0000 Subject: [PATCH v3 12/34] fsmonitor-fs-listen-macos: stub in backend for MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty implementation of fsmonitor--daemon backend for MacOS. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c new file mode 100644 index 00000000000..b91058d1c4f --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) +{ +} diff --git a/config.mak.uname b/config.mak.uname index fcd88b60b14..394355463e1 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macos + BASIC_LDFLAGS += -framework CoreServices endif ifeq ($(uname_S),SunOS) NEEDS_SOCKET = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 1ab94eb284f..aa80671045a 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -266,6 +266,9 @@ endif() if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c) endif() set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) From patchwork Thu Jul 1 14:47:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6571DC11F64 for ; Thu, 1 Jul 2021 14:48:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 38CEE6140C for ; Thu, 1 Jul 2021 14:48:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233003AbhGAOud (ORCPT ); Thu, 1 Jul 2021 10:50:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37398 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232549AbhGAOu0 (ORCPT ); Thu, 1 Jul 2021 10:50:26 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0092C0613DF for ; Thu, 1 Jul 2021 07:47:54 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id u8-20020a7bcb080000b02901e44e9caa2aso4264384wmj.4 for ; Thu, 01 Jul 2021 07:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=c4fuWBjV9jcEFHts8GQw8jpbOZtgaVEZ32qooQ+Shso=; b=E5XP3PylFfNqQjpceRBweEHE0CxwO8p0ZtZ87VzlhLVhov+HGPXz6Bq11EEHqLpDki 8fdq6HcnT9N5uuHwpWsUyb8BhrMyiD6Jyn72K62he+0sULbzjAFCGsPi641bm2XqZsRc 1jrMdpRjTmHtjbpsnWK6AYtWlmOt/WrC932PX24vw7vJG/iFv4oZ5gT8yQ2nWQuHQa3G Y0olZ791BDosic4D49PXDcnl8/X4QRenmhiO12FAtsS6JQo+NjJkL9y19kLBgKdIU8iq nFUjLv9jFZ0bbExA76Ng3FclkeONKp41YpmxsFgGPr3ZAF4qlbB4bjJYWApjEletgfUZ PLUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=c4fuWBjV9jcEFHts8GQw8jpbOZtgaVEZ32qooQ+Shso=; b=q+P/RqsZhRu3S4PUUeEkyPeXx0zv0JQfBOrVKYjntNoAFFuxhUzI5j1GO1K3mURHXo WztCkLL2LeUZHBtljmdGj36mqJJL/8NEhNJf+LdFPTVpA/8tVe64O1Oz+tXjxDsJGzyY w+e8+uYMo9s/VWnffgvazfaWeVk7+svkwl2NzYf9olIaOlhbu9MLm9FrO4p1Lcy94Pm+ 15Wr+vUQDDluQ3C7qsHz8P51kz22IfOeou8jSz8DjtR445+8Z6Xqv58u4yjhrtpCELNF eT+TyE+pyL+q0e5vrd2DKwGXGn/FyucoJ4y1dY68dpIEzDyiNRXj1cvRa8DSOJ660+Pj bX7g== X-Gm-Message-State: AOAM531SGt4jiK52oz/eB40uNtfC+QN6fW/isnxUM1FNIoUWdu05JLTe lK8FJmgJXUvEsbnxIUY9zW9fBykebUI= X-Google-Smtp-Source: ABdhPJyaRDhs+Yp6/1wIj+7cOC3AYu8pxs4XIhcPeKNys2II/8Eo85EvAkHXPul5PV3sZ9EHxtOcCQ== X-Received: by 2002:a05:600c:4f53:: with SMTP id m19mr131292wmq.36.1625150873245; Thu, 01 Jul 2021 07:47:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n18sm4783282wms.3.2021.07.01.07.47.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:52 -0700 (PDT) Message-Id: <5d6646df93a17659af66f136295444d1bd834090.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:23 +0000 Subject: [PATCH v3 13/34] fsmonitor--daemon: implement 'run' command Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement `run` command to try to begin listening for file system events. This version defines the thread structure with a single fsmonitor_fs_listen thread to watch for file system events and a simple IPC thread pool to watch for connection from Git clients over a well-known named pipe or Unix domain socket. This commit does not actually do anything yet because the platform backends are still just stubs. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 210 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 34 ++++++ 2 files changed, 243 insertions(+), 1 deletion(-) create mode 100644 fsmonitor--daemon.h diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 62efd5ea787..a265c962ccc 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,16 +3,39 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon run []"), N_("git fsmonitor--daemon stop"), N_("git fsmonitor--daemon status"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Global state loaded from config. + */ +#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" +static int fsmonitor__ipc_threads = 8; + +static int fsmonitor_config(const char *var, const char *value, void *cb) +{ + if (!strcmp(var, FSMONITOR__IPC_THREADS)) { + int i = git_config_int(var, value); + if (i < 1) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__IPC_THREADS, i); + fsmonitor__ipc_threads = i; + return 0; + } + + return git_default_config(var, value, cb); +} + /* * Acting as a CLIENT. * @@ -57,11 +80,190 @@ static int do_as_client__status(void) } } +static ipc_server_application_cb handle_client; + +static int handle_client(void *data, + const char *command, size_t command_len, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + /* struct fsmonitor_daemon_state *state = data; */ + int result; + + /* + * The Simple IPC API now supports {char*, len} arguments, but + * FSMonitor always uses proper null-terminated strings, so + * we can ignore the command_len argument. (Trust, but verify.) + */ + if (command_len != strlen(command)) + BUG("FSMonitor assumes text messages"); + + trace2_region_enter("fsmonitor", "handle_client", the_repository); + trace2_data_string("fsmonitor", the_repository, "request", command); + + result = 0; /* TODO Do something here. */ + + trace2_region_leave("fsmonitor", "handle_client", the_repository); + + return result; +} + +static void *fsmonitor_fs_listen__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-listen"); + + trace_printf_key(&trace_fsmonitor, "Watching: worktree '%s'", + state->path_worktree_watch.buf); + if (state->nr_paths_watching > 1) + trace_printf_key(&trace_fsmonitor, "Watching: gitdir '%s'", + state->path_gitdir_watch.buf); + + fsmonitor_fs_listen__loop(state); + + trace2_thread_exit(); + return NULL; +} + +static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) +{ + struct ipc_server_opts ipc_opts = { + .nr_threads = fsmonitor__ipc_threads, + + /* + * We know that there are no other active threads yet, + * so we can let the IPC layer temporarily chdir() if + * it needs to when creating the server side of the + * Unix domain socket. + */ + .uds_disallow_chdir = 0 + }; + + /* + * Start the IPC thread pool before the we've started the file + * system event listener thread so that we have the IPC handle + * before we need it. + */ + if (ipc_server_run_async(&state->ipc_server_data, + fsmonitor_ipc__get_path(), &ipc_opts, + handle_client, state)) + return error(_("could not start IPC thread pool")); + + /* + * Start the fsmonitor listener thread to collect filesystem + * events. + */ + if (pthread_create(&state->listener_thread, NULL, + fsmonitor_fs_listen__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + ipc_server_await(state->ipc_server_data); + + return error(_("could not start fsmonitor listener thread")); + } + + /* + * The daemon is now fully functional in background threads. + * Wait for the IPC thread pool to shutdown (whether by client + * request or from filesystem activity). + */ + ipc_server_await(state->ipc_server_data); + + /* + * The fsmonitor listener thread may have received a shutdown + * event from the IPC thread pool, but it doesn't hurt to tell + * it again. And wait for it to shutdown. + */ + fsmonitor_fs_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + + return state->error_code; +} + +static int fsmonitor_run_daemon(void) +{ + struct fsmonitor_daemon_state state; + int err; + + memset(&state, 0, sizeof(state)); + + pthread_mutex_init(&state.main_lock, NULL); + state.error_code = 0; + state.current_token_data = NULL; + + /* Prepare to (recursively) watch the directory. */ + strbuf_init(&state.path_worktree_watch, 0); + strbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree())); + state.nr_paths_watching = 1; + + /* + * We create and delete cookie files somewhere inside the .git + * directory to help us keep sync with the file system. If + * ".git" is not a directory, then is not inside the + * cone of , so set up a second watch to watch + * the so that we get events for the cookie files. + */ + strbuf_init(&state.path_gitdir_watch, 0); + strbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch); + strbuf_addstr(&state.path_gitdir_watch, "/.git"); + if (!is_directory(state.path_gitdir_watch.buf)) { + strbuf_reset(&state.path_gitdir_watch); + strbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir())); + state.nr_paths_watching = 2; + } + + /* + * Confirm that we can create platform-specific resources for the + * filesystem listener before we bother starting all the threads. + */ + if (fsmonitor_fs_listen__ctor(&state)) { + err = error(_("could not initialize listener thread")); + goto done; + } + + err = fsmonitor_run_daemon_1(&state); + +done: + pthread_mutex_destroy(&state.main_lock); + fsmonitor_fs_listen__dtor(&state); + + ipc_server_free(state.ipc_server_data); + + strbuf_release(&state.path_worktree_watch); + strbuf_release(&state.path_gitdir_watch); + + return err; +} + +static int try_to_run_foreground_daemon(void) +{ + /* + * Technically, we don't need to probe for an existing daemon + * process, since we could just call `fsmonitor_run_daemon()` + * and let it fail if the pipe/socket is busy. + * + * However, this method gives us a nicer error message for a + * common error case. + */ + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + die("fsmonitor--daemon is already running '%s'", + the_repository->worktree); + + printf(_("running fsmonitor-daemon in '%s'\n"), + the_repository->worktree); + fflush(stdout); + + return !!fsmonitor_run_daemon(); +} + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { const char *subcmd; struct option options[] = { + OPT_INTEGER(0, "ipc-threads", + &fsmonitor__ipc_threads, + N_("use ipc worker threads")), OPT_END() }; @@ -71,7 +273,7 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_fsmonitor__daemon_usage, options); - git_config(git_default_config, NULL); + git_config(fsmonitor_config, NULL); subcmd = argv[1]; argv--; @@ -79,6 +281,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) argc = parse_options(argc, argv, prefix, options, builtin_fsmonitor__daemon_usage, 0); + if (fsmonitor__ipc_threads < 1) + die(_("invalid 'ipc-threads' value (%d)"), + fsmonitor__ipc_threads); + + if (!strcmp(subcmd, "run")) + return !!try_to_run_foreground_daemon(); if (!strcmp(subcmd, "stop")) return !!do_as_client__send_stop(); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h new file mode 100644 index 00000000000..3009c1a83de --- /dev/null +++ b/fsmonitor--daemon.h @@ -0,0 +1,34 @@ +#ifndef FSMONITOR_DAEMON_H +#define FSMONITOR_DAEMON_H + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +#include "cache.h" +#include "dir.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "thread-utils.h" + +struct fsmonitor_batch; +struct fsmonitor_token_data; + +struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ + +struct fsmonitor_daemon_state { + pthread_t listener_thread; + pthread_mutex_t main_lock; + + struct strbuf path_worktree_watch; + struct strbuf path_gitdir_watch; + int nr_paths_watching; + + struct fsmonitor_token_data *current_token_data; + + int error_code; + struct fsmonitor_daemon_backend_data *backend_data; + + struct ipc_server_data *ipc_server_data; +}; + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_DAEMON_H */ From patchwork Thu Jul 1 14:47:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2D2AC11F6C for ; Thu, 1 Jul 2021 14:48:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8A3C16140C for ; Thu, 1 Jul 2021 14:48:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233016AbhGAOug (ORCPT ); Thu, 1 Jul 2021 10:50:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232907AbhGAOu1 (ORCPT ); Thu, 1 Jul 2021 10:50:27 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 423C6C0613E1 for ; Thu, 1 Jul 2021 07:47:55 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id i8so8665864wrc.0 for ; Thu, 01 Jul 2021 07:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UfZUXel1p6AOKNdy1ef02UiRKIjcEL1X8er7dQKR42M=; b=W0YGGiO21Mdxh1rXRTHfDs9JPCDmO559ehggERJWYfYZDVLyzjHayur2yRA4nLW4EE I9qDvD7+YZHME7Mv8rP1OsvhfAqxU5okvNDe440+qBlNRbJsN95PSQx9tmwGqFk289MS 5MI3PXxbFRpNXoHWYwa4ydvmGMQeaC/+u9iUh6ajvYDhuuf46eicqvtul4zQ8GO4TalH g+t3K5EUIX2AeMakpr6vN4S/qH+/gR9hOSdPlm5h893cMWzgG+zAH2V/h5iOlpSrzWhJ muC4L57Rso+fLjDsxohkj9Scjq6X1Iita40VQhXSTDGSxmy+sT/71aFC+VU8o36/QUYB ze1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UfZUXel1p6AOKNdy1ef02UiRKIjcEL1X8er7dQKR42M=; b=qPcZPHFe8JwBkNc9h2IeNacJpvtIRRsFyULe6g3IUD+MbSsCpYQ6ZgpP/frkFh36pg 3UEooRn/lV0e9oa8ikKzA57YdmYmQLUsyRiVe8sBQb1v2XuYTOOZmPCGd4PKDTUQBD0c 00uTunC8PcA8tubIl5yCkQYbz9v0kz+Uaoxc8mtgeNCS67QtvyWm032RnCdyjrf21rbw x9ecRwSf9nlCuzMlEw8nRJQgh+ehZOBMal/wgYEJrZTOKakBxZjc+NpTar5rNleVRsTw kDrACgFdj+TqxwEDGnaWwIQujFxghpfovUk9EiiyXZPDHaq8ridqGEEXEZokQmrr8Df1 7apA== X-Gm-Message-State: AOAM532CTgBys+FQpaQZKpil3UUlpulS4CO8XfmSY3kZP25SMsSpvk4V rJTLGQRvYosQpA9M4HwnyjYgkfgocB4= X-Google-Smtp-Source: ABdhPJw2LkpQfWpv7I91AtezUZUN+HVaobQEiDFVz7y2Hh36CJo+lGtLY9xyBsAExm7fAqieiUWzLg== X-Received: by 2002:adf:82a1:: with SMTP id 30mr2683wrc.31.1625150873805; Thu, 01 Jul 2021 07:47:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f2sm127500wrq.69.2021.07.01.07.47.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:53 -0700 (PDT) Message-Id: <9fe902aad87f1192705fb69ea212a2d066d0286d.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:24 +0000 Subject: [PATCH v3 14/34] fsmonitor--daemon: implement 'start' command Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement 'git fsmonitor--daemon start' command. This command tries to start a daemon in the background. It creates a background process to run the daemon. The updated daemon does not actually do anything yet because the platform backends are still just stubs. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 208 ++++++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index a265c962ccc..7fcf960652f 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -9,6 +9,7 @@ #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon start []"), N_("git fsmonitor--daemon run []"), N_("git fsmonitor--daemon stop"), N_("git fsmonitor--daemon status"), @@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = { #define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" static int fsmonitor__ipc_threads = 8; +#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout" +static int fsmonitor__start_timeout_sec = 60; + static int fsmonitor_config(const char *var, const char *value, void *cb) { if (!strcmp(var, FSMONITOR__IPC_THREADS)) { @@ -33,6 +37,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, FSMONITOR__START_TIMEOUT)) { + int i = git_config_int(var, value); + if (i < 0) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__START_TIMEOUT, i); + fsmonitor__start_timeout_sec = i; + return 0; + } + return git_default_config(var, value, cb); } @@ -256,6 +269,194 @@ static int try_to_run_foreground_daemon(void) return !!fsmonitor_run_daemon(); } +#ifdef GIT_WINDOWS_NATIVE +/* + * Create a background process to run the daemon. It should be completely + * disassociated from the terminal. + * + * Conceptually like `daemonize()` but different because Windows does not + * have `fork(2)`. Spawn a normal Windows child process but without the + * limitations of `start_command()` and `finish_command()`. + * + * The child process execs the "git fsmonitor--daemon run" command. + * + * The current process returns so that the caller can wait for the child + * to startup before exiting. + */ +static int spawn_background_fsmonitor_daemon(pid_t *pid) +{ + char git_exe[MAX_PATH]; + struct strvec args = STRVEC_INIT; + int in, out; + + GetModuleFileNameA(NULL, git_exe, MAX_PATH); + + in = open("/dev/null", O_RDONLY); + out = open("/dev/null", O_WRONLY); + + strvec_push(&args, git_exe); + strvec_push(&args, "fsmonitor--daemon"); + strvec_push(&args, "run"); + strvec_pushf(&args, "--ipc-threads=%d", fsmonitor__ipc_threads); + + *pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out); + close(in); + close(out); + + strvec_clear(&args); + + if (*pid < 0) + return error(_("could not spawn fsmonitor--daemon in the background")); + + return 0; +} +#else +/* + * Create a background process to run the daemon. It should be completely + * disassociated from the terminal. + * + * This is adapted from `daemonize()`. Use `fork()` to directly + * create and run the daemon in the child process. + * + * The fork-child can just call the run code; it does not need to exec + * it. + * + * The fork-parent returns the child PID so that we can wait for the + * child to startup before exiting. + */ +static int spawn_background_fsmonitor_daemon(pid_t *pid) +{ + *pid = fork(); + + switch (*pid) { + case 0: + if (setsid() == -1) + error_errno(_("setsid failed")); + close(0); + close(1); + close(2); + sanitize_stdfds(); + + return !!fsmonitor_run_daemon(); + + case -1: + return error_errno(_("could not spawn fsmonitor--daemon in the background")); + + default: + return 0; + } +} +#endif + +/* + * This is adapted from `wait_or_whine()`. Watch the child process and + * let it get started and begin listening for requests on the socket + * before reporting our success. + */ +static int wait_for_background_startup(pid_t pid_child) +{ + int status; + pid_t pid_seen; + enum ipc_active_state s; + time_t time_limit, now; + + time(&time_limit); + time_limit += fsmonitor__start_timeout_sec; + + for (;;) { + pid_seen = waitpid(pid_child, &status, WNOHANG); + + if (pid_seen == -1) + return error_errno(_("waitpid failed")); + else if (pid_seen == 0) { + /* + * The child is still running (this should be + * the normal case). Try to connect to it on + * the socket and see if it is ready for + * business. + * + * If there is another daemon already running, + * our child will fail to start (possibly + * after a timeout on the lock), but we don't + * care (who responds) if the socket is live. + */ + s = fsmonitor_ipc__get_state(); + if (s == IPC_STATE__LISTENING) + return 0; + + time(&now); + if (now > time_limit) + return error(_("fsmonitor--daemon not online yet")); + } else if (pid_seen == pid_child) { + /* + * The new child daemon process shutdown while + * it was starting up, so it is not listening + * on the socket. + * + * Try to ping the socket in the odd chance + * that another daemon started (or was already + * running) while our child was starting. + * + * Again, we don't care who services the socket. + */ + s = fsmonitor_ipc__get_state(); + if (s == IPC_STATE__LISTENING) + return 0; + + /* + * We don't care about the WEXITSTATUS() nor + * any of the WIF*(status) values because + * `cmd_fsmonitor__daemon()` does the `!!result` + * trick on all function return values. + * + * So it is sufficient to just report the + * early shutdown as an error. + */ + return error(_("fsmonitor--daemon failed to start")); + } else + return error(_("waitpid is confused")); + } +} + +static int try_to_start_background_daemon(void) +{ + pid_t pid_child; + int ret; + + /* + * Before we try to create a background daemon process, see + * if a daemon process is already listening. This makes it + * easier for us to report an already-listening error to the + * console, since our spawn/daemon can only report the success + * of creating the background process (and not whether it + * immediately exited). + */ + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + die("fsmonitor--daemon is already running '%s'", + the_repository->worktree); + + printf(_("starting fsmonitor-daemon in '%s'\n"), + the_repository->worktree); + fflush(stdout); + + /* + * Run the actual daemon in a background process. + */ + ret = spawn_background_fsmonitor_daemon(&pid_child); + if (pid_child <= 0) + return ret; + + /* + * Wait (with timeout) for the background child process get + * started and begin listening on the socket/pipe. This makes + * the "start" command more synchronous and more reliable in + * tests. + */ + ret = wait_for_background_startup(pid_child); + + return ret; +} + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { const char *subcmd; @@ -264,6 +465,10 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) OPT_INTEGER(0, "ipc-threads", &fsmonitor__ipc_threads, N_("use ipc worker threads")), + OPT_INTEGER(0, "start-timeout", + &fsmonitor__start_timeout_sec, + N_("Max seconds to wait for background daemon startup")), + OPT_END() }; @@ -285,6 +490,9 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + if (!strcmp(subcmd, "start")) + return !!try_to_start_background_daemon(); + if (!strcmp(subcmd, "run")) return !!try_to_run_foreground_daemon(); From patchwork Thu Jul 1 14:47:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 817B1C11F69 for ; Thu, 1 Jul 2021 14:48:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 695606140C for ; Thu, 1 Jul 2021 14:48:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232974AbhGAOug (ORCPT ); Thu, 1 Jul 2021 10:50:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232923AbhGAOu1 (ORCPT ); Thu, 1 Jul 2021 10:50:27 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0CD2C0613E6 for ; Thu, 1 Jul 2021 07:47:55 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id i94so8602675wri.4 for ; Thu, 01 Jul 2021 07:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vMyqH4atwMy0u8EpIhOh4lmPem4fMyJcVWM1txZGxbs=; b=GBs+I4PpzHuu4Y3uxLRIW+4CsoUeQmUBc3jnX8ZejlLahZ7pgx87zoyloaDS1dmC1T l2Had6DuRLIpi6EcXAWKAruyySORq4Icy6nvATBUkx5XYElJjUBlq/Ww8/i605E2B5KC QXgbl1dPAxoptPt5lXjP8QvBGNSEL4SBbColeWoyN88PHwDpVhHO6AnNZUpxw4fwJB2p vatibS205gHChZSXwiDXUn8xo11V8OO5ZeBo/XTXlH6LAZO3swYQeQRo7r90eNSq1UnO +pZpw1LORHjqpVCbY/OlfNRBqE+SEsRmD+tKRNkrtbdvDnNjtwMEhu2mAHn6w/PpWcOU aWyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vMyqH4atwMy0u8EpIhOh4lmPem4fMyJcVWM1txZGxbs=; b=kgEAu8AkZMaF6sVLzsZW8fdPNlsgY+jSg7bRby1Jbtkz+HppbNBeTlPFIrGHDaAZlK CbRM35xPFScwbonTV+yByAzxfg9/qSdDfdOVi2EgIedVU1W0cmvuLNBBXIRGuVsjRN9j HWv94Zr6ND/WRd7rEXuG8h7qRupu54PmfBstK1Y2CfhRDKOGWRWxT8JmfxawwuGJxBdi YQBM4rLRGC9vwYJIraWY7dufymy6gVHHs3YO4tBU39mTK8DEhycaNICnsaO/NSdrMZJq i8R/6cEMYkuj02OCAMxMu9e3U8OnypExba+lBacbJcCXyJN95pDNtaRBgZ435ETt/T5U rIpw== X-Gm-Message-State: AOAM5320zBmV3cUYEwfzEngTploay6YKCWXxJ7vrS9rZIoZQTL2gHz2w ut0sRLj9IFlCaZXIHK4hT04pqBSh9x8= X-Google-Smtp-Source: ABdhPJyKGADITEgtTh4cpQ/4iQVdilhIZVpldaeeJoZX/jqrv/S7tgXAIMTT7DybzTXN4hjF6741PQ== X-Received: by 2002:a5d:488a:: with SMTP id g10mr30379wrq.180.1625150874413; Thu, 01 Jul 2021 07:47:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y66sm235579wmy.39.2021.07.01.07.47.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:54 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:25 +0000 Subject: [PATCH v3 15/34] fsmonitor: do not try to operate on bare repos Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Bare repos do not have a working directory, so there is no directory for the daemon to register a watch upon. And therefore there are no files within the directory for it to actually watch. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 8 ++++++++ t/t7519-status-fsmonitor.sh | 16 ++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 7fcf960652f..d6161ad95a5 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -490,6 +490,14 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + prepare_repo_settings(the_repository); + if (!the_repository->worktree) + return error(_("fsmonitor-daemon does not support bare repos '%s'"), + xgetcwd()); + if (the_repository->settings.fsmonitor_mode == FSMONITOR_MODE_INCOMPATIBLE) + return error(_("fsmonitor-daemon is incompatible with this repo '%s'"), + the_repository->worktree); + if (!strcmp(subcmd, "start")) return !!try_to_start_background_daemon(); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 02919c68ddd..ed20a4f7fb9 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -394,6 +394,22 @@ test_expect_success 'incompatible bare repo' ' test_cmp expect actual ' +test_expect_success FSMONITOR_DAEMON 'try running fsmonitor-daemon in bare repo' ' + test_when_finished "rm -rf ./bare-clone" && + git clone --bare . ./bare-clone && + test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual && + grep "fsmonitor-daemon does not support bare repos" actual +' + +test_expect_success FSMONITOR_DAEMON 'try running fsmonitor-daemon in virtual repo' ' + test_when_finished "rm -rf ./fake-virtual-clone" && + git clone . ./fake-virtual-clone && + test_must_fail git -C ./fake-virtual-clone \ + -c core.virtualfilesystem=true \ + fsmonitor--daemon run 2>actual && + grep "fsmonitor-daemon is incompatible with this repo" actual +' + test_expect_success 'incompatible core.virtualfilesystem' ' test_when_finished "rm -rf ./fake-gvfs-clone" && git clone . ./fake-gvfs-clone && From patchwork Thu Jul 1 14:47:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2F3AC11F64 for ; Thu, 1 Jul 2021 14:48:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9CEA56140C for ; Thu, 1 Jul 2021 14:48:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233094AbhGAOui (ORCPT ); Thu, 1 Jul 2021 10:50:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232817AbhGAOu2 (ORCPT ); Thu, 1 Jul 2021 10:50:28 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DADEC061762 for ; Thu, 1 Jul 2021 07:47:56 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id g8-20020a1c9d080000b02901f13dd1672aso4392592wme.0 for ; Thu, 01 Jul 2021 07:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=e9Y6nq/FoJ44oZvKJhw5ixuYaT7LRyHPOTQ3LNw2vyQ=; b=EG1Ihcx8vi7tFeaucaCuhWV8BaalCFgJykui/8EwhFNdffa5pfY9s3U+L93MRGf30a ePr2BnBJcSf1HLBv7uvm/iREmAe8Jz46sTMofH/biPQlh3vIMTAM3DsufJ1SSfr6G9ve pzUGnd/CM3w+LHGnpAS1j9Tl/qGhzLdOOOn4amK58gbApIkRFyxlDXqCHCtCcWO0RWfp ePFEtDVZZ15cL2+TGZkPesGqZQ4ODB6+bzd5G5EG8+1GGSlzS8ZdYVgMRyoCzDD+fXnw 6H8ATPN9X7OVzj0oPvPTxegOtZ6lS+hQCrjdIC0r+Hr/dF1ih8Aa4hEG//KlAGECvRKH mXUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=e9Y6nq/FoJ44oZvKJhw5ixuYaT7LRyHPOTQ3LNw2vyQ=; b=oAxkCPuwXXT8rgnajhqJR2N5tTWJQ3b/aJkkPxumcqkH8hNMCjWX+KVoRX7BXOvbU4 Mug0z3X2cSpcH6RKP67AzjwAQGtZ+ZpI+ynx0JKSBYvDm0x5slrZX5/AUumKIgd/vopX Gsdsjbk7WfWm5rHqrkHGxpCMpHgl3lOnxzsbexQhrs9VgcFRfbKwFQN9pykPdEeP7s9T CpO1JDNR22dh3B/DfsV80jPfYAnrba01zNQtMAKdP4KKIH1F/myfIVyCLcTmfrWuUwzk /C1mkmKYzK1XLCNHYHRq+5uWfGAD0ziwC5BRafda3MnFN0NOgojR1cPhlxeAAmKD/Roz wkcA== X-Gm-Message-State: AOAM533XdQc2NxatBugO+vca6Mep2ATgcFzrO3hBE/k73JcOu4o1hD2T Pf3J2t1R8nQJc2BMg8XA7sol+17m1+Y= X-Google-Smtp-Source: ABdhPJxB55g74BC++KUx2M1JIiMzdvSzdCBkOmuWe+XzpRAWskC4KeYZT7ZTySD/jKWs9bamovpjOA== X-Received: by 2002:a1c:7f4a:: with SMTP id a71mr11185167wmd.33.1625150874983; Thu, 01 Jul 2021 07:47:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p7sm134749wrr.68.2021.07.01.07.47.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:54 -0700 (PDT) Message-Id: <3b12f668060fc757d7d36dcad2c86bbb917a8b8b.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:26 +0000 Subject: [PATCH v3 16/34] fsmonitor--daemon: add pathname classification Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to classify relative and absolute pathnames and decide how they should be handled. This will be used by the platform-specific backend to respond to each filesystem event. When we register for filesystem notifications on a directory, we get events for everything (recursively) in the directory. We want to report to clients changes to tracked and untracked paths within the working directory. We do not want to report changes within the .git directory, for example. This classification will be used in a later commit by the different backends to classify paths as events are received. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index d6161ad95a5..e942f7c5840 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -121,6 +121,87 @@ static int handle_client(void *data, return result; } +#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" + +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *rel) +{ + if (fspathncmp(rel, ".git", 4)) + return IS_WORKDIR_PATH; + rel += 4; + + if (!*rel) + return IS_DOT_GIT; + if (*rel != '/') + return IS_WORKDIR_PATH; /* e.g. .gitignore */ + rel++; + + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX; + + return IS_INSIDE_DOT_GIT; +} + +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *rel) +{ + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX; + + return IS_INSIDE_GITDIR; +} + +static enum fsmonitor_path_type try_classify_workdir_abs_path( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + + if (fspathncmp(path, state->path_worktree_watch.buf, + state->path_worktree_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_worktree_watch.len; + + if (!*rel) + return IS_WORKDIR_PATH; /* it is the root dir exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_workdir_relative(rel); +} + +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + enum fsmonitor_path_type t; + + t = try_classify_workdir_abs_path(state, path); + if (state->nr_paths_watching == 1) + return t; + if (t != IS_OUTSIDE_CONE) + return t; + + if (fspathncmp(path, state->path_gitdir_watch.buf, + state->path_gitdir_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_gitdir_watch.len; + + if (!*rel) + return IS_GITDIR; /* it is the exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_gitdir_relative(rel); +} + static void *fsmonitor_fs_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 3009c1a83de..7bbb3a27a1c 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -30,5 +30,66 @@ struct fsmonitor_daemon_state { struct ipc_server_data *ipc_server_data; }; +/* + * Pathname classifications. + * + * The daemon classifies the pathnames that it receives from file + * system notification events into the following categories and uses + * that to decide whether clients are told about them. (And to watch + * for file system synchronization events.) + * + * The client should only care about paths within the working + * directory proper (inside the working directory and not ".git" nor + * inside of ".git/"). That is, the client has read the index and is + * asking for a list of any paths in the working directory that have + * been modified since the last token. The client does not care about + * file system changes within the .git directory (such as new loose + * objects or packfiles). So the client will only receive paths that + * are classified as IS_WORKDIR_PATH. + * + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the + * exact ".git" directory or GITDIR. If the daemon receives a delete + * event for either of these directories, it will automatically + * shutdown, for example. + * + * Note that the daemon DOES NOT explicitly watch nor special case the + * ".git/index" file. The daemon does not read the index and does not + * have any internal index-relative state. The daemon only collects + * the set of modified paths within the working directory. + */ +enum fsmonitor_path_type { + IS_WORKDIR_PATH = 0, + + IS_DOT_GIT, + IS_INSIDE_DOT_GIT, + IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX, + + IS_GITDIR, + IS_INSIDE_GITDIR, + IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX, + + IS_OUTSIDE_CONE, +}; + +/* + * Classify a pathname relative to the root of the working directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *relative_path); + +/* + * Classify a pathname relative to a that is external to the + * worktree directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *relative_path); + +/* + * Classify an absolute pathname received from a filesystem event. + */ +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Thu Jul 1 14:47:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75B9AC11F69 for ; Thu, 1 Jul 2021 14:48:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5F1E761414 for ; Thu, 1 Jul 2021 14:48:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233122AbhGAOuj (ORCPT ); Thu, 1 Jul 2021 10:50:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232625AbhGAOu2 (ORCPT ); Thu, 1 Jul 2021 10:50:28 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22FC6C061765 for ; Thu, 1 Jul 2021 07:47:57 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id i94so8602764wri.4 for ; Thu, 01 Jul 2021 07:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wR7kYAWIpdTP/+5YQ3Emp8FTipGBv/xXPqjGmoTjFMw=; b=mJJ7ashoKQVjtWiim0AQV8cbigR0mD0vE0zD+ljBBMJWFDRCCv+LC412YZ46zLpkAS df451Z+rLbrhS59WKI7Hw+CZvkpJmRR2VTzsxzp2DlMPfQsxqJ5bI635YL3oXdlASWoP lL+Fppq+Gw9UGygKyw1JncIapEoZBWgDO+4mz+QwUBm7jE3M2GFvxiyZNahN7HIvr2dF U8GQmFo8TakGmaGmPBAZc7nO7PtoH7XLIU3qxAnlAh3nu45L32OLbE0MsuJGl59BAKy5 QevIWFf+6B68nbwb9fzfjqdJCIxdcoWfoIonVDMK/Z6OquePZeJTovJbOQZfSDL7QCqG 2EoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wR7kYAWIpdTP/+5YQ3Emp8FTipGBv/xXPqjGmoTjFMw=; b=ApsPouxhFIlOxMizGuWi7gCYcX5T+Eut9enunUImHVsReyJQJoazA0M1sdZURAQVpJ bXTD6I3M2sM26jcwQ48oEfFoiaG2EtC9ruu5HBzXLlQe2KChRtZO74tU+LrL4PohNT3S I3Ar/acd/HhKo9xK6oyConaZC3gIAoHSyDJ404RKSYr7lSKwMYQmW2ZSlFjiKTN+HwfK rYcOzgFgG9+DyKQSR5Wvg/EzlQBwb+jicjqYAu9wXQubCYC8/vQ4KqSGkckIz5e/2sKk JPqk+88QZKQJ7uktB2316hA5CP+M0AA4pwMTQMzIeGhdLEGNc23YDZU123FsQ2jjolof wAtQ== X-Gm-Message-State: AOAM532tkFrcMxENU6Lf7Lgmf0Io0jFzT3l0xY49gJxLUVR+C9uFh7D/ XUttjS+KXCb81PmHHonsbR7pGIVf+oE= X-Google-Smtp-Source: ABdhPJyy9pr+XTYg1uC2mERpEBTbJ4tTrKuCee7espsGVfUANblMV9LiZfSOt1OXfeFiml7gg0kiBg== X-Received: by 2002:a05:6000:42:: with SMTP id k2mr21370wrx.294.1625150875720; Thu, 01 Jul 2021 07:47:55 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u15sm9680812wmq.48.2021.07.01.07.47.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:55 -0700 (PDT) Message-Id: <37fdce5ec3afaa9aae5001c648fced0675dae0c4.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:27 +0000 Subject: [PATCH v3 17/34] fsmonitor--daemon: define token-ids Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to create token-ids and define the overall token naming scheme. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++- 1 file changed, 115 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index e942f7c5840..e991925fafc 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -93,6 +93,120 @@ static int do_as_client__status(void) } } +/* + * Requests to and from a FSMonitor Protocol V2 provider use an opaque + * "token" as a virtual timestamp. Clients can request a summary of all + * created/deleted/modified files relative to a token. In the response, + * clients receive a new token for the next (relative) request. + * + * + * Token Format + * ============ + * + * The contents of the token are private and provider-specific. + * + * For the built-in fsmonitor--daemon, we define a token as follows: + * + * "builtin" ":" ":" + * + * The "builtin" prefix is used as a namespace to avoid conflicts + * with other providers (such as Watchman). + * + * The is an arbitrary OPAQUE string, such as a GUID, + * UUID, or {timestamp,pid}. It is used to group all filesystem + * events that happened while the daemon was monitoring (and in-sync + * with the filesystem). + * + * Unlike FSMonitor Protocol V1, it is not defined as a timestamp + * and does not define less-than/greater-than relationships. + * (There are too many race conditions to rely on file system + * event timestamps.) + * + * The is a simple integer incremented whenever the + * daemon needs to make its state public. For example, if 1000 file + * system events come in, but no clients have requested the data, + * the daemon can continue to accumulate file changes in the same + * bin and does not need to advance the sequence number. However, + * as soon as a client does arrive, the daemon needs to start a new + * bin and increment the sequence number. + * + * The sequence number serves as the boundary between 2 sets + * of bins -- the older ones that the client has already seen + * and the newer ones that it hasn't. + * + * When a new is created, the is reset to + * zero. + * + * + * About Token Ids + * =============== + * + * A new token_id is created: + * + * [1] each time the daemon is started. + * + * [2] any time that the daemon must re-sync with the filesystem + * (such as when the kernel drops or we miss events on a very + * active volume). + * + * [3] in response to a client "flush" command (for dropped event + * testing). + * + * When a new token_id is created, the daemon is free to discard all + * cached filesystem events associated with any previous token_ids. + * Events associated with a non-current token_id will never be sent + * to a client. A token_id change implicitly means that the daemon + * has gap in its event history. + * + * Therefore, clients that present a token with a stale (non-current) + * token_id will always be given a trivial response. + */ +struct fsmonitor_token_data { + struct strbuf token_id; + struct fsmonitor_batch *batch_head; + struct fsmonitor_batch *batch_tail; + uint64_t client_ref_count; +}; + +static struct fsmonitor_token_data *fsmonitor_new_token_data(void) +{ + static int test_env_value = -1; + static uint64_t flush_count = 0; + struct fsmonitor_token_data *token; + + CALLOC_ARRAY(token, 1); + + strbuf_init(&token->token_id, 0); + token->batch_head = NULL; + token->batch_tail = NULL; + token->client_ref_count = 0; + + if (test_env_value < 0) + test_env_value = git_env_bool("GIT_TEST_FSMONITOR_TOKEN", 0); + + if (!test_env_value) { + struct timeval tv; + struct tm tm; + time_t secs; + + gettimeofday(&tv, NULL); + secs = tv.tv_sec; + gmtime_r(&secs, &tm); + + strbuf_addf(&token->token_id, + "%"PRIu64".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ", + flush_count++, + getpid(), + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (long)tv.tv_usec); + } else { + strbuf_addf(&token->token_id, "test_%08x", test_env_value++); + } + + return token; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -283,7 +397,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); state.error_code = 0; - state.current_token_data = NULL; + state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ strbuf_init(&state.path_worktree_watch, 0); From patchwork Thu Jul 1 14:47:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BB88C11F67 for ; Thu, 1 Jul 2021 14:48:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 37A126140C for ; Thu, 1 Jul 2021 14:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233104AbhGAOui (ORCPT ); Thu, 1 Jul 2021 10:50:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232934AbhGAOu2 (ORCPT ); Thu, 1 Jul 2021 10:50:28 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0D8BC061764 for ; Thu, 1 Jul 2021 07:47:57 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id i8so8666017wrc.0 for ; Thu, 01 Jul 2021 07:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2KYx43qXPqIulIoUrZvaz5cMNrWrv3ENjl/SFqtWWDk=; b=WTzecqUd7o13ICzJ9R3s6Zcu5vfgaH6qmxyNV64+jos7Qb7VU0te9u/xTXRA0ODyjX UZdCnntTkvqaMwpg3YuLANw+l0QxnmXIxGZdwdZ9WGlEgA+9/hO6l+jz4RV6lVPPUKFF YIomyWJx1L4m8y/Bi1DPuEUp9pkZiZoowG7s5KUNyYM8ZrxhUzip0EQFLw1crij9eQ+y A1AP9ofL5m4KN5TYs58gctHbiajVLDhhEWnBxcqb/JyBUXurMKI1Pmk07Do2UHOyX3Eb xno08fUt8GxqyDKyoSANy/E9eP2pYVj+mw175EfX4mBducmBJ7NjfH27pgdJuPm2JivI nPCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2KYx43qXPqIulIoUrZvaz5cMNrWrv3ENjl/SFqtWWDk=; b=M1VIoEOKMMX1ZeE3nazKQuas5vBqpdEWQKTVfyeqG+x2nj2ubZNjoidFE1AnTCTRrX nHgiFXf9duic7nc9W+C+LpYKGouNT+LDFynBKwWFlVsF0K9KK0kR1IBT/poFYCzNdsBk /V1afo1nuMg3iyqCuKs4Iti722lnShmZNhofoAE4qWuPHCpgACeIpX2VabA21Vb4DNQ6 Zsza4D/owT494WqgBP1cCxNpKCvyAQeYZITQs4Qt5LnYWwkMbDhGdJWaMudLa1jJdKYD Q03F/xvQ1gr8XmHE0aDfHq+TgFIuD0WeGFWzVx2cnJ/OUdIdc8VjdcDSiIC/A+/bZiV8 nyEA== X-Gm-Message-State: AOAM532fm99XtvVYJOIz5Qqv5bp7115O9dY5fbnPBUsrGcjfIwFOlUlw +4fF2QUPT48KBkC9kIJsRXEFpU3Tiu4= X-Google-Smtp-Source: ABdhPJzZAN/g14yqqBmIPn0TaxWzbutZCjhR5ZkfKJ36Kyvxmv1MXBkyGg/bbxOkMFrKhJIdnXUx8g== X-Received: by 2002:a05:6000:108a:: with SMTP id y10mr2400wrw.49.1625150876289; Thu, 01 Jul 2021 07:47:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b187sm10959464wmh.32.2021.07.01.07.47.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:56 -0700 (PDT) Message-Id: <84444c44c324751ecaed30a4fa2a868014da4397.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:28 +0000 Subject: [PATCH v3 18/34] fsmonitor--daemon: create token-based changed path cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to build a list of changed paths and associate them with a token-id. This will be used by the platform-specific backends to accumulate changed paths in response to filesystem events. The platform-specific file system listener thread receives file system events containing one or more changed pathnames (with whatever bucketing or grouping that is convenient for the file system). These paths are accumulated (without locking) by the file system layer into a `fsmonitor_batch`. When the file system layer has drained the kernel event queue, it will "publish" them to our token queue and make them visible to concurrent client worker threads. The token layer is free to combine and/or de-dup paths within these batches for efficient presentation to clients. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 234 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 40 ++++++ 2 files changed, 272 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index e991925fafc..ea3a52d34e3 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -168,17 +168,27 @@ struct fsmonitor_token_data { uint64_t client_ref_count; }; +struct fsmonitor_batch { + struct fsmonitor_batch *next; + uint64_t batch_seq_nr; + const char **interned_paths; + size_t nr, alloc; + time_t pinned_time; +}; + static struct fsmonitor_token_data *fsmonitor_new_token_data(void) { static int test_env_value = -1; static uint64_t flush_count = 0; struct fsmonitor_token_data *token; + struct fsmonitor_batch *batch; CALLOC_ARRAY(token, 1); + batch = fsmonitor_batch__new(); strbuf_init(&token->token_id, 0); - token->batch_head = NULL; - token->batch_tail = NULL; + token->batch_head = batch; + token->batch_tail = batch; token->client_ref_count = 0; if (test_env_value < 0) @@ -204,9 +214,147 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void) strbuf_addf(&token->token_id, "test_%08x", test_env_value++); } + /* + * We created a new and are starting a new series + * of tokens with a zero . + * + * Since clients cannot guess our new (non test) + * they will always receive a trivial response (because of the + * mismatch on the ). The trivial response will + * tell them our new so that subsequent requests + * will be relative to our new series. (And when sending that + * response, we pin the current head of the batch list.) + * + * Even if the client correctly guesses the , their + * request of "builtin::0" asks for all changes MORE + * RECENT than batch/bin 0. + * + * This implies that it is a waste to accumulate paths in the + * initial batch/bin (because they will never be transmitted). + * + * So the daemon could be running for days and watching the + * file system, but doesn't need to actually accumulate any + * paths UNTIL we need to set a reference point for a later + * relative request. + * + * However, it is very useful for testing to always have a + * reference point set. Pin batch 0 to force early file system + * events to accumulate. + */ + if (test_env_value) + batch->pinned_time = time(NULL); + return token; } +struct fsmonitor_batch *fsmonitor_batch__new(void) +{ + struct fsmonitor_batch *batch; + + CALLOC_ARRAY(batch, 1); + + return batch; +} + +struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch) +{ + struct fsmonitor_batch *next; + + if (!batch) + return NULL; + + next = batch->next; + + /* + * The actual strings within the array are interned, so we don't + * own them. + */ + free(batch->interned_paths); + + return next; +} + +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, + const char *path) +{ + const char *interned_path = strintern(path); + + trace_printf_key(&trace_fsmonitor, "event: %s", interned_path); + + ALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc); + batch->interned_paths[batch->nr++] = interned_path; +} + +static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, + const struct fsmonitor_batch *batch_src) +{ + size_t k; + + ALLOC_GROW(batch_dest->interned_paths, + batch_dest->nr + batch_src->nr + 1, + batch_dest->alloc); + + for (k = 0; k < batch_src->nr; k++) + batch_dest->interned_paths[batch_dest->nr++] = + batch_src->interned_paths[k]; +} + +static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) +{ + struct fsmonitor_batch *p; + + if (!token) + return; + + assert(token->client_ref_count == 0); + + strbuf_release(&token->token_id); + + for (p = token->batch_head; p; p = fsmonitor_batch__pop(p)) + ; + + free(token); +} + +/* + * Flush all of our cached data about the filesystem. Call this if we + * lose sync with the filesystem and miss some notification events. + * + * [1] If we are missing events, then we no longer have a complete + * history of the directory (relative to our current start token). + * We should create a new token and start fresh (as if we just + * booted up). + * + * If there are no concurrent threads readering the current token data + * series, we can free it now. Otherwise, let the last reader free + * it. + * + * Either way, the old token data series is no longer associated with + * our state data. + */ +static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + struct fsmonitor_token_data *free_me = NULL; + struct fsmonitor_token_data *new_one = NULL; + + new_one = fsmonitor_new_token_data(); + + if (state->current_token_data->client_ref_count == 0) + free_me = state->current_token_data; + state->current_token_data = new_one; + + fsmonitor_free_token_data(free_me); +} + +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) +{ + pthread_mutex_lock(&state->main_lock); + with_lock__do_force_resync(state); + pthread_mutex_unlock(&state->main_lock); +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -316,6 +464,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( return fsmonitor_classify_path_gitdir_relative(rel); } +/* + * We try to combine small batches at the front of the batch-list to avoid + * having a long list. This hopefully makes it a little easier when we want + * to truncate and maintain the list. However, we don't want the paths array + * to just keep growing and growing with realloc, so we insert an arbitrary + * limit. + */ +#define MY_COMBINE_LIMIT (1024) + +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names) +{ + if (!batch && !cookie_names->nr) + return; + + pthread_mutex_lock(&state->main_lock); + + if (batch) { + struct fsmonitor_batch *head; + + head = state->current_token_data->batch_head; + if (!head) { + BUG("token does not have batch"); + } else if (head->pinned_time) { + /* + * We cannot alter the current batch list + * because: + * + * [a] it is being transmitted to at least one + * client and the handle_client() thread has a + * ref-count, but not a lock on the batch list + * starting with this item. + * + * [b] it has been transmitted in the past to + * at least one client such that future + * requests are relative to this head batch. + * + * So, we can only prepend a new batch onto + * the front of the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else if (!head->batch_seq_nr) { + /* + * Batch 0 is unpinned. See the note in + * `fsmonitor_new_token_data()` about why we + * don't need to accumulate these paths. + */ + fsmonitor_batch__pop(batch); + } else if (head->nr + batch->nr > MY_COMBINE_LIMIT) { + /* + * The head batch in the list has never been + * transmitted to a client, but folding the + * contents of the new batch onto it would + * exceed our arbitrary limit, so just prepend + * the new batch onto the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else { + /* + * We are free to append the paths in the given + * batch onto the end of the current head batch. + */ + fsmonitor_batch__combine(head, batch); + fsmonitor_batch__pop(batch); + } + } + + pthread_mutex_unlock(&state->main_lock); +} + static void *fsmonitor_fs_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -330,6 +553,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state) fsmonitor_fs_listen__loop(state); + pthread_mutex_lock(&state->main_lock); + if (state->current_token_data && + state->current_token_data->client_ref_count == 0) + fsmonitor_free_token_data(state->current_token_data); + state->current_token_data = NULL; + pthread_mutex_unlock(&state->main_lock); + trace2_thread_exit(); return NULL; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 7bbb3a27a1c..89a9ef20b24 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -12,6 +12,27 @@ struct fsmonitor_batch; struct fsmonitor_token_data; +/* + * Create a new batch of path(s). The returned batch is considered + * private and not linked into the fsmonitor daemon state. The caller + * should fill this batch with one or more paths and then publish it. + */ +struct fsmonitor_batch *fsmonitor_batch__new(void); + +/* + * Free this batch and return the value of the batch->next field. + */ +struct fsmonitor_batch *fsmonitor_batch__pop(struct fsmonitor_batch *batch); + +/* + * Add this path to this batch of modified files. + * + * The batch should be private and NOT (yet) linked into the fsmonitor + * daemon state and therefore not yet visible to worker threads and so + * no locking is required. + */ +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); + struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ struct fsmonitor_daemon_state { @@ -91,5 +112,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( struct fsmonitor_daemon_state *state, const char *path); +/* + * Prepend the this batch of path(s) onto the list of batches associated + * with the current token. This makes the batch visible to worker threads. + * + * The caller no longer owns the batch and must not free it. + * + * Wake up the client threads waiting on these cookies. + */ +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names); + +/* + * If the platform-specific layer loses sync with the filesystem, + * it should call this to invalidate cached data and abort waiting + * threads. + */ +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Thu Jul 1 14:47:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C364FC11F69 for ; Thu, 1 Jul 2021 14:48:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ABCC961403 for ; Thu, 1 Jul 2021 14:48:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233179AbhGAOum (ORCPT ); Thu, 1 Jul 2021 10:50:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232566AbhGAOu3 (ORCPT ); Thu, 1 Jul 2021 10:50:29 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FA69C061764 for ; Thu, 1 Jul 2021 07:47:58 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id g7so8563152wri.7 for ; Thu, 01 Jul 2021 07:47:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=AkkuI1rDyt13LEjLwe9K5RcZYp+AXheqRPpzBtH+UoY=; b=bWNAIlRc9uZjahv3D2zsl7yBOnQfZpTy5GBZfDeyCj8xozublidE5MJuTaGqQOAf6m mAfW3wegSBHoxAmDkGPDes8dbqBOq9JaGz4AUTGRsR/E4DMiPBsQwGOzrXAh+Yq7ypRJ UOiEBl/ywhE3yWu8XrBzTEam/wj8ZAYgFNMPNVCqYizFqwN0VmW+fh4F2SeSsnumFGWS G2GIiwZkCyhuL4pvYQlJVFEKwr9HCf7IYIz5zwoQNwDnDbPicqUNGlRg7AxwlWNakTDc QLtywJxhW2idnzfkchG3qf4d2BNDuXvY9Bq7RjAKuzeqprIbKqRMu0PggXkcsW2vngJl Jpug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=AkkuI1rDyt13LEjLwe9K5RcZYp+AXheqRPpzBtH+UoY=; b=tafUpihpElEnpdXVuiX7AT2WI6O5plSL8ykH+vRBs8aVGCQRPvhM4A3a5I937HsyVi eJEUQ/X3qmKP/NGz2Yv9nvTVHpKu8LXdQYYFlJIVqygpPx7/TB74XhV1iTlPsV3cnbsK 8CBufj7ZOyLjkqHqNpmMyJVwzCpvVCQyUKdqkbIw1Z2Bexni7020dJHmoNpRQCvyq+BR 0R4JO7Pw/40TeJj0CcfxKNROp10zf22B6jw8fsp0HgXYE0q/wx01dcnqKBPUy7LISWnP ppBySXLlx9Fee0j5R+MMRa9sRaurt4kO34naGEIvmwe45v+d4IoFDypRc+Q0WHK6jMvr Pv0g== X-Gm-Message-State: AOAM533SAqTlqtLphcs46vqj/s5UsBFAFD/7ZP20TDZ7Xl0Fb0PbN9X1 MjQNHFCnOC0pSckfhIXR/vHbPrru40Q= X-Google-Smtp-Source: ABdhPJxGF8lp2o1dSf9+S84BJrgogjVc30zWKHS7qPKng9MrGW4NP+ubyp0aTHypNyhKootHC1ldaw== X-Received: by 2002:adf:f048:: with SMTP id t8mr2824wro.35.1625150876820; Thu, 01 Jul 2021 07:47:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q6sm9849038wma.16.2021.07.01.07.47.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:56 -0700 (PDT) Message-Id: <5bba5eb3d1bd172f09fdf6eb2e9b8ac4dd7f940f.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:29 +0000 Subject: [PATCH v3 19/34] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the win32 backend to register a watch on the working tree root directory (recursively). Also watch the if it is not inside the working tree. And to collect path change notifications into batches and publish. Signed-off-by: Jeff Hostetler Signed-off-by: Johannes Schindelin --- compat/fsmonitor/fsmonitor-fs-listen-win32.c | 530 +++++++++++++++++++ 1 file changed, 530 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c index 880446b49e3..d707d47a0d7 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -2,20 +2,550 @@ #include "config.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +/* + * The documentation of ReadDirectoryChangesW() states that the maximum + * buffer size is 64K when the monitored directory is remote. + * + * Larger buffers may be used when the monitored directory is local and + * will help us receive events faster from the kernel and avoid dropped + * events. + * + * So we try to use a very large buffer and silently fallback to 64K if + * we get an error. + */ +#define MAX_RDCW_BUF_FALLBACK (65536) +#define MAX_RDCW_BUF (65536 * 8) + +struct one_watch +{ + char buffer[MAX_RDCW_BUF]; + DWORD buf_len; + DWORD count; + + struct strbuf path; + HANDLE hDir; + HANDLE hEvent; + OVERLAPPED overlapped; + + /* + * Is there an active ReadDirectoryChangesW() call pending. If so, we + * need to later call GetOverlappedResult() and possibly CancelIoEx(). + */ + BOOL is_active; +}; + +struct fsmonitor_daemon_backend_data +{ + struct one_watch *watch_worktree; + struct one_watch *watch_gitdir; + + HANDLE hEventShutdown; + + HANDLE hListener[3]; /* we don't own these handles */ +#define LISTENER_SHUTDOWN 0 +#define LISTENER_HAVE_DATA_WORKTREE 1 +#define LISTENER_HAVE_DATA_GITDIR 2 + int nr_listener_handles; +}; + +/* + * Convert the WCHAR path from the notification into UTF8 and + * then normalize it. + */ +static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, + struct strbuf *normalized_path) +{ + int reserve; + int len = 0; + + strbuf_reset(normalized_path); + if (!info->FileNameLength) + goto normalize; + + /* + * Pre-reserve enough space in the UTF8 buffer for + * each Unicode WCHAR character to be mapped into a + * sequence of 2 UTF8 characters. That should let us + * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. + */ + reserve = info->FileNameLength + 1; + strbuf_grow(normalized_path, reserve); + + for (;;) { + len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, + info->FileNameLength / sizeof(WCHAR), + normalized_path->buf, + strbuf_avail(normalized_path) - 1, + NULL, NULL); + if (len > 0) + goto normalize; + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + error("[GLE %ld] could not convert path to UTF-8: '%.*ls'", + GetLastError(), + (int)(info->FileNameLength / sizeof(WCHAR)), + info->FileName); + return -1; + } + + strbuf_grow(normalized_path, + strbuf_avail(normalized_path) + reserve); + } + +normalize: + strbuf_setlen(normalized_path, len); + return strbuf_normalize_path(normalized_path); +} void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) { + SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); +} + +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, + const char *path) +{ + struct one_watch *watch = NULL; + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileA(path, + desired_access, share_mode, NULL, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, + NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] could not watch '%s'"), + GetLastError(), path); + return NULL; + } + + CALLOC_ARRAY(watch, 1); + + watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ + + strbuf_init(&watch->path, 0); + strbuf_addstr(&watch->path, path); + + watch->hDir = hDir; + watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); + + return watch; +} + +static void destroy_watch(struct one_watch *watch) +{ + if (!watch) + return; + + strbuf_release(&watch->path); + if (watch->hDir != INVALID_HANDLE_VALUE) + CloseHandle(watch->hDir); + if (watch->hEvent != INVALID_HANDLE_VALUE) + CloseHandle(watch->hEvent); + + free(watch); +} + +static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, + struct one_watch *watch) +{ + DWORD dwNotifyFilter = + FILE_NOTIFY_CHANGE_FILE_NAME | + FILE_NOTIFY_CHANGE_DIR_NAME | + FILE_NOTIFY_CHANGE_ATTRIBUTES | + FILE_NOTIFY_CHANGE_SIZE | + FILE_NOTIFY_CHANGE_LAST_WRITE | + FILE_NOTIFY_CHANGE_CREATION; + + ResetEvent(watch->hEvent); + + memset(&watch->overlapped, 0, sizeof(watch->overlapped)); + watch->overlapped.hEvent = watch->hEvent; + +start_watch: + /* + * Queue an async call using Overlapped IO. This returns immediately. + * Our event handle will be signalled when the real result is available. + * + * The return value here just means that we successfully queued it. + * We won't know if the Read...() actually produces data until later. + */ + watch->is_active = ReadDirectoryChangesW( + watch->hDir, watch->buffer, watch->buf_len, TRUE, + dwNotifyFilter, &watch->count, &watch->overlapped, NULL); + + /* + * The kernel throws an invalid parameter error when our buffer + * is too big and we are pointed at a remote directory (and possibly + * for other reasons). Quietly set it down and try again. + * + * See note about MAX_RDCW_BUF at the top. + */ + if (!watch->is_active && + GetLastError() == ERROR_INVALID_PARAMETER && + watch->buf_len > MAX_RDCW_BUF_FALLBACK) { + watch->buf_len = MAX_RDCW_BUF_FALLBACK; + goto start_watch; + } + + if (watch->is_active) + return 0; + + error("ReadDirectoryChangedW failed on '%s' [GLE %ld]", + watch->path.buf, GetLastError()); + return -1; +} + +static int recv_rdcw_watch(struct one_watch *watch) +{ + watch->is_active = FALSE; + + /* + * The overlapped result is ready. If the Read...() was successful + * we finally receive the actual result into our buffer. + */ + if (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count, + TRUE)) + return 0; + + /* + * NEEDSWORK: If an external is deleted, the above + * returns an error. I'm not sure that there's anything that + * we can do here other than failing -- the /.git + * link file would be broken anyway. We might try to check + * for that and return a better error message, but I'm not + * sure it is worth it. + */ + + error("GetOverlappedResult failed on '%s' [GLE %ld]", + watch->path.buf, GetLastError()); + return -1; +} + +static void cancel_rdcw_watch(struct one_watch *watch) +{ + DWORD count; + + if (!watch || !watch->is_active) + return; + + /* + * The calls to ReadDirectoryChangesW() and GetOverlappedResult() + * form a "pair" (my term) where we queue an IO and promise to + * hang around and wait for the kernel to give us the result. + * + * If for some reason after we queue the IO, we have to quit + * or otherwise not stick around for the second half, we must + * tell the kernel to abort the IO. This prevents the kernel + * from writing to our buffer and/or signalling our event + * after we free them. + * + * (Ask me how much fun it was to track that one down). + */ + CancelIoEx(watch->hDir, &watch->overlapped); + GetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE); + watch->is_active = FALSE; +} + +/* + * Process filesystem events that happen anywhere (recursively) under the + * root directory. For a normal working directory, this includes + * both version controlled files and the contents of the .git/ directory. + * + * If /.git is a file, then we only see events for the file + * itself. + */ +static int process_worktree_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_worktree; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + struct fsmonitor_batch *batch = NULL; + const char *p = watch->buffer; + + /* + * If the kernel gets more events than will fit in the kernel + * buffer associated with our RDCW handle, it drops them and + * returns a count of zero. + * + * Yes, the call returns WITHOUT error and with length zero. + * + * (The "overflow" case is not ambiguous with the "no data" case + * because we did an INFINITE wait.) + * + * This means we have a gap in coverage. Tell the daemon layer + * to resync. + */ + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_WORKTREE; + } + + /* + * On Windows, `info` contains an "array" of paths that are + * relative to the root of whichever directory handle received + * the event. + */ + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_workdir_relative(path.buf); + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info->Action == FILE_ACTION_REMOVED) || + (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, path.buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, batch, &cookie_list); + batch = NULL; + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_WORKTREE; + +force_shutdown: + fsmonitor_batch__pop(batch); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_SHUTDOWN; +} + +/* + * Process filesystem events that happened anywhere (recursively) under the + * external (such as non-primary worktrees or submodules). + * We only care about cookie files that our client threads created here. + * + * Note that we DO NOT get filesystem events on the external + * itself (it is not inside something that we are watching). In particular, + * we do not get an event if the external is deleted. + */ +static int process_gitdir_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_gitdir; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *p = watch->buffer; + + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_GITDIR; + } + + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_gitdir_relative(path.buf); + + switch (t) { + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_GITDIR: + goto skip_this_path; + + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, NULL, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_GITDIR; } void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data = state->backend_data; + DWORD dwWait; + + state->error_code = 0; + + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + + if (data->watch_gitdir && + start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + + for (;;) { + dwWait = WaitForMultipleObjects(data->nr_listener_handles, + data->hListener, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) { + if (recv_rdcw_watch(data->watch_worktree) == -1) + goto force_error_stop; + if (process_worktree_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) { + if (recv_rdcw_watch(data->watch_gitdir) == -1) + goto force_error_stop; + if (process_gitdir_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN) + goto clean_shutdown; + + error(_("could not read directory changes [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->error_code = -1; + +force_shutdown: + /* + * Tell the IPC thead pool to stop (which completes the await + * in the main thread (which will also signal this thread (if + * we are still alive))). + */ + ipc_server_stop_async(state->ipc_server_data); + +clean_shutdown: + cancel_rdcw_watch(data->watch_worktree); + cancel_rdcw_watch(data->watch_gitdir); } int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->watch_worktree = create_watch(state, + state->path_worktree_watch.buf); + if (!data->watch_worktree) + goto failed; + + if (state->nr_paths_watching > 1) { + data->watch_gitdir = create_watch(state, + state->path_gitdir_watch.buf); + if (!data->watch_gitdir) + goto failed; + } + + data->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown; + data->nr_listener_handles++; + + data->hListener[LISTENER_HAVE_DATA_WORKTREE] = + data->watch_worktree->hEvent; + data->nr_listener_handles++; + + if (data->watch_gitdir) { + data->hListener[LISTENER_HAVE_DATA_GITDIR] = + data->watch_gitdir->hEvent; + data->nr_listener_handles++; + } + + state->backend_data = data; + return 0; + +failed: + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + return -1; } void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + + FREE_AND_NULL(state->backend_data); } From patchwork Thu Jul 1 14:47:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3698FC11F64 for ; Thu, 1 Jul 2021 14:48:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1E8666140C for ; Thu, 1 Jul 2021 14:48:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233163AbhGAOul (ORCPT ); Thu, 1 Jul 2021 10:50:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232953AbhGAOua (ORCPT ); Thu, 1 Jul 2021 10:50:30 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6B8FC061765 for ; Thu, 1 Jul 2021 07:47:58 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id v5so8607380wrt.3 for ; Thu, 01 Jul 2021 07:47:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IdVJWUWBVm2Iwiwmd97pkicNUmd71GC3yTAVdU5z6UM=; b=lmZLh49uXuwCwrWK3isnYQxMG9JMltS1Il/GpgHLReRFYbR75dJYv75dY+Xcv2TlFJ uJbUfDYPG37K3LdkZuzdE+l6kxlE+VJr8xbcphz5D5rzwkEY9NaWybWzBePJX/8A6zQB aPIUbF9ZzixZymLzBhUG3W/JVMCFAairXGr0axd/148ztJZfMJlDCVktrZ+D4QBoM941 RDqhGD31X2VfOdWhNlnOPCTJRdQ8lrVsHPf/dxaJgtfx/SCYB4uYg/mnriJTxSRTV4SW +Hxbth0HoupMUQNGBK3uDLEdP9gAXnUGwvIeqiLQs97PnBrwu7qb/3Yp9TqsRbUr/46z gahQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IdVJWUWBVm2Iwiwmd97pkicNUmd71GC3yTAVdU5z6UM=; b=p/bcJZeu8nLP5nFjlruAJsS/gxeXHtidAz4iE8YYYXDcasJMxUTukxshm+zQSWSYVK pRP5WBbPdZbEe2h+GFQwWrWbiotisjBAcKiC7AU+hZovCPPXDsdfLSw7DKwxqls6931P feOb2mkd5W0x4tYEwIXQX9P+ux3u/zGGwY6zgsZw6CFKKnJN3uaq//j57FBppLfMwG71 8wddsVA5PJ6a7nf31Ri5LqEx6ocB1dGiU2Aya4k4R3gsITpeJcJDpbdn/gRGahFuiUGP KmGEdEmnWFEBN0kJuBeFzG7VhkRkQgqDA5KQotlF6ORI7xXYmoHyMG2GnQI9auucwyfH OP2w== X-Gm-Message-State: AOAM532JS5SFczfftZM40bFtXJ4Lffig59gM+AI0VHdq0x14YIdPAn8r R+4TG+sGKpkhQGO/iFYfsIpxAhV4GXs= X-Google-Smtp-Source: ABdhPJz07kUGEr/tP5AAjbSCg7vlbLImML1TS9Q0AZrM5RRR4dt7mPNCtpXas8P3mXX/ubiiAKQxWA== X-Received: by 2002:a5d:59a6:: with SMTP id p6mr11103333wrr.277.1625150877372; Thu, 01 Jul 2021 07:47:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c10sm201876wmb.40.2021.07.01.07.47.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:57 -0700 (PDT) Message-Id: <175ae9a757e33fb119b4e01c76f851fa71ff4c9b.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:30 +0000 Subject: [PATCH v3 20/34] fsmonitor-fs-listen-macos: add macos header files for FSEvent Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Include MacOS system declarations to allow us to use FSEvent and CoreFoundation APIs. We need GCC and clang versions because of compiler and header file conflicts. While it is quite possible to #include Apple's CoreServices.h when compiling C source code with clang, trying to build it with GCC currently fails with this error: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45, /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope 193 | char bytes[kAuthorizationExternalFormLength]; | ^~~~~ The underlying reason is that GCC (rightfully) objects that an `enum` value such as `kAuthorizationExternalFormLength` is not a constant (because it is not, the preprocessor has no knowledge of it, only the actual C compiler does) and can therefore not be used to define the size of a C array. This is a known problem and tracked in GCC's bug tracker: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 In the meantime, let's not block things and go the slightly ugly route of declaring/defining the FSEvents constants, data structures and functions that we need, so that we can avoid above-mentioned issue. Let's do this _only_ for GCC, though, so that the CI/PR builds (which build both with clang and with GCC) can guarantee that we _are_ using the correct data types. Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c index b91058d1c4f..bec5130d9e1 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -1,3 +1,99 @@ +#if defined(__GNUC__) +/* + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling + * with clang, but not with GCC as of time of writing. + * + * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details. + */ +typedef unsigned int FSEventStreamCreateFlags; +#define kFSEventStreamEventFlagNone 0x00000000 +#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001 +#define kFSEventStreamEventFlagUserDropped 0x00000002 +#define kFSEventStreamEventFlagKernelDropped 0x00000004 +#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008 +#define kFSEventStreamEventFlagHistoryDone 0x00000010 +#define kFSEventStreamEventFlagRootChanged 0x00000020 +#define kFSEventStreamEventFlagMount 0x00000040 +#define kFSEventStreamEventFlagUnmount 0x00000080 +#define kFSEventStreamEventFlagItemCreated 0x00000100 +#define kFSEventStreamEventFlagItemRemoved 0x00000200 +#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400 +#define kFSEventStreamEventFlagItemRenamed 0x00000800 +#define kFSEventStreamEventFlagItemModified 0x00001000 +#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000 +#define kFSEventStreamEventFlagItemChangeOwner 0x00004000 +#define kFSEventStreamEventFlagItemXattrMod 0x00008000 +#define kFSEventStreamEventFlagItemIsFile 0x00010000 +#define kFSEventStreamEventFlagItemIsDir 0x00020000 +#define kFSEventStreamEventFlagItemIsSymlink 0x00040000 +#define kFSEventStreamEventFlagOwnEvent 0x00080000 +#define kFSEventStreamEventFlagItemIsHardlink 0x00100000 +#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000 +#define kFSEventStreamEventFlagItemCloned 0x00400000 + +typedef struct __FSEventStream *FSEventStreamRef; +typedef const FSEventStreamRef ConstFSEventStreamRef; + +typedef unsigned int CFStringEncoding; +#define kCFStringEncodingUTF8 0x08000100 + +typedef const struct __CFString *CFStringRef; +typedef const struct __CFArray *CFArrayRef; +typedef const struct __CFRunLoop *CFRunLoopRef; + +struct FSEventStreamContext { + long long version; + void *cb_data, *retain, *release, *copy_description; +}; + +typedef struct FSEventStreamContext FSEventStreamContext; +typedef unsigned int FSEventStreamEventFlags; +#define kFSEventStreamCreateFlagNoDefer 0x02 +#define kFSEventStreamCreateFlagWatchRoot 0x04 +#define kFSEventStreamCreateFlagFileEvents 0x10 + +typedef unsigned long long FSEventStreamEventId; +#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL + +typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef, + void *context, + __SIZE_TYPE__ num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]); +typedef double CFTimeInterval; +FSEventStreamRef FSEventStreamCreate(void *allocator, + FSEventStreamCallback callback, + FSEventStreamContext *context, + CFArrayRef paths_to_watch, + FSEventStreamEventId since_when, + CFTimeInterval latency, + FSEventStreamCreateFlags flags); +CFStringRef CFStringCreateWithCString(void *allocator, const char *string, + CFStringEncoding encoding); +CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count, + void *callbacks); +void CFRunLoopRun(void); +void CFRunLoopStop(CFRunLoopRef run_loop); +CFRunLoopRef CFRunLoopGetCurrent(void); +extern CFStringRef kCFRunLoopDefaultMode; +void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream, + CFRunLoopRef run_loop, + CFStringRef run_loop_mode); +unsigned char FSEventStreamStart(FSEventStreamRef stream); +void FSEventStreamStop(FSEventStreamRef stream); +void FSEventStreamInvalidate(FSEventStreamRef stream); +void FSEventStreamRelease(FSEventStreamRef stream); +#else +/* + * Let Apple's headers declare `isalnum()` first, before + * Git's headers override it via a constant + */ +#include +#include +#include +#endif + #include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" From patchwork Thu Jul 1 14:47:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A906C11F69 for ; Thu, 1 Jul 2021 14:48:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1F39A61414 for ; Thu, 1 Jul 2021 14:48:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233073AbhGAOup (ORCPT ); Thu, 1 Jul 2021 10:50:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232988AbhGAOub (ORCPT ); Thu, 1 Jul 2021 10:50:31 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4159AC0613DE for ; Thu, 1 Jul 2021 07:47:59 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id p8so8624396wrr.1 for ; Thu, 01 Jul 2021 07:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5PQz+2CvMKmvpMLU6cbmrW4OfbgImdWZKOr+C66kQLA=; b=Y0H30/4kSJ+wst/Gm+L2/liDqJOGPtBKGNSVPPPaTUyLATZNPwNvQiZwzeCHMLp4cp GflLPIPHC33UMijJNpW7IDxX6Ptb5bLhd2BndaZOqWURLDRK1S0dhDcKjCnXWA6C/eJA IhugOXyU7XBGV/p+c6wMi9m6DAsCd23ziYhqfkTt9RwWqldGu/kYj6FyjfTfJh2Alwf/ xBPJkvI81+fSCJcZLZPrl8Psu91pK9cYGhgB98TZILJmDrMyFreQNnxjTrhDhh2lvrg+ 8zdaVlen8C7CXtlyLujr3CutbMzziDyN4qAJZ73Z/dJK0mgCBSbcTR/WZ8gayEvUDPDi NMYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5PQz+2CvMKmvpMLU6cbmrW4OfbgImdWZKOr+C66kQLA=; b=Opr96zQRIFlNGAUxQdFbouRNaOOUK/vgUjWk7K+vXn3xfpJU+tZIA1B0XX9n0uomsW OOxFCNwhU7JDkZ3ofzDI6kuxDNHZsZSO8WVeYGV42x82TisaD9yM1Zhm4E9TrinLA2sa AAOLXMQajtElupzdWweZjsc8BrCzv4KOBaT2bRTBwUm5rD5lGOuyaqqtYmX/ktiigr8t gXt7bb2WffbZwVp3VEE0FbUmvRoKlTlMCupCDGlDL1tEmDieByGwVM9NE4JfYRyhtucX NDHlE5/wz1VuO+LaLWdAzIaJXL+mUCOK08XgD42Ps/pamCbIp6yiUbP90zOikCuGQSOF rV9Q== X-Gm-Message-State: AOAM5321ozv9uGF1+xhgbNPzTn4zBDKIqSoTz7cRVdxOwMAmK1Cl7wch jZ70acImNY+Y2lUFzqJy6sSyYSLrC10= X-Google-Smtp-Source: ABdhPJxvp+MTp5vrCBXqdpDV6vejXeNhlhXX6GusnjJF8526Evc5QZt1OnJNItF6/x25MFOf8utxrQ== X-Received: by 2002:a5d:4e10:: with SMTP id p16mr2126wrt.63.1625150877879; Thu, 01 Jul 2021 07:47:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v15sm9997570wmj.39.2021.07.01.07.47.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:57 -0700 (PDT) Message-Id: <5d12b5d808a4bcdff87fe0c3a30f415081003459.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:31 +0000 Subject: [PATCH v3 21/34] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement file system event listener on MacOS using FSEvent, CoreFoundation, and CoreServices. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 381 +++++++++++++++++++ 1 file changed, 381 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c index bec5130d9e1..02f89de216e 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -97,20 +97,401 @@ void FSEventStreamRelease(FSEventStreamRef stream); #include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +struct fsmonitor_daemon_backend_data +{ + CFStringRef cfsr_worktree_path; + CFStringRef cfsr_gitdir_path; + + CFArrayRef cfar_paths_to_watch; + int nr_paths_watching; + + FSEventStreamRef stream; + + CFRunLoopRef rl; + + enum shutdown_style { + SHUTDOWN_EVENT = 0, + FORCE_SHUTDOWN, + FORCE_ERROR_STOP, + } shutdown_style; + + unsigned int stream_scheduled:1; + unsigned int stream_started:1; +}; + +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) +{ + struct strbuf msg = STRBUF_INIT; + + if (flag & kFSEventStreamEventFlagMustScanSubDirs) + strbuf_addstr(&msg, "MustScanSubDirs|"); + if (flag & kFSEventStreamEventFlagUserDropped) + strbuf_addstr(&msg, "UserDropped|"); + if (flag & kFSEventStreamEventFlagKernelDropped) + strbuf_addstr(&msg, "KernelDropped|"); + if (flag & kFSEventStreamEventFlagEventIdsWrapped) + strbuf_addstr(&msg, "EventIdsWrapped|"); + if (flag & kFSEventStreamEventFlagHistoryDone) + strbuf_addstr(&msg, "HistoryDone|"); + if (flag & kFSEventStreamEventFlagRootChanged) + strbuf_addstr(&msg, "RootChanged|"); + if (flag & kFSEventStreamEventFlagMount) + strbuf_addstr(&msg, "Mount|"); + if (flag & kFSEventStreamEventFlagUnmount) + strbuf_addstr(&msg, "Unmount|"); + if (flag & kFSEventStreamEventFlagItemChangeOwner) + strbuf_addstr(&msg, "ItemChangeOwner|"); + if (flag & kFSEventStreamEventFlagItemCreated) + strbuf_addstr(&msg, "ItemCreated|"); + if (flag & kFSEventStreamEventFlagItemFinderInfoMod) + strbuf_addstr(&msg, "ItemFinderInfoMod|"); + if (flag & kFSEventStreamEventFlagItemInodeMetaMod) + strbuf_addstr(&msg, "ItemInodeMetaMod|"); + if (flag & kFSEventStreamEventFlagItemIsDir) + strbuf_addstr(&msg, "ItemIsDir|"); + if (flag & kFSEventStreamEventFlagItemIsFile) + strbuf_addstr(&msg, "ItemIsFile|"); + if (flag & kFSEventStreamEventFlagItemIsHardlink) + strbuf_addstr(&msg, "ItemIsHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsLastHardlink) + strbuf_addstr(&msg, "ItemIsLastHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsSymlink) + strbuf_addstr(&msg, "ItemIsSymlink|"); + if (flag & kFSEventStreamEventFlagItemModified) + strbuf_addstr(&msg, "ItemModified|"); + if (flag & kFSEventStreamEventFlagItemRemoved) + strbuf_addstr(&msg, "ItemRemoved|"); + if (flag & kFSEventStreamEventFlagItemRenamed) + strbuf_addstr(&msg, "ItemRenamed|"); + if (flag & kFSEventStreamEventFlagItemXattrMod) + strbuf_addstr(&msg, "ItemXattrMod|"); + if (flag & kFSEventStreamEventFlagOwnEvent) + strbuf_addstr(&msg, "OwnEvent|"); + if (flag & kFSEventStreamEventFlagItemCloned) + strbuf_addstr(&msg, "ItemCloned|"); + + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + path, flag, msg.buf); + + strbuf_release(&msg); +} + +static int ef_is_root_delete(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRemoved); +} + +static int ef_is_root_renamed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRenamed); +} + +static int ef_is_dropped(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagKernelDropped || + ef & kFSEventStreamEventFlagUserDropped); +} + +static void fsevent_callback(ConstFSEventStreamRef streamRef, + void *ctx, + size_t num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]) +{ + struct fsmonitor_daemon_state *state = ctx; + struct fsmonitor_daemon_backend_data *data = state->backend_data; + char **paths = (char **)event_paths; + struct fsmonitor_batch *batch = NULL; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *path_k; + const char *slash; + int k; + struct strbuf tmp = STRBUF_INIT; + + /* + * Build a list of all filesystem changes into a private/local + * list and without holding any locks. + */ + for (k = 0; k < num_of_events; k++) { + /* + * On Mac, we receive an array of absolute paths. + */ + path_k = paths[k]; + + /* + * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. + * Please don't log them to Trace2. + * + * trace_printf_key(&trace_fsmonitor, "Path: '%s'", path_k); + */ + + /* + * If event[k] is marked as dropped, we assume that we have + * lost sync with the filesystem and should flush our cached + * data. We need to: + * + * [1] Abort/wake any client threads waiting for a cookie and + * flush the cached state data (the current token), and + * create a new token. + * + * [2] Discard the batch that we were locally building (since + * they are conceptually relative to the just flushed + * token). + */ + if (ef_is_dropped(event_flags[k])) { + /* + * see also kFSEventStreamEventFlagMustScanSubDirs + */ + trace_printf_key(&trace_fsmonitor, "event: dropped"); + + fsmonitor_force_resync(state); + fsmonitor_batch__pop(batch); + string_list_clear(&cookie_list, 0); + + /* + * We assume that any events that we received + * in this callback after this dropped event + * may still be valid, so we continue rather + * than break. (And just in case there is a + * delete of ".git" hiding in there.) + */ + continue; + } + + switch (fsmonitor_classify_path_absolute(state, path_k)) { + + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within .git or gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path_k); + string_list_append(&cookie_list, + slash ? slash + 1 : path_k); + break; + + case IS_INSIDE_DOT_GIT: + case IS_INSIDE_GITDIR: + /* ignore all other paths inside of .git or gitdir */ + break; + + case IS_DOT_GIT: + case IS_GITDIR: + /* + * If .git directory is deleted or renamed away, + * we have to quit. + */ + if (ef_is_root_delete(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "event: gitdir removed"); + goto force_shutdown; + } + if (ef_is_root_renamed(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "event: gitdir renamed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* try to queue normal pathnames */ + + if (trace_pass_fl(&trace_fsmonitor)) + log_flags_set(path_k, event_flags[k]); + + /* + * Because of the implicit "binning" (the + * kernel calls us at a given frequency) and + * de-duping (the kernel is free to combine + * multiple events for a given pathname), an + * individual fsevent could be marked as both + * a file and directory. Add it to the queue + * with both spellings so that the client will + * know how much to invalidate/refresh. + */ + + if (event_flags[k] & kFSEventStreamEventFlagItemIsFile) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, rel); + } + + if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + strbuf_reset(&tmp); + strbuf_addstr(&tmp, rel); + strbuf_addch(&tmp, '/'); + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, tmp.buf); + } + + break; + + case IS_OUTSIDE_CONE: + default: + trace_printf_key(&trace_fsmonitor, + "ignoring '%s'", path_k); + break; + } + } + + fsmonitor_publish(state, batch, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&tmp); + return; + +force_shutdown: + fsmonitor_batch__pop(batch); + string_list_clear(&cookie_list, 0); + + data->shutdown_style = FORCE_SHUTDOWN; + CFRunLoopStop(data->rl); + strbuf_release(&tmp); + return; +} + +/* + * NEEDSWORK: Investigate the proper value for the `latency` argument + * in the call to `FSEventStreamCreate()`. I'm not sure that this + * needs to be a config setting or just something that we tune after + * some testing. + * + * With a latency of 0.1, I was seeing lots of dropped events during + * the "touch 100000" files test within t/perf/p7519, but with a + * latency of 0.001 I did not see any dropped events. So the + * "correct" value may be somewhere in between. + * + * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate + */ int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { + FSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer | + kFSEventStreamCreateFlagWatchRoot | + kFSEventStreamCreateFlagFileEvents; + FSEventStreamContext ctx = { + 0, + state, + NULL, + NULL, + NULL + }; + struct fsmonitor_daemon_backend_data *data; + const void *dir_array[2]; + + CALLOC_ARRAY(data, 1); + state->backend_data = data; + + data->cfsr_worktree_path = CFStringCreateWithCString( + NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_worktree_path; + + if (state->nr_paths_watching > 1) { + data->cfsr_gitdir_path = CFStringCreateWithCString( + NULL, state->path_gitdir_watch.buf, + kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path; + } + + data->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array, + data->nr_paths_watching, + NULL); + data->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx, + data->cfar_paths_to_watch, + kFSEventStreamEventIdSinceNow, + 0.001, flags); + if (data->stream == NULL) + goto failed; + + /* + * `data->rl` needs to be set inside the listener thread. + */ + + return 0; + +failed: + error("Unable to create FSEventStream."); + + FREE_AND_NULL(state->backend_data); return -1; } void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + if (data->stream) { + if (data->stream_started) + FSEventStreamStop(data->stream); + if (data->stream_scheduled) + FSEventStreamInvalidate(data->stream); + FSEventStreamRelease(data->stream); + } + + FREE_AND_NULL(state->backend_data); } void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + data->shutdown_style = SHUTDOWN_EVENT; + + CFRunLoopStop(data->rl); } void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + + data->rl = CFRunLoopGetCurrent(); + + FSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode); + data->stream_scheduled = 1; + + if (!FSEventStreamStart(data->stream)) { + error("Failed to start the FSEventStream"); + goto force_error_stop_without_loop; + } + data->stream_started = 1; + + CFRunLoopRun(); + + switch (data->shutdown_style) { + case FORCE_ERROR_STOP: + state->error_code = -1; + /* fall thru */ + case FORCE_SHUTDOWN: + ipc_server_stop_async(state->ipc_server_data); + /* fall thru */ + case SHUTDOWN_EVENT: + default: + break; + } + return; + +force_error_stop_without_loop: + state->error_code = -1; + ipc_server_stop_async(state->ipc_server_data); + return; } From patchwork Thu Jul 1 14:47:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A804C11F67 for ; Thu, 1 Jul 2021 14:48:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 82D3561403 for ; Thu, 1 Jul 2021 14:48:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233193AbhGAOuo (ORCPT ); Thu, 1 Jul 2021 10:50:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232983AbhGAOub (ORCPT ); Thu, 1 Jul 2021 10:50:31 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6A0DC0613E0 for ; Thu, 1 Jul 2021 07:47:59 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id t15so5234407wry.11 for ; Thu, 01 Jul 2021 07:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Jq0McXzt7Q/ckZKHUxaVjFz6gBrnQ6VtHhxaJ7TCsk0=; b=Q08ZRPkxsFtXEhITNEvlCcMRamcPZwjTePD4LQWXa4K7sF2NrYd4MtwA/TLH2YbCZP QBrbAFoIXdqLQpgJdKhJ6S/FTDD1elox7FLsY3cF0E8qZGADO9uqvAcMLFMknxTEyQT3 8SEWWBHhf5t4hFaDIXOab8EfXVrBbiVbpKAvdyh/WacD8uIKHnSKAZ3iBAZ9BJysuMLe 0pgF7AxvWk4PQW1hXQOu02vcps07MVBTLEZNBM60pYbtIQKrSOMbzHraJcK7Hu4D1UlZ vhHbHLLHBN+D+IwVWMD+nAMV6qWOVvTJoIIKKHHG1HtxM93A54E2jSUx3dP+qBP3BNDs 1CpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Jq0McXzt7Q/ckZKHUxaVjFz6gBrnQ6VtHhxaJ7TCsk0=; b=pjG/d+GjAdrkFzy6SP3moavx8ClnIB6M6GVaE949RSDxZQqnltBrgQudpKyHztKAGw vdyDMs+Sl5SDeeYXTnJQVveMY4LHK4OEjKMP/qALvc+R1q70/CRaYNhI54ADkIUgakqe kYllpZ7yCoWc5zzSZEKhDbo5BTu+MDK1vW1Lf+zrvNgCXUJc86NgBk8XnpI0BCFeRTvD tWLko8oK+CcY7qeku91SeuNzdwV0CRu05ZbkDyc5kSYaxipBH5oMdvZIpu5/qOEKEZW2 Jzm6yGThGGCABa1CVb+COZhb4AGkPfx+WDhQpFAWQiXphlKaOVn+5Mw7S0yhHbZ2tGdq v+Iw== X-Gm-Message-State: AOAM531TF84GivpYNqP2Gm0O6wM77cKVLsOu+P+STfW+sc+RNn+dALOc lEwioK+saS4VsWfVOoichVJPTQAER5g= X-Google-Smtp-Source: ABdhPJwbWJAdktlxILgzbhbwQyhBsSE5WCehnhPCmQH7a9dlNkwrKqck3xZm6y+EImKiSdSkN/QCUQ== X-Received: by 2002:adf:ea8e:: with SMTP id s14mr50555wrm.38.1625150878482; Thu, 01 Jul 2021 07:47:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l9sm160885wrp.14.2021.07.01.07.47.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:58 -0700 (PDT) Message-Id: <39df123143bd35fcce72d589fb8761eda34e212e.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:32 +0000 Subject: [PATCH v3 22/34] fsmonitor--daemon: implement handle_client callback Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to respond to IPC requests from client Git processes and respond with a list of modified pathnames relative to the provided token. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 312 +++++++++++++++++++++++++++++++++++- 1 file changed, 310 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index ea3a52d34e3..7a7fef681fe 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,6 +7,7 @@ #include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" +#include "pkt-line.h" static const char * const builtin_fsmonitor__daemon_usage[] = { N_("git fsmonitor--daemon start []"), @@ -355,6 +356,311 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) pthread_mutex_unlock(&state->main_lock); } +/* + * Format an opaque token string to send to the client. + */ +static void with_lock__format_response_token( + struct strbuf *response_token, + const struct strbuf *response_token_id, + const struct fsmonitor_batch *batch) +{ + /* assert current thread holding state->main_lock */ + + strbuf_reset(response_token); + strbuf_addf(response_token, "builtin:%s:%"PRIu64, + response_token_id->buf, batch->batch_seq_nr); +} + +/* + * Parse an opaque token from the client. + * Returns -1 on error. + */ +static int fsmonitor_parse_client_token(const char *buf_token, + struct strbuf *requested_token_id, + uint64_t *seq_nr) +{ + const char *p; + char *p_end; + + strbuf_reset(requested_token_id); + *seq_nr = 0; + + if (!skip_prefix(buf_token, "builtin:", &p)) + return -1; + + while (*p && *p != ':') + strbuf_addch(requested_token_id, *p++); + if (!*p++) + return -1; + + *seq_nr = (uint64_t)strtoumax(p, &p_end, 10); + if (*p_end) + return -1; + + return 0; +} + +KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal); + +static int do_handle_client(struct fsmonitor_daemon_state *state, + const char *command, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + struct fsmonitor_token_data *token_data = NULL; + struct strbuf response_token = STRBUF_INIT; + struct strbuf requested_token_id = STRBUF_INIT; + struct strbuf payload = STRBUF_INIT; + uint64_t requested_oldest_seq_nr = 0; + uint64_t total_response_len = 0; + const char *p; + const struct fsmonitor_batch *batch_head; + const struct fsmonitor_batch *batch; + intmax_t count = 0, duplicates = 0; + kh_str_t *shown; + int hash_ret; + int do_trivial = 0; + int do_flush = 0; + + /* + * We expect `command` to be of the form: + * + * := quit NUL + * | flush NUL + * | NUL + * | NUL + */ + + if (!strcmp(command, "quit")) { + /* + * A client has requested over the socket/pipe that the + * daemon shutdown. + * + * Tell the IPC thread pool to shutdown (which completes + * the await in the main thread (which can stop the + * fsmonitor listener thread)). + * + * There is no reply to the client. + */ + return SIMPLE_IPC_QUIT; + + } else if (!strcmp(command, "flush")) { + /* + * Flush all of our cached data and generate a new token + * just like if we lost sync with the filesystem. + * + * Then send a trivial response using the new token. + */ + do_flush = 1; + do_trivial = 1; + + } else if (!skip_prefix(command, "builtin:", &p)) { + /* assume V1 timestamp or garbage */ + + char *p_end; + + strtoumax(command, &p_end, 10); + trace_printf_key(&trace_fsmonitor, + ((*p_end) ? + "fsmonitor: invalid command line '%s'" : + "fsmonitor: unsupported V1 protocol '%s'"), + command); + do_trivial = 1; + + } else { + /* We have "builtin:*" */ + if (fsmonitor_parse_client_token(command, &requested_token_id, + &requested_oldest_seq_nr)) { + trace_printf_key(&trace_fsmonitor, + "fsmonitor: invalid V2 protocol token '%s'", + command); + do_trivial = 1; + + } else { + /* + * We have a V2 valid token: + * "builtin::" + */ + } + } + + pthread_mutex_lock(&state->main_lock); + + if (!state->current_token_data) + BUG("fsmonitor state does not have a current token"); + + if (do_flush) + with_lock__do_force_resync(state); + + /* + * We mark the current head of the batch list as "pinned" so + * that the listener thread will treat this item as read-only + * (and prevent any more paths from being added to it) from + * now on. + */ + token_data = state->current_token_data; + batch_head = token_data->batch_head; + ((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL); + + /* + * FSMonitor Protocol V2 requires that we send a response header + * with a "new current token" and then all of the paths that changed + * since the "requested token". We send the seq_nr of the just-pinned + * head batch so that future requests from a client will be relative + * to it. + */ + with_lock__format_response_token(&response_token, + &token_data->token_id, batch_head); + + reply(reply_data, response_token.buf, response_token.len + 1); + total_response_len += response_token.len + 1; + + trace2_data_string("fsmonitor", the_repository, "response/token", + response_token.buf); + trace_printf_key(&trace_fsmonitor, "response token: %s", + response_token.buf); + + if (!do_trivial) { + if (strcmp(requested_token_id.buf, token_data->token_id.buf)) { + /* + * The client last spoke to a different daemon + * instance -OR- the daemon had to resync with + * the filesystem (and lost events), so reject. + */ + trace2_data_string("fsmonitor", the_repository, + "response/token", "different"); + do_trivial = 1; + + } else if (requested_oldest_seq_nr < + token_data->batch_tail->batch_seq_nr) { + /* + * The client wants older events than we have for + * this token_id. This means that the end of our + * batch list was truncated and we cannot give the + * client a complete snapshot relative to their + * request. + */ + trace_printf_key(&trace_fsmonitor, + "client requested truncated data"); + do_trivial = 1; + } + } + + if (do_trivial) { + pthread_mutex_unlock(&state->main_lock); + + reply(reply_data, "/", 2); + + trace2_data_intmax("fsmonitor", the_repository, + "response/trivial", 1); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + return 0; + } + + /* + * We're going to hold onto a pointer to the current + * token-data while we walk the list of batches of files. + * During this time, we will NOT be under the lock. + * So we ref-count it. + * + * This allows the listener thread to continue prepending + * new batches of items to the token-data (which we'll ignore). + * + * AND it allows the listener thread to do a token-reset + * (and install a new `current_token_data`). + */ + token_data->client_ref_count++; + + pthread_mutex_unlock(&state->main_lock); + + /* + * The client request is relative to the token that they sent, + * so walk the batch list backwards from the current head back + * to the batch (sequence number) they named. + * + * We use khash to de-dup the list of pathnames. + * + * NEEDSWORK: each batch contains a list of interned strings, + * so we only need to do pointer comparisons here to build the + * hash table. Currently, we're still comparing the string + * values. + */ + shown = kh_init_str(); + for (batch = batch_head; + batch && batch->batch_seq_nr > requested_oldest_seq_nr; + batch = batch->next) { + size_t k; + + for (k = 0; k < batch->nr; k++) { + const char *s = batch->interned_paths[k]; + size_t s_len; + + if (kh_get_str(shown, s) != kh_end(shown)) + duplicates++; + else { + kh_put_str(shown, s, &hash_ret); + + trace_printf_key(&trace_fsmonitor, + "send[%"PRIuMAX"]: %s", + count, s); + + /* Each path gets written with a trailing NUL */ + s_len = strlen(s) + 1; + + if (payload.len + s_len >= + LARGE_PACKET_DATA_MAX) { + reply(reply_data, payload.buf, + payload.len); + total_response_len += payload.len; + strbuf_reset(&payload); + } + + strbuf_add(&payload, s, s_len); + count++; + } + } + } + + if (payload.len) { + reply(reply_data, payload.buf, payload.len); + total_response_len += payload.len; + } + + kh_release_str(shown); + + pthread_mutex_lock(&state->main_lock); + + if (token_data->client_ref_count > 0) + token_data->client_ref_count--; + + if (token_data->client_ref_count == 0) { + if (token_data != state->current_token_data) { + /* + * The listener thread did a token-reset while we were + * walking the batch list. Therefore, this token is + * stale and can be discarded completely. If we are + * the last reader thread using this token, we own + * that work. + */ + fsmonitor_free_token_data(token_data); + } + } + + pthread_mutex_unlock(&state->main_lock); + + trace2_data_intmax("fsmonitor", the_repository, "response/length", total_response_len); + trace2_data_intmax("fsmonitor", the_repository, "response/count/files", count); + trace2_data_intmax("fsmonitor", the_repository, "response/count/duplicates", duplicates); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + strbuf_release(&payload); + + return 0; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -362,7 +668,7 @@ static int handle_client(void *data, ipc_server_reply_cb *reply, struct ipc_server_reply_data *reply_data) { - /* struct fsmonitor_daemon_state *state = data; */ + struct fsmonitor_daemon_state *state = data; int result; /* @@ -373,10 +679,12 @@ static int handle_client(void *data, if (command_len != strlen(command)) BUG("FSMonitor assumes text messages"); + trace_printf_key(&trace_fsmonitor, "requested token: %s", command); + trace2_region_enter("fsmonitor", "handle_client", the_repository); trace2_data_string("fsmonitor", the_repository, "request", command); - result = 0; /* TODO Do something here. */ + result = do_handle_client(state, command, reply, reply_data); trace2_region_leave("fsmonitor", "handle_client", the_repository); From patchwork Thu Jul 1 14:47:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 373F6C11F67 for ; Thu, 1 Jul 2021 14:48:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2361161414 for ; Thu, 1 Jul 2021 14:48:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232907AbhGAOun (ORCPT ); Thu, 1 Jul 2021 10:50:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232832AbhGAOub (ORCPT ); Thu, 1 Jul 2021 10:50:31 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61E53C0613DC for ; Thu, 1 Jul 2021 07:48:00 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id w13so4690591wmc.3 for ; Thu, 01 Jul 2021 07:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=PCs4gtWKEQfwwXjZSPa+bpZ08P6Kqw0LvliYJMx3Soo=; b=VrIRgwzP6GHjCN0YWGDHmo2If392Y+X183vixaZ3f+imkCbPjq5tckdyp+6I3UZK68 D93O8S8llFXK+1GX/xgFf+M/zGAVxwq9oafYi6GsZXGPEFxwzDEgeZvqMuSJrmnhDZeV 4I+NVA/ppNVOkTunpFmA1b1fOjK+TNnR2OAXvwVj75yjpQfurq4jiRj0lSTbdz7MdSWn QUh0pt78hZd748d1Dd2+rEGl205hDiJDBipiUyw5urAcqM53WIQR6jNCxz8/GFR5seKs vAL//tuuqlaS4Io9VfbMyfa0AfIHsKImYqtWo/LY6ZCeCaN9Fb5xkHoP/VJj9gK8h+ap Ldmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=PCs4gtWKEQfwwXjZSPa+bpZ08P6Kqw0LvliYJMx3Soo=; b=tws5mkiwuK2NRUscyLz46wCYEZtCblXkDRm0kFlm8CICxEvGOLS2bLO19EMGD6nPuo 5rzFUXPu2cKZc1w7eX55GoK8LfVO8eroaPhMbZMK42QgES1xEizX1NGhukaz/BFC/mt6 8RIqwI5WuKMx5ELG3XlsYOflYn3ozLACaRV4lIOmNjSgQnASQA1f9cwfr1eAXoTTLqSC cQFRII3yCLdBBXlWW9OPOcCFoz1jC2p4Yy5EhxKOI68nUHA0SY3wrBCftmm5SahYGc6D d6XyvpLyJ1YuwxlgA/6FM1sW79YTFH+EXpsONL7qWKFYd7BQKLuzGBCYNN4lgnTjaTdn mzfQ== X-Gm-Message-State: AOAM530zgZAd9DxcQVSvRbK2Q1WjNZ2AUZ0LjJcqvEjUjtJ6SHMzqNMv zx7v1YE8AbeshrQxta0b64OUQ2z4fr0= X-Google-Smtp-Source: ABdhPJzfkrm692t5yKea187buOsgZ19VPFRlsFtfcE46shFUdPUTX66/42fad38kjJNRBAb1GOMepg== X-Received: by 2002:a1c:771a:: with SMTP id t26mr11345577wmi.36.1625150879003; Thu, 01 Jul 2021 07:47:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g10sm230548wmh.33.2021.07.01.07.47.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:58 -0700 (PDT) Message-Id: <3cf8f3cd7717dad69262ae884a89c1a6ca0ce5fc.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:33 +0000 Subject: [PATCH v3 23/34] t/helper/test-touch: add helper to touch a series of files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create `test-tool touch` that can update a series of files using either a pattern given on the command line or a list of files read from stdin. This will be used in a later commit to speed up p7519 which needs to generate/update many thousands of files. Signed-off-by: Jeff Hostetler --- Makefile | 1 + t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/helper/test-touch.c | 126 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 129 insertions(+) create mode 100644 t/helper/test-touch.c diff --git a/Makefile b/Makefile index a2a6e1f20f6..c07cfb75532 100644 --- a/Makefile +++ b/Makefile @@ -757,6 +757,7 @@ TEST_BUILTINS_OBJS += test-string-list.o TEST_BUILTINS_OBJS += test-submodule-config.o TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o TEST_BUILTINS_OBJS += test-subprocess.o +TEST_BUILTINS_OBJS += test-touch.o TEST_BUILTINS_OBJS += test-trace2.o TEST_BUILTINS_OBJS += test-urlmatch-normalization.o TEST_BUILTINS_OBJS += test-userdiff.o diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index af879e4a5d7..1ad8d5fbd82 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -73,6 +73,7 @@ static struct test_cmd cmds[] = { { "submodule-config", cmd__submodule_config }, { "submodule-nested-repo-config", cmd__submodule_nested_repo_config }, { "subprocess", cmd__subprocess }, + { "touch", cmd__touch }, { "trace2", cmd__trace2 }, { "userdiff", cmd__userdiff }, { "urlmatch-normalization", cmd__urlmatch_normalization }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 6c5134b46d9..58fde0a62e5 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -63,6 +63,7 @@ int cmd__string_list(int argc, const char **argv); int cmd__submodule_config(int argc, const char **argv); int cmd__submodule_nested_repo_config(int argc, const char **argv); int cmd__subprocess(int argc, const char **argv); +int cmd__touch(int argc, const char **argv); int cmd__trace2(int argc, const char **argv); int cmd__userdiff(int argc, const char **argv); int cmd__urlmatch_normalization(int argc, const char **argv); diff --git a/t/helper/test-touch.c b/t/helper/test-touch.c new file mode 100644 index 00000000000..e9b3b754f1f --- /dev/null +++ b/t/helper/test-touch.c @@ -0,0 +1,126 @@ +/* + * test-touch.c: variation on /usr/bin/touch to speed up tests + * with a large number of files (primarily on Windows where child + * process are very, very expensive). + */ + +#include "test-tool.h" +#include "cache.h" +#include "parse-options.h" + +char *seq_pattern; +int seq_start = 1; +int seq_count = 1; + +static int do_touch_one(const char *path) +{ + int fd; + + if (!utime(path, NULL)) + return 0; + + if (errno != ENOENT) { + warning_errno("could not touch '%s'", path); + return 0; + } + + fd = open(path, O_RDWR | O_CREAT, 0644); + if (fd == -1) { + warning_errno("could not create '%s'", path); + return 0; + } + close(fd); + + return 0; +} + +/* + * Touch a series of files. We assume that any required subdirs + * already exist. This function allows us to replace the following + * test script fragment: + * + * for i in $(test_seq 1 10000); do touch 10000_files/$i; done && + * + * with a single process: + * + * test-tool touch sequence --pattern="10000_files/%d" --start=1 --count=10000 + * + * which is much faster on Windows. + */ +static int do_sequence(void) +{ + struct strbuf buf = STRBUF_INIT; + int k; + + for (k = seq_start; k < seq_start + seq_count; k++) { + strbuf_reset(&buf); + strbuf_addf(&buf, seq_pattern, k); + + if (do_touch_one(buf.buf)) + return 1; + } + + return 0; +} + +/* + * Read a list of pathnames from stdin and touch them. We assume that + * any required subdirs already exist. + */ +static int do_stdin(void) +{ + struct strbuf buf = STRBUF_INIT; + + while (strbuf_getline(&buf, stdin) != EOF && buf.len) + if (do_touch_one(buf.buf)) + return 1; + + return 0; +} + +int cmd__touch(int argc, const char **argv) +{ + const char *touch_usage[] = { + N_("test-tool touch sequence "), + N_("test-tool touch stdin"), + NULL, + }; + + struct option touch_options[] = { + OPT_GROUP(N_("sequence")), + OPT_STRING(0, "pattern", &seq_pattern, N_("format"), + N_("sequence pathname pattern")), + OPT_INTEGER(0, "start", &seq_start, + N_("sequence starting value")), + OPT_INTEGER(0, "count", &seq_count, + N_("sequence count")), + OPT_END() + }; + + const char *subcmd; + + if (argc < 2) + usage_with_options(touch_usage, touch_options); + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(touch_usage, touch_options); + + subcmd = argv[1]; + argv--; + argc++; + + argc = parse_options(argc, argv, NULL, touch_options, touch_usage, 0); + + if (!strcmp(subcmd, "sequence")) { + if (!seq_pattern || !strstr(seq_pattern, "%d")) + die("invalid sequence pattern"); + if (seq_count < 1) + die("invalid sequence count: %d", seq_count); + return !!do_sequence(); + } + + if (!strcmp(subcmd, "stdin")) { + return !!do_stdin(); + } + + die("Unhandled subcommand: '%s'", subcmd); +} From patchwork Thu Jul 1 14:47:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67E79C11F64 for ; Thu, 1 Jul 2021 14:48:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5431A61403 for ; Thu, 1 Jul 2021 14:48:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233008AbhGAOun (ORCPT ); Thu, 1 Jul 2021 10:50:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233002AbhGAOuc (ORCPT ); Thu, 1 Jul 2021 10:50:32 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7E0AC061762 for ; Thu, 1 Jul 2021 07:48:00 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id g7so8563333wri.7 for ; Thu, 01 Jul 2021 07:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7qmpK0v+kMbY8lqE5mMlTf3cnnNqYbVpa6kPQCWVcF8=; b=goHvE+zwCO8wMxC/gXVhWvNBnzLdsfZZfwTdclzQHQZ41hkvSjqsinDICMIWXdp8Nv FAAHNppZ1VAkmLKTUzQnvvQzzPC8iLhtpqGa8YPjjZKcQgWk/bzDsPJusrblk6GgjQIZ X2DybOsVe8wPf1qfRYv+l4aDw0KfaTdCyshtsKc1KcjrcNjF+bv/EpBa7HX2CQNvgrLj b+TwocJD9Li1B4SCDmrXzFUMvHye3lJ9T9aLLED7AAyPe0IqD+dN+NjBA0/Bm1dxPiNq WKjAvhWt5ZHRpzYqiCFZxzJ/OnhJcD7KgFetZLQmgED03pol1XWUFVmov50xQbuFoOFW /alQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7qmpK0v+kMbY8lqE5mMlTf3cnnNqYbVpa6kPQCWVcF8=; b=lad21nJHZvF9OtI01L3Nsm1t/Mdh7RxSqHaOqYOzWg9xTWnj9uIIdFdyz2kS/npe79 m0NdEaC154ASDMmA9svTNjHV8dkhkiY3ta4K3MT1JEOTfHh8PSNNT5KuSZeaZQNIXBdK 7zqVROzTBGxU7zPs3CZsAwLGgTVjEqly+QOSsVf3OFcD52BK4HDawx4P4CkeBA9V2z/g 1hqzCdAnWScwIMWj8YjBPMao4d9HM6+eIAA6efABAA+Cao0X12ZjKXzPEs210ASPg9fr avLSDIP9SELPtq8NS13lnhnj4t5zsDv9+Gn2UNco3DFGpB+qqQeKj1mOHPYyiKMbTaeB GEGA== X-Gm-Message-State: AOAM532SXM4x3w5pmQIE06cJsF9cA0JTlOt+QQqLfpCFRVJKMMyJnAqB PnEHws9HWj0/nrk7ZyUUxUSJJfVb+Yw= X-Google-Smtp-Source: ABdhPJyZb66uAd75G4HIyLWHJWLGN4OT5YYGGb6Bk2YkCEqJ1LdfwL19k/QpvcAE94uMrNf5pWHF/Q== X-Received: by 2002:a5d:5388:: with SMTP id d8mr45529386wrv.423.1625150879584; Thu, 01 Jul 2021 07:47:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o20sm9719761wms.3.2021.07.01.07.47.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:34 +0000 Subject: [PATCH v3 24/34] t/perf/p7519: speed up test using "test-tool touch" Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Change p7519 to use a single "test-tool touch" command to update the mtime on a series of (thousands) files instead of invoking thousands of commands to update a single file. This is primarily for Windows where process creation is so very slow and reduces the test run time by minutes. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 5eb5044a103..f74e6014a0a 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -119,10 +119,11 @@ test_expect_success "one time repo setup" ' fi && mkdir 1_file 10_files 100_files 1000_files 10000_files && - for i in $(test_seq 1 10); do touch 10_files/$i; done && - for i in $(test_seq 1 100); do touch 100_files/$i; done && - for i in $(test_seq 1 1000); do touch 1000_files/$i; done && - for i in $(test_seq 1 10000); do touch 10000_files/$i; done && + test-tool touch sequence --pattern="10_files/%d" --start=1 --count=10 && + test-tool touch sequence --pattern="100_files/%d" --start=1 --count=100 && + test-tool touch sequence --pattern="1000_files/%d" --start=1 --count=1000 && + test-tool touch sequence --pattern="10000_files/%d" --start=1 --count=10000 && + git add 1_file 10_files 100_files 1000_files 10000_files && git commit -qm "Add files" && @@ -200,15 +201,12 @@ test_fsmonitor_suite() { # Update the mtimes on upto 100k files to make status think # that they are dirty. For simplicity, omit any files with # LFs (i.e. anything that ls-files thinks it needs to dquote). - # Then fully backslash-quote the paths to capture any - # whitespace so that they pass thru xargs properly. # test_perf_w_drop_caches "status (dirty) ($DESC)" ' git ls-files | \ head -100000 | \ grep -v \" | \ - sed '\''s/\(.\)/\\\1/g'\'' | \ - xargs test-tool chmtime -300 && + test-tool touch stdin && git status ' From patchwork Thu Jul 1 14:47:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9474C11F64 for ; Thu, 1 Jul 2021 14:48:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C32E661414 for ; Thu, 1 Jul 2021 14:48:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233110AbhGAOuq (ORCPT ); Thu, 1 Jul 2021 10:50:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233045AbhGAOuf (ORCPT ); Thu, 1 Jul 2021 10:50:35 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EFF9C0613DB for ; Thu, 1 Jul 2021 07:48:01 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id v5so8607558wrt.3 for ; Thu, 01 Jul 2021 07:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=g237GL0OfwbXvqxmbpOXauDHEgil7enRJ/PKcAc4nOc=; b=mZdWxdjSbllHDphsmXNYE17R236n/eofGPJP/f1wyV94oqAKb6/TQKJr4I0QHNUKVe SRISQiFi7heWk1aEg+rR+4+e5SgrQIxtMvpT21kxIPKQBtE0hT4HbWVE8XzdsdRLkLJD KcOjD5BKxBEY8+xQe3kZaAnIEMWhU0I/ctztHyTn9eDdX9TQ/tCxzLdvadNCnwnIO2ac ylX4wno3L8fmwL7RQfexEUW76jsoi9WZOmvbN2VxSWuTTlYl4TPFoyBhV3TbfNlXVXEQ O98jMVrr1QT1zy89d50ZFQfz2pn82pVRQOXhnOAbGNl0XXX7XiR4NFz16C4o0ywKEIOk kdNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=g237GL0OfwbXvqxmbpOXauDHEgil7enRJ/PKcAc4nOc=; b=hQalvMCosHBWDxFK/2jjYFclrLmGIN7wnLdXPEWJ/xr3yVfSlXvPsha6gD3MDK/AU+ ruNBrShaTEh+ALsFgMc7TRjgOLMt6XQHAB+4qGJwYhpAqELS1GHKu3hikJA8wmmpQxSm ELAbXzlCC45xjBYgUbfpo2WuMV9p7iWTidRhL+Ldus8fZbkZiu99M3+WY0kZk4mUNjlj rUD/Ep54hkUa3C7/EEFmlh2v5Yc39AV1G9yMriG4uOKf2J4c28uHzdXAhzOvPxt1IbZr qAf+h+LtcpIKNvyhyjp34HJ728LXGlZons4lUHboz2RO64lUbF/+fx0TlpqHfM5S06gX Woxg== X-Gm-Message-State: AOAM532GSmxR/NkVqA5qX5Aafy7YWj2fOqfYSDkWF0oiAHHOi6XuxWsV wcmiFUrAZR+zzSmcwaLUVkytNHmUuOs= X-Google-Smtp-Source: ABdhPJy3d5+oqzH1zCxe0J2YGlduSmVRk7BDvcP6F+RwkMoh2v8sQ0IC3EvC8guS6bbTVdTOEpvIIQ== X-Received: by 2002:a05:6000:1361:: with SMTP id q1mr23492789wrz.179.1625150880096; Thu, 01 Jul 2021 07:48:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t17sm201663wmi.47.2021.07.01.07.47.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:47:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:35 +0000 Subject: [PATCH v3 25/34] t/perf: avoid copying builtin fsmonitor files into test repo Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Do not try to copy a fsmonitor--daemon socket from the current development directory into the test trash directory. When we run the perf suite without an explicit source repo set, we copy of the current $GIT_DIR into the test trash directory. Unix domain sockets cannot be copied in that manner, so the test setup fails. Additionally, omit any other fsmonitor--daemon temp files inside the $GIT_DIR directory. Signed-off-by: Jeff Hostetler --- t/perf/perf-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 601d9f67ddb..3b97e3fc0f2 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -74,7 +74,7 @@ test_perf_copy_repo_contents () { for stuff in "$1"/* do case "$stuff" in - */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees) + */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*) ;; *) cp -R "$stuff" "$repo/.git/" || exit 1 From patchwork Thu Jul 1 14:47:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C35C8C11F69 for ; Thu, 1 Jul 2021 14:48:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A84CA61414 for ; Thu, 1 Jul 2021 14:48:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233316AbhGAOut (ORCPT ); Thu, 1 Jul 2021 10:50:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233059AbhGAOuf (ORCPT ); Thu, 1 Jul 2021 10:50:35 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E635BC0613E2 for ; Thu, 1 Jul 2021 07:48:01 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id u8so8569289wrq.8 for ; Thu, 01 Jul 2021 07:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WsvE8w2+xndu9RcmlXA2qQ7mMyfEJgy5keLnSjpuVzY=; b=gowvMbnh/cWOg2+lbHdLPJUt85p0erkCwlsWvpPgn9NcUcP+Tm2y8zdL+3hmnneM8K b83P+yygTLTklDDxIAQobAve2mChJAH8b0tv7lgCyzrCWoKotZUwLvzLcUimE1rnOAwU x8Uaf1LQhGQUL0NJuUq3gGxCjeedzEdP/kE5TEiqz5uGoy3XGF9MZ41Zv+23CSMj7v4c IiwbNJTtsDllQXDz4hHS0qxyYLSZLG/65LyM/vkTo/DTMV9neIaiNZ4YxEkqI3U/06RO ASH0inqQsc20/rcSSKmnmRAGzmuUmsQvJKsz9k3ztCK4ZgIYm8Q9Lktvi7kSDAiGcgdM v00A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WsvE8w2+xndu9RcmlXA2qQ7mMyfEJgy5keLnSjpuVzY=; b=tQtT8bvCTHTTWCgQ2ipmcgambZdMewsclRyakjLL7E7va9U0yq0GLPl2rROmAvQsko nREEZjJyFeq+KgQCiq1z+khIBWSOrNzyWXUQcAONb4BOP7NwZteh2uBYBngMyrLdp97e ujV+HZiUMXv5uaiUc/EMjGzXRlWA7U/pUIKkEu4mqnvTQMlGJZGw/1wpuMpdyTTaL9d3 s8klEQMMWeTX4lqG1VQZrLwKzWfqI+hGZ8RPlFbzwdk72dnWw0z8ycLSG18omhQ0Nh/P AsOPvwknPIgiDTXxixRLbbgMQ+gV1z/SZ8Vn57TIKtSR0DQKIMJWPD7/WIg6hxuQtTU7 giuw== X-Gm-Message-State: AOAM5328bdpOxGPMh3ZJt0o/2mTkjjjhfIwO16Ov6qHZ95eq1o9Fzz/1 hNiotyGBUP3FwUk+inrJeoxQeAZ3u0k= X-Google-Smtp-Source: ABdhPJxOlzL5yCKOFxMbZXTikpjKMD0W6UCyh4eFLyL2M3YwH+HKPgAZz0aO1xWdW8y7FfifBFTezA== X-Received: by 2002:adf:e384:: with SMTP id e4mr30071538wrm.317.1625150880611; Thu, 01 Jul 2021 07:48:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r7sm137186wrs.73.2021.07.01.07.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:00 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:36 +0000 Subject: [PATCH v3 26/34] t/perf/p7519: add fsmonitor--daemon test cases Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and the "Simple IPC" interface. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index f74e6014a0a..3a3fc5748ae 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -24,7 +24,8 @@ test_description="Test core.fsmonitor" # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an # absolute path to an integration. May be a space delimited list of -# absolute paths to integrations. +# absolute paths to integrations. (This hook or list of hooks does not +# include the built-in fsmonitor--daemon.) # # The big win for using fsmonitor is the elimination of the need to scan the # working directory looking for changed and untracked files. If the file @@ -136,10 +137,16 @@ test_expect_success "one time repo setup" ' setup_for_fsmonitor() { # set INTEGRATION_SCRIPT depending on the environment - if test -n "$INTEGRATION_PATH" + if test -n "$USE_FSMONITOR_DAEMON" then + git config core.useBuiltinFSMonitor true && + INTEGRATION_SCRIPT=false + elif test -n "$INTEGRATION_PATH" + then + git config core.useBuiltinFSMonitor false && INTEGRATION_SCRIPT="$INTEGRATION_PATH" else + git config core.useBuiltinFSMonitor false && # # Choose integration script based on existence of Watchman. # Fall back to an empty integration script. @@ -175,7 +182,10 @@ test_perf_w_drop_caches () { } test_fsmonitor_suite() { - if test -n "$INTEGRATION_SCRIPT"; then + if test -n "$USE_FSMONITOR_DAEMON" + then + DESC="builtin fsmonitor--daemon" + elif test -n "$INTEGRATION_SCRIPT"; then DESC="fsmonitor=$(basename $INTEGRATION_SCRIPT)" else DESC="fsmonitor=disabled" @@ -283,4 +293,25 @@ test_expect_success "setup without fsmonitor" ' test_fsmonitor_suite trace_stop +# +# Run a full set of perf tests using the built-in fsmonitor--daemon. +# It does not use the Hook API, so it has a different setup. +# Explicitly start the daemon here and before we start client commands +# so that we can later add custom tracing. +# +if test_have_prereq FSMONITOR_DAEMON +then + USE_FSMONITOR_DAEMON=t + + trace_start fsmonitor--daemon--server + git fsmonitor--daemon start + + trace_start fsmonitor--daemon--client + test_expect_success "setup for fsmonitor--daemon" 'setup_for_fsmonitor' + test_fsmonitor_suite + + git fsmonitor--daemon stop + trace_stop +fi + test_done From patchwork Thu Jul 1 14:47:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21EB6C11F64 for ; Thu, 1 Jul 2021 14:48:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0888E6141C for ; Thu, 1 Jul 2021 14:48:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233267AbhGAOur (ORCPT ); Thu, 1 Jul 2021 10:50:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233042AbhGAOuf (ORCPT ); Thu, 1 Jul 2021 10:50:35 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EBF2C0613E8 for ; Thu, 1 Jul 2021 07:48:02 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id l8so8544243wry.13 for ; Thu, 01 Jul 2021 07:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=isJoYhm2+oaxv7Cx3ZTwRhx3+algwlrSdiIX84BJk2M=; b=Y6AJ8if8NNi0iq4C4Xl7k3eOHvff2W5J8BbT3I8NyQPZ2f+7V8kCpK3jfwunCEEYlV b9OxaXFOSuvgsq6PdZ0e5gIeyz5jqirvid4ibGzLG2zVWybvE3MZOE3IMYaD+DbdTVr6 9rbAWgusDfd/wiOckj1O1zmUeKYL9bobiWBfHzFk9lmNzAUsXLZTP4cQvjl8brQiNArU cU4VEJbXJGlplQM/qTiHpawDjpWOfh9NgsC4GCC4IkHvutwmkYZSb15/eDS/yQYQvoiN Gh1HpRkVSdDyp54Ax8ViitPhNH0msIzo3VsuKibTaTUKr7LXIwZhzTLqW1LtlmzQheIk c1WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=isJoYhm2+oaxv7Cx3ZTwRhx3+algwlrSdiIX84BJk2M=; b=h3FRQtz1z5SHu7iQYkZGp6uoSDyunDZszsFXjtd2s0uRBCtGb9agOWd1Z+3HKTfmO/ I3hXZ0DRUVWdTU0069ZdEEmislvz+YeTPWPpqGx3dTC/W9BYL5OyA1Zq87mO83SUe3Dj RWWKH6d9c72EO5bgmdXVLvRXurxo8jZ5Xz4DRuCeo80Ww/JTPktdleAwZKN7rJz17sGs EGtZ9TTO68r2KMHuf5TY4GVgHpG9/18+PAeGYWm+Nl1/UVRitW8Glil4AVdBj4misoEs E9EfAGnIVrXGnYbd3JZCFeFMHzJEUmnQQk1ZHLg9iDvPz8o4BBzBoukbnvNDFTV+mdQt 0HOg== X-Gm-Message-State: AOAM530k5B2oAGmKvPKsFF0TE6cXTsJ+uMoUfy6cJARq22IzqQ6+GXiE p/hRsFL38uFVjNb71w+5QrQlaO/NRWE= X-Google-Smtp-Source: ABdhPJzTjQmJg63pe1oJJvxQduENIFMuGAasDbFkuCDi9+RNnUp2uE7IKQsxpSyoimXqnhKS5gtRzg== X-Received: by 2002:adf:b64b:: with SMTP id i11mr46909417wre.393.1625150881196; Thu, 01 Jul 2021 07:48:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g17sm157565wrw.31.2021.07.01.07.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:00 -0700 (PDT) Message-Id: <99279c0ebd2b36fe276301f018ca4e24e4cdff36.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:37 +0000 Subject: [PATCH v3 27/34] t7527: create test for fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 497 +++++++++++++++++++++++++++++++++++ 1 file changed, 497 insertions(+) create mode 100755 t/t7527-builtin-fsmonitor.sh diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh new file mode 100755 index 00000000000..58b56dc9940 --- /dev/null +++ b/t/t7527-builtin-fsmonitor.sh @@ -0,0 +1,497 @@ +#!/bin/sh + +test_description='built-in file system watcher' + +. ./test-lib.sh + +if ! test_have_prereq FSMONITOR_DAEMON +then + skip_all="fsmonitor--daemon is not supported on this platform" + test_done +fi + +stop_daemon_delete_repo () { + r=$1 + git -C $r fsmonitor--daemon stop >/dev/null 2>/dev/null + rm -rf $1 + return 0 +} + +start_daemon () { + case "$#" in + 1) r="-C $1";; + *) r=""; + esac + + git $r fsmonitor--daemon start || return $? + git $r fsmonitor--daemon status || return $? + + return 0 +} + +test_expect_success 'explicit daemon start and stop' ' + test_when_finished "stop_daemon_delete_repo test_explicit" && + + git init test_explicit && + start_daemon test_explicit && + + git -C test_explicit fsmonitor--daemon stop && + test_must_fail git -C test_explicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon start' ' + test_when_finished "stop_daemon_delete_repo test_implicit" && + + git init test_implicit && + test_must_fail git -C test_implicit fsmonitor--daemon status && + + # query will implicitly start the daemon. + # + # for test-script simplicity, we send a V1 timestamp rather than + # a V2 token. either way, the daemon response to any query contains + # a new V2 token. (the daemon may complain that we sent a V1 request, + # but this test case is only concerned with whether the daemon was + # implicitly started.) + + GIT_TRACE2_EVENT="$(pwd)/.git/trace" \ + test-tool -C test_implicit fsmonitor-client query --token 0 >actual && + nul_to_q actual.filtered && + grep "builtin:" actual.filtered && + + # confirm that a daemon was started in the background. + # + # since the mechanism for starting the background daemon is platform + # dependent, just confirm that the foreground command received a + # response from the daemon. + + grep :\"query/response-length\" .git/trace && + + git -C test_implicit fsmonitor--daemon status && + git -C test_implicit fsmonitor--daemon stop && + test_must_fail git -C test_implicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (delete .git)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1" && + + git init test_implicit_1 && + + start_daemon test_implicit_1 && + + # deleting the .git directory will implicitly stop the daemon. + rm -rf test_implicit_1/.git && + + # [1] Create an empty .git directory so that the following Git + # command will stay relative to the `-C` directory. + # + # Without this, the Git command will override the requested + # -C argument and crawl out to the containing Git source tree. + # This would make the test result dependent upon whether we + # were using fsmonitor on our development worktree. + # + sleep 1 && + mkdir test_implicit_1/.git && + + test_must_fail git -C test_implicit_1 fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (rename .git)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_2" && + + git init test_implicit_2 && + + start_daemon test_implicit_2 && + + # renaming the .git directory will implicitly stop the daemon. + mv test_implicit_2/.git test_implicit_2/.xxx && + + # See [1] above. + # + sleep 1 && + mkdir test_implicit_2/.git && + + test_must_fail git -C test_implicit_2 fsmonitor--daemon status +' + +test_expect_success 'cannot start multiple daemons' ' + test_when_finished "stop_daemon_delete_repo test_multiple" && + + git init test_multiple && + + start_daemon test_multiple && + + test_must_fail git -C test_multiple fsmonitor--daemon start 2>actual && + grep "fsmonitor--daemon is already running" actual && + + git -C test_multiple fsmonitor--daemon stop && + test_must_fail git -C test_multiple fsmonitor--daemon status +' + +# These tests use the main repo in the trash directory + +test_expect_success 'setup' ' + >tracked && + >modified && + >delete && + >rename && + mkdir dir1 && + >dir1/tracked && + >dir1/modified && + >dir1/delete && + >dir1/rename && + mkdir dir2 && + >dir2/tracked && + >dir2/modified && + >dir2/delete && + >dir2/rename && + mkdir dirtorename && + >dirtorename/a && + >dirtorename/b && + + cat >.gitignore <<-\EOF && + .gitignore + expect* + actual* + EOF + + git -c core.useBuiltinFSMonitor= add . && + test_tick && + git -c core.useBuiltinFSMonitor= commit -m initial && + + git config core.useBuiltinFSMonitor true +' + +# The test already explicitly stopped (or tried to stop) the daemon. +# This is here in case something else fails first. +# +redundant_stop_daemon () { + git fsmonitor--daemon stop + return 0 +} + +test_expect_success 'update-index implicitly starts daemon' ' + test_when_finished redundant_stop_daemon && + + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$(pwd)/.git/trace_implicit_1" \ + git update-index --fsmonitor && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_1 +' + +test_expect_success 'status implicitly starts daemon' ' + test_when_finished redundant_stop_daemon && + + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$(pwd)/.git/trace_implicit_2" \ + git status >actual && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_2 +' + +edit_files() { + echo 1 >modified + echo 2 >dir1/modified + echo 3 >dir2/modified + >dir1/untracked +} + +delete_files() { + rm -f delete + rm -f dir1/delete + rm -f dir2/delete +} + +create_files() { + echo 1 >new + echo 2 >dir1/new + echo 3 >dir2/new +} + +rename_files() { + mv rename renamed + mv dir1/rename dir1/renamed + mv dir2/rename dir2/renamed +} + +file_to_directory() { + rm -f delete + mkdir delete + echo 1 >delete/new +} + +directory_to_file() { + rm -rf dir1 + echo 1 >dir1 +} + +verify_status() { + git status >actual && + GIT_INDEX_FILE=.git/fresh-index git read-tree master && + GIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect && + test_cmp expect actual && + echo HELLO AFTER && + cat .git/trace && + echo HELLO AFTER +} + +# The next few test cases confirm that our fsmonitor daemon sees each type +# of OS filesystem notification that we care about. At this layer we just +# ensure we are getting the OS notifications and do not try to confirm what +# is reported by `git status`. +# +# We run a simple query after modifying the filesystem just to introduce +# a bit of a delay so that the trace logging from the daemon has time to +# get flushed to disk. +# +# We `reset` and `clean` at the bottom of each test (and before stopping the +# daemon) because these commands might implicitly restart the daemon. + +clean_up_repo_and_stop_daemon () { + git reset --hard HEAD + git clean -fd + git fsmonitor--daemon stop + rm -f .git/trace +} + +test_expect_success 'edit some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + edit_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/modified$" .git/trace && + grep "^event: dir2/modified$" .git/trace && + grep "^event: modified$" .git/trace && + grep "^event: dir1/untracked$" .git/trace +' + +test_expect_success 'create some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + create_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/new$" .git/trace && + grep "^event: dir2/new$" .git/trace && + grep "^event: new$" .git/trace +' + +test_expect_success 'delete some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + delete_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/delete$" .git/trace && + grep "^event: dir2/delete$" .git/trace && + grep "^event: delete$" .git/trace +' + +test_expect_success 'rename some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + rename_files && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1/rename$" .git/trace && + grep "^event: dir2/rename$" .git/trace && + grep "^event: rename$" .git/trace && + grep "^event: dir1/renamed$" .git/trace && + grep "^event: dir2/renamed$" .git/trace && + grep "^event: renamed$" .git/trace +' + +test_expect_success 'rename directory' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + mv dirtorename dirrenamed && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace +' + +test_expect_success 'file changes to directory' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + file_to_directory && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: delete$" .git/trace && + grep "^event: delete/new$" .git/trace +' + +test_expect_success 'directory changes to a file' ' + test_when_finished clean_up_repo_and_stop_daemon && + + ( + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + directory_to_file && + + test-tool fsmonitor-client query --token 0 >/dev/null 2>&1 && + + grep "^event: dir1$" .git/trace +' + +# The next few test cases exercise the token-resync code. When filesystem +# drops events (because of filesystem velocity or because the daemon isn't +# polling fast enough), we need to discard the cached data (relative to the +# current token) and start collecting events under a new token. +# +# the 'test-tool fsmonitor-client flush' command can be used to send a +# "flush" message to a running daemon and ask it to do a flush/resync. + +test_expect_success 'flush cached data' ' + test_when_finished "stop_daemon_delete_repo test_flush" && + + git init test_flush && + + ( + GIT_TEST_FSMONITOR_TOKEN=true && + export GIT_TEST_FSMONITOR_TOKEN && + + GIT_TRACE_FSMONITOR="$(pwd)/.git/trace_daemon" && + export GIT_TRACE_FSMONITOR && + + start_daemon test_flush + ) && + + # The daemon should have an initial token with no events in _0 and + # then a few (probably platform-specific number of) events in _1. + # These should both have the same . + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_0 && + nul_to_q actual_q0 && + + touch test_flush/file_1 && + touch test_flush/file_2 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_1 && + nul_to_q actual_q1 && + + grep "file_1" actual_q1 && + + # Force a flush. This will change the , reset the , and + # flush the file data. Then create some events and ensure that the file + # again appears in the cache. It should have the new . + + test-tool -C test_flush fsmonitor-client flush >flush_0 && + nul_to_q flush_q0 && + grep "^builtin:test_00000002:0Q/Q$" flush_q0 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && + nul_to_q actual_q2 && + + grep "^builtin:test_00000002:0Q$" actual_q2 && + + touch test_flush/file_3 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_3 && + nul_to_q actual_q3 && + + grep "file_3" actual_q3 +' + +# The next few test cases create repos where the .git directory is NOT +# inside the one of the working directory. That is, where .git is a file +# that points to a directory elsewhere. This happens for submodules and +# non-primary worktrees. + +test_expect_success 'setup worktree base' ' + git init wt-base && + echo 1 >wt-base/file1 && + git -C wt-base add file1 && + git -C wt-base commit -m "c1" +' + +test_expect_success 'worktree with .git file' ' + git -C wt-base worktree add ../wt-secondary && + + ( + GIT_TRACE2_PERF="$(pwd)/trace2_wt_secondary" && + export GIT_TRACE2_PERF && + + GIT_TRACE_FSMONITOR="$(pwd)/trace_wt_secondary" && + export GIT_TRACE_FSMONITOR && + + start_daemon wt-secondary + ) && + + git -C wt-secondary fsmonitor--daemon stop && + test_must_fail git -C wt-secondary fsmonitor--daemon status +' + +# NEEDSWORK: Repeat one of the "edit" tests on wt-secondary and +# confirm that we get the same events and behavior -- that is, that +# fsmonitor--daemon correctly watches BOTH the working directory and +# the external GITDIR directory and behaves the same as when ".git" +# is a directory inside the working directory. + +test_expect_success 'cleanup worktrees' ' + stop_daemon_delete_repo wt-secondary && + stop_daemon_delete_repo wt-base +' + +test_done From patchwork Thu Jul 1 14:47:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAFB3C11F6A for ; Thu, 1 Jul 2021 14:48:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B41BD6141C for ; Thu, 1 Jul 2021 14:48:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233339AbhGAOuu (ORCPT ); Thu, 1 Jul 2021 10:50:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232866AbhGAOuf (ORCPT ); Thu, 1 Jul 2021 10:50:35 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A5B0C061786 for ; Thu, 1 Jul 2021 07:48:03 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id u6so8581472wrs.5 for ; Thu, 01 Jul 2021 07:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ev14wSKF78AibylIM3i+MwqoPuIbXki/gTA9K5amaaI=; b=dA7n8EWZajS8LJVp3LmVcdiC41CkqMMMgSCkBbq3UC17rzOIjSlfwbTSA0eZMtA+3G OH3QNo8o+KPSj7ziyA2N7UR0lWbAaMZmxXc8FVWDerfiliDZzpGeyogurv9ka7sluLkI 6hVfUU4nUavyIGfSvsr282ApbnqaR+YMaDL5jRQiVFAfxs6pC0ZQRrs0q+Bdsz+kvxPl rUABYGakvp2NWMcNFG/Wa5mjRCOOB5HUlIjaozbEEoWEIZazDlclxwRBdGXIg6QUOp5A zPctQmYLHw02RTXPdzwhucqt51vFCrdfK1SoVo/hprzF46B8z39MzpiKO85cHKkLAID1 MjFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ev14wSKF78AibylIM3i+MwqoPuIbXki/gTA9K5amaaI=; b=blZMUxCtB80UH6H9G1uXeGNjeKg7civPB/jrBDhW0TsJGvO4tBN1zosliKbQGf/mNG p3HzHHP8x6x/+7L2/nLkYpFrLzczG6YhWOCSHCT0lU8dRphzHqgtI8s/vITPi4DW+CHX DZ10znD0h7a6nV2on+psHZHK4heptY9g6oOnl+FEsUttsJuX/Y9MRYRSMMVS+4AG9tIo b20itLVAy03pOA2FvYEI7r5jh4RnxwKblI+CgHywT9++x3sJKeqGamBYCLH8tggLjU4o x+Ew5KUDb+UFyA+S5mPFP+DU6NrblxdLaUTUVZdCQ/btuEd87yIVQykAsN7RGPhxPYON B4hw== X-Gm-Message-State: AOAM530i5gHthslvNv/dh7hS1v+EoDgGIK5Sm8m4ME3DrNKZfsWEWLJl e2qCgyGfMKaHh+sh2yhZAitykoLUejQ= X-Google-Smtp-Source: ABdhPJwR5buY4VnU/Azt5EaPlFxqgk32nm37lFaMvPrzDfn/8XPQ222WfVGjwe58qDcWIloupfLNpw== X-Received: by 2002:a5d:66d0:: with SMTP id k16mr40735wrw.124.1625150881746; Thu, 01 Jul 2021 07:48:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f18sm150042wru.53.2021.07.01.07.48.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:01 -0700 (PDT) Message-Id: <3f36a31eb42147867f2cfd33f95187bb16c44a8b.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:38 +0000 Subject: [PATCH v3 28/34] fsmonitor--daemon: periodically truncate list of modified files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to periodically truncate the list of modified files to save some memory. Clients will ask for the set of changes relative to a token that they found in the FSMN index extension in the index. (This token is like a point in time, but different). Clients will then update the index to contain the response token (so that subsequent commands will be relative to this new token). Therefore, the daemon can gradually truncate the in-memory list of changed paths as they become obsolete (older than the previous token). Since we may have multiple clients making concurrent requests with a skew of tokens and clients may be racing to the talk to the daemon, we lazily truncate the list. We introduce a 5 minute delay and truncate batches 5 minutes after they are considered obsolete. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 80 +++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 7a7fef681fe..8249420ba18 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -300,6 +300,77 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, batch_src->interned_paths[k]; } +/* + * To keep the batch list from growing unbounded in response to filesystem + * activity, we try to truncate old batches from the end of the list as + * they become irrelevant. + * + * We assume that the .git/index will be updated with the most recent token + * any time the index is updated. And future commands will only ask for + * recent changes *since* that new token. So as tokens advance into the + * future, older batch items will never be requested/needed. So we can + * truncate them without loss of functionality. + * + * However, multiple commands may be talking to the daemon concurrently + * or perform a slow command, so a little "token skew" is possible. + * Therefore, we want this to be a little bit lazy and have a generous + * delay. + * + * The current reader thread walked backwards in time from `token->batch_head` + * back to `batch_marker` somewhere in the middle of the batch list. + * + * Let's walk backwards in time from that marker an arbitrary delay + * and truncate the list there. Note that these timestamps are completely + * artificial (based on when we pinned the batch item) and not on any + * filesystem activity. + */ +#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */ + +static void with_lock__truncate_old_batches( + struct fsmonitor_daemon_state *state, + const struct fsmonitor_batch *batch_marker) +{ + /* assert current thread holding state->main_lock */ + + const struct fsmonitor_batch *batch; + struct fsmonitor_batch *rest; + struct fsmonitor_batch *p; + + if (!batch_marker) + return; + + trace_printf_key(&trace_fsmonitor, "Truncate: mark (%"PRIu64",%"PRIu64")", + batch_marker->batch_seq_nr, + (uint64_t)batch_marker->pinned_time); + + for (batch = batch_marker; batch; batch = batch->next) { + time_t t; + + if (!batch->pinned_time) /* an overflow batch */ + continue; + + t = batch->pinned_time + MY_TIME_DELAY_SECONDS; + if (t > batch_marker->pinned_time) /* too close to marker */ + continue; + + goto truncate_past_here; + } + + return; + +truncate_past_here: + state->current_token_data->batch_tail = (struct fsmonitor_batch *)batch; + + rest = ((struct fsmonitor_batch *)batch)->next; + ((struct fsmonitor_batch *)batch)->next = NULL; + + for (p = rest; p; p = fsmonitor_batch__pop(p)) { + trace_printf_key(&trace_fsmonitor, + "Truncate: kill (%"PRIu64",%"PRIu64")", + p->batch_seq_nr, (uint64_t)p->pinned_time); + } +} + static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) { struct fsmonitor_batch *p; @@ -645,6 +716,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, * that work. */ fsmonitor_free_token_data(token_data); + } else if (batch) { + /* + * This batch is the first item in the list + * that is older than the requested sequence + * number and might be considered to be + * obsolete. See if we can truncate the list + * and save some memory. + */ + with_lock__truncate_old_batches(state, batch); } } From patchwork Thu Jul 1 14:47:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C4A7C11F6A for ; Thu, 1 Jul 2021 14:48:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5347D61414 for ; Thu, 1 Jul 2021 14:48:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233235AbhGAOur (ORCPT ); Thu, 1 Jul 2021 10:50:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232894AbhGAOuf (ORCPT ); Thu, 1 Jul 2021 10:50:35 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7693C06178A for ; Thu, 1 Jul 2021 07:48:03 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id v5so8607706wrt.3 for ; Thu, 01 Jul 2021 07:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TOIsbteyOxly7Ebv5T9N4h7JB9VV+mW6FkAPqdMo/4k=; b=PA/nclOQw2yvYYNrnEaaUCBzGpjWSMKEl9X8OigaY9jXH88KRky3HEcFuqL3NF6GU5 uVbIEPOHA+mLsf+phknKSagsrGNIVhY3HZQFlAhJicWUpvwbIukR6/x3cTQUv7gDxQhE gFiA0vcXR1eddBRxz/mcBAtucsc42S1pvLTrj/uR+i77w0LF8PcmoYWD9Vvvup8HxvW9 oritKfV+0XtQUajOIqj3STfA/ME2qp+J7yL/SVEFIgbcT3r9diuP7XFs/lvYfj3G4mB6 aM+S+Mdul4l5zFFQAJkF0U3SyOi9NvBv2tIeuMiO0pR2V9zUMSxGGFVztc+vxnRKLy4+ FMnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TOIsbteyOxly7Ebv5T9N4h7JB9VV+mW6FkAPqdMo/4k=; b=OFhj3+E6S0szUZEF2SPSYGxMrnCm0pMG0k3h9B+fr7LaSidDfa1MbCaQkyuObXO4EN 8288UBOzo4rI/jbdJQEHPTpYWst4/QAo0zxxaZvNC5n/kyzdwmPQ8pIYq3euOXvp8flP YFvfQ5B+zmzWwoW8uqYeMgOlMuel4QlOcZ0Ub+spNqZlX9ajzdyxVc8+VnmkGtTzyD+3 7qFaI+ZTf4x5GLfytV06/RBelyIZBt4rWt5hMpXkBwRC6ea3fSyUx7P/FTzMg/HXBo1h jrLD15OPOFWOa1LzAByDpgsCWUp65R0rqxYfTtk08qRu/OCmMoWSteiUfC3WS+EQOWcU EpBw== X-Gm-Message-State: AOAM532R53Knb9CqqGgzW4blU5urnwSTLP31ANayE9epl5Jaz65Fme46 vdqZcKz3tU3Jqekwmf4jIgllhhZ9z+U= X-Google-Smtp-Source: ABdhPJx258gqv6MY88BdLfAV3CzP/hNWrEIA2AkPWUU2lbjQqfgGYBAZ9hGM74Sb9On0iKHiVbflrg== X-Received: by 2002:a5d:498d:: with SMTP id r13mr21350332wrq.379.1625150882346; Thu, 01 Jul 2021 07:48:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y7sm9726141wma.22.2021.07.01.07.48.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:02 -0700 (PDT) Message-Id: <555caca2216dd3e459c118d76b46eb983a58e051.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:39 +0000 Subject: [PATCH v3 29/34] fsmonitor--daemon: use a cookie file to sync with file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon client threads to create a cookie file inside the .git directory and then wait until FS events for the cookie are observed by the FS listener thread. This helps address the racy nature of file system events by blocking the client response until the kernel has drained any event backlog. This is especially important on MacOS where kernel events are only issued with a limited frequency. See the `latency` argument of `FSeventStreamCreate()`. The kernel only signals every `latency` seconds, but does not guarantee that the kernel queue is completely drained, so we may have to wait more than one interval. If we increase the frequency, the system is more likely to drop events. We avoid these issues by having each client thread create a unique cookie file and then wait until it is seen in the event stream. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 5 + 2 files changed, 232 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 8249420ba18..25f18f2726b 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -94,6 +94,149 @@ static int do_as_client__status(void) } } +enum fsmonitor_cookie_item_result { + FCIR_ERROR = -1, /* could not create cookie file ? */ + FCIR_INIT = 0, + FCIR_SEEN, + FCIR_ABORT, +}; + +struct fsmonitor_cookie_item { + struct hashmap_entry entry; + const char *name; + enum fsmonitor_cookie_item_result result; +}; + +static int cookies_cmp(const void *data, const struct hashmap_entry *he1, + const struct hashmap_entry *he2, const void *keydata) +{ + const struct fsmonitor_cookie_item *a = + container_of(he1, const struct fsmonitor_cookie_item, entry); + const struct fsmonitor_cookie_item *b = + container_of(he2, const struct fsmonitor_cookie_item, entry); + + return strcmp(a->name, keydata ? keydata : b->name); +} + +static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie( + struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + int fd; + struct fsmonitor_cookie_item *cookie; + struct strbuf cookie_pathname = STRBUF_INIT; + struct strbuf cookie_filename = STRBUF_INIT; + enum fsmonitor_cookie_item_result result; + int my_cookie_seq; + + CALLOC_ARRAY(cookie, 1); + + my_cookie_seq = state->cookie_seq++; + + strbuf_addf(&cookie_filename, "%i-%i", getpid(), my_cookie_seq); + + strbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix); + strbuf_addbuf(&cookie_pathname, &cookie_filename); + + cookie->name = strbuf_detach(&cookie_filename, NULL); + cookie->result = FCIR_INIT; + hashmap_entry_init(&cookie->entry, strhash(cookie->name)); + + hashmap_add(&state->cookies, &cookie->entry); + + trace_printf_key(&trace_fsmonitor, "cookie-wait: '%s' '%s'", + cookie->name, cookie_pathname.buf); + + /* + * Create the cookie file on disk and then wait for a notification + * that the listener thread has seen it. + */ + fd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600); + if (fd >= 0) { + close(fd); + unlink(cookie_pathname.buf); + + /* + * NEEDSWORK: This is an infinite wait (well, unless another + * thread sends us an abort). I'd like to change this to + * use `pthread_cond_timedwait()` and return an error/timeout + * and let the caller do the trivial response thing. + */ + while (cookie->result == FCIR_INIT) + pthread_cond_wait(&state->cookies_cond, + &state->main_lock); + } else { + error_errno(_("could not create fsmonitor cookie '%s'"), + cookie->name); + + cookie->result = FCIR_ERROR; + } + + hashmap_remove(&state->cookies, &cookie->entry, NULL); + + result = cookie->result; + + free((char*)cookie->name); + free(cookie); + strbuf_release(&cookie_pathname); + + return result; +} + +/* + * Mark these cookies as _SEEN and wake up the corresponding client threads. + */ +static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state, + const struct string_list *cookie_names) +{ + /* assert current thread holding state->main_lock */ + + int k; + int nr_seen = 0; + + for (k = 0; k < cookie_names->nr; k++) { + struct fsmonitor_cookie_item key; + struct fsmonitor_cookie_item *cookie; + + key.name = cookie_names->items[k].string; + hashmap_entry_init(&key.entry, strhash(key.name)); + + cookie = hashmap_get_entry(&state->cookies, &key, entry, NULL); + if (cookie) { + trace_printf_key(&trace_fsmonitor, "cookie-seen: '%s'", + cookie->name); + cookie->result = FCIR_SEEN; + nr_seen++; + } + } + + if (nr_seen) + pthread_cond_broadcast(&state->cookies_cond); +} + +/* + * Set _ABORT on all pending cookies and wake up all client threads. + */ +static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + struct hashmap_iter iter; + struct fsmonitor_cookie_item *cookie; + int nr_aborted = 0; + + hashmap_for_each_entry(&state->cookies, &iter, cookie, entry) { + trace_printf_key(&trace_fsmonitor, "cookie-abort: '%s'", + cookie->name); + cookie->result = FCIR_ABORT; + nr_aborted++; + } + + if (nr_aborted) + pthread_cond_broadcast(&state->cookies_cond); +} + /* * Requests to and from a FSMonitor Protocol V2 provider use an opaque * "token" as a virtual timestamp. Clients can request a summary of all @@ -397,6 +540,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) * We should create a new token and start fresh (as if we just * booted up). * + * [2] Some of those lost events may have been for cookie files. We + * should assume the worst and abort them rather letting them starve. + * * If there are no concurrent threads readering the current token data * series, we can free it now. Otherwise, let the last reader free * it. @@ -418,6 +564,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) state->current_token_data = new_one; fsmonitor_free_token_data(free_me); + + with_lock__abort_all_cookies(state); } void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) @@ -492,6 +640,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, int hash_ret; int do_trivial = 0; int do_flush = 0; + int do_cookie = 0; + enum fsmonitor_cookie_item_result cookie_result; /* * We expect `command` to be of the form: @@ -552,6 +702,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, * We have a V2 valid token: * "builtin::" */ + do_cookie = 1; } } @@ -560,6 +711,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, if (!state->current_token_data) BUG("fsmonitor state does not have a current token"); + /* + * Write a cookie file inside the directory being watched in + * an effort to flush out existing filesystem events that we + * actually care about. Suspend this client thread until we + * see the filesystem events for this cookie file. + * + * Creating the cookie lets us guarantee that our FS listener + * thread has drained the kernel queue and we are caught up + * with the kernel. + * + * If we cannot create the cookie (or otherwise guarantee that + * we are caught up), we send a trivial response. We have to + * assume that there might be some very, very recent activity + * on the FS still in flight. + */ + if (do_cookie) { + cookie_result = with_lock__wait_for_cookie(state); + if (cookie_result != FCIR_SEEN) { + error(_("fsmonitor: cookie_result '%d' != SEEN"), + cookie_result); + do_trivial = 1; + } + } + if (do_flush) with_lock__do_force_resync(state); @@ -771,7 +946,9 @@ static int handle_client(void *data, return result; } -#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" +#define FSMONITOR_DIR "fsmonitor--daemon" +#define FSMONITOR_COOKIE_DIR "cookies" +#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR "/" FSMONITOR_COOKIE_DIR "/") enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( const char *rel) @@ -924,6 +1101,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, } } + if (cookie_names->nr) + with_lock__mark_cookies_seen(state, cookie_names); + pthread_mutex_unlock(&state->main_lock); } @@ -1013,7 +1193,9 @@ static int fsmonitor_run_daemon(void) memset(&state, 0, sizeof(state)); + hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); + pthread_cond_init(&state.cookies_cond, NULL); state.error_code = 0; state.current_token_data = fsmonitor_new_token_data(); @@ -1038,6 +1220,44 @@ static int fsmonitor_run_daemon(void) state.nr_paths_watching = 2; } + /* + * We will write filesystem syncing cookie files into + * ///-. + * + * The extra layers of subdirectories here keep us from + * changing the mtime on ".git/" or ".git/foo/" when we create + * or delete cookie files. + * + * There have been problems with some IDEs that do a + * non-recursive watch of the ".git/" directory and run a + * series of commands any time something happens. + * + * For example, if we place our cookie files directly in + * ".git/" or ".git/foo/" then a `git status` (or similar + * command) from the IDE will cause a cookie file to be + * created in one of those dirs. This causes the mtime of + * those dirs to change. This triggers the IDE's watch + * notification. This triggers the IDE to run those commands + * again. And the process repeats and the machine never goes + * idle. + * + * Adding the extra layers of subdirectories prevents the + * mtime of ".git/" and ".git/foo" from changing when a + * cookie file is created. + */ + strbuf_init(&state.path_cookie_prefix, 0); + strbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch); + + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR); + mkdir(state.path_cookie_prefix.buf, 0777); + + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR); + mkdir(state.path_cookie_prefix.buf, 0777); + + strbuf_addch(&state.path_cookie_prefix, '/'); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1050,6 +1270,7 @@ static int fsmonitor_run_daemon(void) err = fsmonitor_run_daemon_1(&state); done: + pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsmonitor_fs_listen__dtor(&state); @@ -1057,6 +1278,11 @@ done: strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); + strbuf_release(&state.path_cookie_prefix); + + /* + * NEEDSWORK: Consider "rm -rf /" + */ return err; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 89a9ef20b24..e9fc099bae9 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -45,6 +45,11 @@ struct fsmonitor_daemon_state { struct fsmonitor_token_data *current_token_data; + struct strbuf path_cookie_prefix; + pthread_cond_t cookies_cond; + int cookie_seq; + struct hashmap cookies; + int error_code; struct fsmonitor_daemon_backend_data *backend_data; From patchwork Thu Jul 1 14:47:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 903C3C11F64 for ; Thu, 1 Jul 2021 14:48:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 798F06141C for ; Thu, 1 Jul 2021 14:48:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233147AbhGAOuu (ORCPT ); Thu, 1 Jul 2021 10:50:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233004AbhGAOuh (ORCPT ); Thu, 1 Jul 2021 10:50:37 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2838C061765 for ; Thu, 1 Jul 2021 07:48:05 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id u6so8581640wrs.5 for ; Thu, 01 Jul 2021 07:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UFxndu8AKBnI+OL+LbIJO1y+ehiYxgLiTHwS0jWo67M=; b=Gt1rIpvq2KfnOmgs9W4cmBQ9sKMcZ3clakZESd1zHbKmKAXbE468wPKVVu3ROYq8/Z CRmH1T4vgSzEBxDDSnu0cr588viqDXm1syI8noPo2IvUnL8RagQKxtbp4WHhref/bDJf nqWzuXfHBmnvP1GSoQP/TaxJgA6Pa1YD6gn9tY1PuzZ6x0QfUXTVxzE+97R1rjDUmsEk f0leJ9piSsmrRKl+9u8VWb54BN9XZljAAVNubejYeRrlcUiBVTGFZwv7NoxQF6mEMkFu WfDY9Ou9l/D4MLzUD+dhgbRZL+kyonXJVbrXh41iNhY7xjMNGdi5qyzRcNJXatmgfSR/ aBGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UFxndu8AKBnI+OL+LbIJO1y+ehiYxgLiTHwS0jWo67M=; b=Puo4KxyhA3vqF5bdGr3nQchrf/Bo+eghIgh2AWHLTUMSeCdgK7SoPyrXQd1KXyQnc5 ZqWwliCd7k528fWwd07FjNkORoJWlTzS7qGC6y9NzXY+SD5XxDVcxrVWNf2dyw1jf/RD /Z4TYQd1uI4NnrrKFebTvTksy0av25HaMQsgLsns3W0XzWVFu8w8vwi8y7AusCu7F2vj 8aMUTKW01EPeboGLoMO7/iXG0CxgkEaq2IQ/tQ/kFPQxxa5jmqAZjazFSBRwcEfVAqta O2TZdhkoO1XKsSViOEtQ39hAYDdLQFU6Ps/2EsDK7KqE6JLQdMOB2mQonqglNGtfQxle tKDw== X-Gm-Message-State: AOAM530F1U+aWf3BoCT1NoY6/a+5De/JSx7VJsXuzFWJMNqYhLr0M1nz AQUnngYqH+d/8CfAv1nDaAuVSABvr4k= X-Google-Smtp-Source: ABdhPJzaOquRpd9cZ81Cz70ul/bcNGudVPuhoDh+DU2u8gpOlEICn+o79Pu+OH5VozAgCX+cqwVLug== X-Received: by 2002:a5d:6502:: with SMTP id x2mr45398200wru.327.1625150884348; Thu, 01 Jul 2021 07:48:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n20sm202659wmk.12.2021.07.01.07.48.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:02 -0700 (PDT) Message-Id: <75bb4bc846364ec80e087d15bb1cb84265257ce3.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:40 +0000 Subject: [PATCH v3 30/34] fsmonitor: enhance existing comments Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- fsmonitor.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 3719ddfeec9..f53791c8674 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -360,9 +360,25 @@ void refresh_fsmonitor(struct index_state *istate) } apply_results: - /* a fsmonitor process can return '/' to indicate all entries are invalid */ + /* + * The response from FSMonitor (excluding the header token) is + * either: + * + * [a] a (possibly empty) list of NUL delimited relative + * pathnames of changed paths. This list can contain + * files and directories. Directories have a trailing + * slash. + * + * [b] a single '/' to indicate the provider had no + * information and that we should consider everything + * invalid. We call this a trivial response. + */ if (query_success && query_result.buf[bol] != '/') { - /* Mark all entries returned by the monitor as dirty */ + /* + * Mark all pathnames returned by the monitor as dirty. + * + * This updates both the cache-entries and the untracked-cache. + */ buf = query_result.buf; for (i = bol; i < query_result.len; i++) { if (buf[i] != '\0') @@ -377,11 +393,15 @@ apply_results: if (istate->untracked) istate->untracked->use_fsmonitor = 1; } else { - - /* We only want to run the post index changed hook if we've actually changed entries, so keep track - * if we actually changed entries or not */ + /* + * We received a trivial response, so invalidate everything. + * + * We only want to run the post index changed hook if + * we've actually changed entries, so keep track if we + * actually changed entries or not. + */ int is_cache_changed = 0; - /* Mark all entries invalid */ + for (i = 0; i < istate->cache_nr; i++) { if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) { is_cache_changed = 1; @@ -389,7 +409,10 @@ apply_results: } } - /* If we're going to check every file, ensure we save the results */ + /* + * If we're going to check every file, ensure we save + * the results. + */ if (is_cache_changed) istate->cache_changed |= FSMONITOR_CHANGED; From patchwork Thu Jul 1 14:47:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37F94C11F67 for ; Thu, 1 Jul 2021 14:48:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 22D006141C for ; Thu, 1 Jul 2021 14:48:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233383AbhGAOu4 (ORCPT ); Thu, 1 Jul 2021 10:50:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233028AbhGAOuh (ORCPT ); Thu, 1 Jul 2021 10:50:37 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EB6FC0613DC for ; Thu, 1 Jul 2021 07:48:06 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id w13so4690859wmc.3 for ; Thu, 01 Jul 2021 07:48:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=smA/CGHifxx0NwUOVPrq185XmIP/kTroOWItviZLrjw=; b=Z7qfWOYyNOnwg30GmBpez0yEtBwTu4U1NPOVs1rdmGcs6D1I/LWHT4Tg0oecAVJ3aF iPHT+kEWCmDapc7YEsjOduPx4CkWTG6X7cuMbvMOspoeflX8RRKG5Zle4Sq0fv6uXC7h tPP0PVYdLwhTuTi2GH6civaa5XAwdFcPDDBswNoi8yxSDVWDY7aRGNbXH4UafrYWBK9J Y5ZgpkLyM6mYjauOBCJXJRCsxuTdYpinusgJz1SdjZR9agjGcpanwb7nS8QMcx1JRPyN 9MtUhpKBUIHxQFyLQIBXVRhLQ34FpG14YycJ4vQ9KigH7F9hUr1TPBGhfP7e5ZoxDGfE bxHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=smA/CGHifxx0NwUOVPrq185XmIP/kTroOWItviZLrjw=; b=ZM0zK+l/ac0d3w5JqEku/MUqUroPbp8dtP8Cg4aOIG99p+4DekOMs8DAeVO9GMW3x4 /cgL6Ej7mWjmk9oFig9NgsVM61LuK2vNzpi90/gcZ1xIrNKCWyw84pnpknDBcUGau2oD +YVJeui6yxVjJnrw0hVh92V4a25P7AKYxmyZ59dZjzmMhNLhpdQmGPkQjXeKpk/NkFJp QNwd1RDH+Uk3S7F1GyumsA+6o93Q/eIRbu1R1IGA+dw8EsikUi/O9smKEUVm3yLw88lm y+/frmnaGpnAx3T/gDiz/bkZ91vAMAeZvZi++GqYXSH1RNeYx0yX0PGFE/mOW05OKd/x IzPg== X-Gm-Message-State: AOAM5307MGAqFFvWZRn7E0fjOBOV+R3HUQA1EYdf6i/znnscphUeiPv7 CSBs5MhLXIJwLKkF4wEHhKiaf1aM9ho= X-Google-Smtp-Source: ABdhPJydrFTZHndAaymtc0JrS2svkKUSkZjQQnWlmtsEY4uQWtBxh4MfhjemV7yZIs5pJMFn7/HNnA== X-Received: by 2002:a7b:cd99:: with SMTP id y25mr11273161wmj.184.1625150884865; Thu, 01 Jul 2021 07:48:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z4sm231056wmf.9.2021.07.01.07.48.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:04 -0700 (PDT) Message-Id: <8b3c4f4e6dd989de88b0b7b543751e362fa0d649.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:41 +0000 Subject: [PATCH v3 31/34] fsmonitor: force update index after large responses Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Set the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when FSMonitor returns a very large repsonse to ensure that the index is written to disk. Normally, when the FSMonitor response includes a tracked file, the index is always updated. Similarly, the index might be updated when the response alters the untracked-cache (when enabled). However, in cases where neither of those cause the index to be considered changed, the FSMonitor response is wasted. Subsequent Git commands will make requests with the same token and receive the same response. If that response is very large, performance may suffer. It would be more efficient to force update the index now (and the token in the index extension) in order to reduce the size of the response received by future commands. This was observed on Windows after a large checkout. On Windows, the kernel emits events for the files that are changed as they are changed. However, it might delay events for the containing directories until the system is more idle (or someone scans the directory (so it seems)). The first status following a checkout would get the list of files. The subsequent status commands would get the list of directories as the events trickled out. But they would never catch up because the token was not advanced because the index wasn't updated. This list of directories caused `wt_status_collect_untracked()` to unnecessarily spend time actually scanning them during each command. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/fsmonitor.c b/fsmonitor.c index f53791c8674..eee653f9337 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -236,6 +236,45 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) untracked_cache_invalidate_path(istate, name, 0); } +/* + * The number of pathnames that we need to receive from FSMonitor + * before we force the index to be updated. + * + * Note that any pathname within the set of received paths MAY cause + * cache-entry or istate flag bits to be updated and thus cause the + * index to be updated on disk. + * + * However, the response may contain many paths (such as ignored + * paths) that will not update any flag bits. And thus not force the + * index to be updated. (This is fine and normal.) It also means + * that the token will not be updated in the FSMonitor index + * extension. So the next Git command will find the same token in the + * index, make the same token-relative request, and receive the same + * response (plus any newly changed paths). If this response is large + * (and continues to grow), performance could be impacted. + * + * For example, if the user runs a build and it writes 100K object + * files but doesn't modify any source files, the index would not need + * to be updated. The FSMonitor response (after the build and + * relative to a pre-build token) might be 5MB. Each subsequent Git + * command will receive that same 100K/5MB response until something + * causes the index to be updated. And `refresh_fsmonitor()` will + * have to iterate over those 100K paths each time. + * + * Performance could be improved if we optionally force update the + * index after a very large response and get an updated token into + * the FSMonitor index extension. This should allow subsequent + * commands to get smaller and more current responses. + * + * The value chosen here does not need to be precise. The index + * will be updated automatically the first time the user touches + * a tracked file and causes a command like `git status` to + * update an mtime to be updated and/or set a flag bit. + * + * NEEDSWORK: Does this need to be a config value? + */ +static int fsmonitor_force_update_threshold = 100; + void refresh_fsmonitor(struct index_state *istate) { struct strbuf query_result = STRBUF_INIT; @@ -379,19 +418,28 @@ apply_results: * * This updates both the cache-entries and the untracked-cache. */ + int count = 0; + buf = query_result.buf; for (i = bol; i < query_result.len; i++) { if (buf[i] != '\0') continue; fsmonitor_refresh_callback(istate, buf + bol); bol = i + 1; + count++; } - if (bol < query_result.len) + if (bol < query_result.len) { fsmonitor_refresh_callback(istate, buf + bol); + count++; + } /* Now mark the untracked cache for fsmonitor usage */ if (istate->untracked) istate->untracked->use_fsmonitor = 1; + + if (count > fsmonitor_force_update_threshold) + istate->cache_changed |= FSMONITOR_CHANGED; + } else { /* * We received a trivial response, so invalidate everything. From patchwork Thu Jul 1 14:47:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C249C11F69 for ; Thu, 1 Jul 2021 14:48:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 85C306141C for ; Thu, 1 Jul 2021 14:48:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233366AbhGAOuz (ORCPT ); Thu, 1 Jul 2021 10:50:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233024AbhGAOuh (ORCPT ); Thu, 1 Jul 2021 10:50:37 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7167C0613DE for ; Thu, 1 Jul 2021 07:48:06 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id k16-20020a05600c1c90b02901f4ed0fcfe7so3145364wms.5 for ; Thu, 01 Jul 2021 07:48:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CanizR/UrFodlfR4fnEKm8+XPFHtacDLVABF8p2xfR4=; b=LO/5hAaLBO8lxBK0Kgt+4lIChL5qOpwDGriLdfumMpum69O+bTbe9ArH64zRQvBHP+ pyUvP5MIjqD72xK/rA4ofnCURe2U7k7Fy7XfrBDiaHa5u4xAWCrtUzpstqG4dbF/rr9q cCRQ2S4Y2vcDDabQWcQcVRZ83MjcVV/KhYBDd1xs8vAbQUEfxKH45mIdQHKmdt2qMVlU lY4uUVwwmBKV85UNIuSzpvl+cAnSokr5QK3xXWtCbMlDZ9JArA6Kv3iIe5Vn0Dou5Vqs GP91plcEYAxEvrMb0l1AjyPzYod4wrMH8OGawG8QLWxbJ2lA1rWNxSlPd7iDqvzfVYpu GohQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CanizR/UrFodlfR4fnEKm8+XPFHtacDLVABF8p2xfR4=; b=FOacgZIfxUlg3089VPTSGdngfHNran3V+9S2PVHv5t8jJTp2/6X3MaxGa0lZ5c0pjg +/LhAYc9ZQa42UzKiCXRXYAL64oiqS2AwSfyLMDb0GAyqAj9ZXcmxijP/Gx4iFsrGWiB EJ6Hn9TkzFZnRxWD6w3p/aEr2AvHHwxo8YH/pgvRWaEZB/UrrKshS0oOyD1593f2o6FL sCmSQz7h8pEctah819m8Cfi20PC9dWfv/xkX/z60wlsRkFi5Vb9t560x6HjrNzyqtYxj 8+rRNWQjhYXm1eYo7MNy9MlA8z704C311U/l+bOAw/33B717sG33uwaI3UYPx7GoS6DC 7IVA== X-Gm-Message-State: AOAM532/nuSou/t2wW+sYbtF880ZY5OlaC8BaCY+kOiLpA3AyvcLtyZs M7LXKVw3GZAr6NH7haVthq+hBouQYpw= X-Google-Smtp-Source: ABdhPJzZbzyWVOQXREC6Ydx4nECNICpB2r5d18I6jkybLUgLyPUwFVkz8KAWyAS8Zd2lBuQIorDe7Q== X-Received: by 2002:a7b:c417:: with SMTP id k23mr11192356wmi.87.1625150885439; Thu, 01 Jul 2021 07:48:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w9sm169405wru.3.2021.07.01.07.48.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:05 -0700 (PDT) Message-Id: <97dce46d1d04527a540de37d814ca094e487623a.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:42 +0000 Subject: [PATCH v3 32/34] t7527: test status with untracked-cache and fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon features and a series of edits and verify that status output is identical. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 87 ++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 58b56dc9940..d1832702397 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -152,6 +152,8 @@ test_expect_success 'setup' ' .gitignore expect* actual* + flush* + trace* EOF git -c core.useBuiltinFSMonitor= add . && @@ -494,4 +496,89 @@ test_expect_success 'cleanup worktrees' ' stop_daemon_delete_repo wt-base ' +# The next few tests perform arbitrary/contrived file operations and +# confirm that status is correct. That is, that the data (or lack of +# data) from fsmonitor doesn't cause incorrect results. And doesn't +# cause incorrect results when the untracked-cache is enabled. + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop +' + +matrix_clean_up_repo () { + git reset --hard HEAD + git clean -fd +} + +matrix_try () { + uc=$1 + fsm=$2 + fn=$3 + + test_expect_success "Matrix[uc:$uc][fsm:$fsm] $fn" ' + matrix_clean_up_repo && + $fn && + if test $uc = false -a $fsm = false + then + git status --porcelain=v1 >.git/expect.$fn + else + git status --porcelain=v1 >.git/actual.$fn + test_cmp .git/expect.$fn .git/actual.$fn + fi + ' + + return $? +} + +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" +for uc_val in $uc_values +do + if test $uc_val = false + then + test_expect_success "Matrix[uc:$uc_val] disable untracked cache" ' + git config core.untrackedcache false && + git update-index --no-untracked-cache + ' + else + test_expect_success "Matrix[uc:$uc_val] enable untracked cache" ' + git config core.untrackedcache true && + git update-index --untracked-cache + ' + fi + + fsm_values="false true" + for fsm_val in $fsm_values + do + if test $fsm_val = false + then + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor" ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop 2>/dev/null + ' + else + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor" ' + git config core.useBuiltinFSMonitor true && + git fsmonitor--daemon start && + git update-index --fsmonitor + ' + fi + + matrix_try $uc_val $fsm_val edit_files + matrix_try $uc_val $fsm_val delete_files + matrix_try $uc_val $fsm_val create_files + matrix_try $uc_val $fsm_val rename_files + matrix_try $uc_val $fsm_val file_to_directory + matrix_try $uc_val $fsm_val directory_to_file + done +done + test_done From patchwork Thu Jul 1 14:47:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D5B5C11F69 for ; Thu, 1 Jul 2021 14:48:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5CF5461414 for ; Thu, 1 Jul 2021 14:48:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233412AbhGAOu5 (ORCPT ); Thu, 1 Jul 2021 10:50:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233067AbhGAOui (ORCPT ); Thu, 1 Jul 2021 10:50:38 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DD49C0613DF for ; Thu, 1 Jul 2021 07:48:07 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id j34so4679527wms.5 for ; Thu, 01 Jul 2021 07:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=VhA8e70cmeJE/8taWH8a2ZkqsNWrr0+EV4Og5Cn7GBw=; b=nP7INjaYjqUto4SKdXQM5YdsCHjOh/AZXC35uTTT53iUclpCt1g7lBvl6gmzmYkZu7 JH4+YqaoenxN6ROrkQZP0sUuNURhfnE11upP4UL93CBB0l+UaKGyXFunfbDKpMnn1Nfr 7368EDHFVIpaWIxXycZ6QAtUjTGVeWme+DIbYwjBvJ85gGk19uyoaMHKENb0yhKH1Jkj y2aw6ucrvmMb4VAtrNKpvfPgXxUhVTm7QLz4CmnWyW+15zovEtdzOZlzpDz5GU91p3+6 /a28arQMrbWwK3UU/sFN3B0sCUVQTVSvxmfdIdpHuoeTW5uJVWeOVFhM0Er5Jo0IcR3n LafA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=VhA8e70cmeJE/8taWH8a2ZkqsNWrr0+EV4Og5Cn7GBw=; b=SrEza7sDLgxbCqb8E3issqdUdIOhYaSJTtfLtfZRP3lIYLqeISBkezVevHwmNhgVVk 0sqUAUHMVCcgzrG+DTAYr41BrsxVJyX0x/OMaiBB7keLXYb5kN8HPAr1aRRV8vw/0xPY VfQdkPfRGGXUszWbw4d4Hcpub5/5fwfGztEZ+90Du/9YUpIdZL5cOOntw3NyYu7r1YCi 7l7IGuvwyUEaIVoD1tIVA3FMKnREPTmT+Zzyfpm7+XPpvtP/SCLhE3C09WoTxQ6/54kF vMTindKeZLq1sH6F9CGHNb+jZH87iHkxRspwyZ1/CWoDUT8o4AG40KV3tXiQ/pkNjqKV UM3A== X-Gm-Message-State: AOAM5328A5NgD/kuLMM4FaxS2fW65dQkNNHUYw5PWUaA6i7u+mfvHJiI RQlMDIUkGiVjSTznWCg01DHnvKDPyUU= X-Google-Smtp-Source: ABdhPJw+gDT92KkYOMFJn34Muw7Wk93tFeHpfWjqXPxGYP0O4nEjDQ12zZ1Mg7uewOt8TbmSn9SS5g== X-Received: by 2002:a7b:c405:: with SMTP id k5mr132882wmi.34.1625150885991; Thu, 01 Jul 2021 07:48:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n15sm170890wrr.22.2021.07.01.07.48.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:05 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:43 +0000 Subject: [PATCH v3 33/34] fsmonitor: handle shortname for .git Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler On Windows, teach FSMonitor to recognize the shortname of ".git" as an alias for ".git". Sometimes we receive FS events using the shortname, such as when a CMD shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1". The FS notification arrives using whatever combination of long and shortnames used by the other process. (Shortnames do seem to be case normalized, however.) NEEDSWORK: This only addresses the case of removing or renaming the ".git" directory using the shortname alias, so that the daemon properly shuts down. I'm leaving it a task for later to handle the general case of shortnames and report them to the fsmonitor client process. This would include tracked and untracked paths that just happen to have a shortname alias. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-win32.c | 192 +++++++++++++++---- t/t7527-builtin-fsmonitor.sh | 65 +++++++ 2 files changed, 217 insertions(+), 40 deletions(-) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c index d707d47a0d7..f2ea5940790 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -48,6 +48,8 @@ struct fsmonitor_daemon_backend_data #define LISTENER_HAVE_DATA_WORKTREE 1 #define LISTENER_HAVE_DATA_GITDIR 2 int nr_listener_handles; + + struct strbuf dot_git_shortname; }; /* @@ -250,6 +252,62 @@ static void cancel_rdcw_watch(struct one_watch *watch) watch->is_active = FALSE; } +/* + * Process a single relative pathname event. + * Return 1 if we should shutdown. + */ +static int process_1_worktree_event( + FILE_NOTIFY_INFORMATION *info, + struct string_list *cookie_list, + struct fsmonitor_batch **batch, + const struct strbuf *path, + enum fsmonitor_path_type t) +{ + const char *slash; + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path->buf); + string_list_append(cookie_list, + slash ? slash + 1 : path->buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info->Action == FILE_ACTION_REMOVED) || + (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + return 1; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!*batch) + *batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(*batch, path->buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path->buf); + } + + return 0; +} + /* * Process filesystem events that happen anywhere (recursively) under the * root directory. For a normal working directory, this includes @@ -294,7 +352,6 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) */ for (;;) { FILE_NOTIFY_INFORMATION *info = (void *)p; - const char *slash; enum fsmonitor_path_type t; strbuf_reset(&path); @@ -303,45 +360,45 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) t = fsmonitor_classify_path_workdir_relative(path.buf); - switch (t) { - case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: - /* special case cookie files within .git */ - - /* Use just the filename of the cookie file. */ - slash = find_last_dir_sep(path.buf); - string_list_append(&cookie_list, - slash ? slash + 1 : path.buf); - break; - - case IS_INSIDE_DOT_GIT: - /* ignore everything inside of "/.git/" */ - break; - - case IS_DOT_GIT: - /* "/.git" was deleted (or renamed away) */ - if ((info->Action == FILE_ACTION_REMOVED) || - (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { - trace2_data_string("fsmonitor", NULL, - "fsm-listen/dotgit", - "removed"); - goto force_shutdown; - } - break; - - case IS_WORKDIR_PATH: - /* queue normal pathname */ - if (!batch) - batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, path.buf); - break; - - case IS_GITDIR: - case IS_INSIDE_GITDIR: - case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: - default: - BUG("unexpected path classification '%d' for '%s'", - t, path.buf); - } + if (process_1_worktree_event(info, &cookie_list, &batch, + &path, t)) + goto force_shutdown; + + /* + * NEEDSWORK: If `path` contains a shortname (that is, + * if any component within it is a shortname), we + * should expand it to a longname (See + * `GetLongPathNameW()`) and re-normalize, classify, + * and process it because our client is probably + * expecting "normal" paths. + * + * HOWEVER, if our process has called `chdir()` to get + * us out of the root of the worktree (so that the + * root directory is not busy), then we have to be + * careful to convert the paths in the INFO array + * (which are relative to the directory of the RDCW + * watch and not the CWD) into absolute paths before + * calling GetLongPathNameW() and then convert the + * computed value back to a RDCW-relative pathname + * (which is what we and the client expect). + * + * FOR NOW, just handle case (1) exactly so that we + * shutdown properly when ".git" is deleted via the + * shortname alias. + * + * We might see case (2) events for cookie files, but + * we can ignore them. + * + * FOR LATER, handle case (3) where the worktree + * events contain shortnames. We should convert + * them to longnames to avoid confusing the client. + */ + if (data->dot_git_shortname.len && + !strcmp(path.buf, data->dot_git_shortname.buf) && + process_1_worktree_event(info, &cookie_list, &batch, + &data->dot_git_shortname, + IS_DOT_GIT)) + goto force_shutdown; skip_this_path: if (!info->NextEntryOffset) @@ -415,6 +472,14 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state) t, path.buf); } + /* + * WRT shortnames, this external gitdir will not see + * case (1) nor case (3) events. + * + * We might see case (2) events for cookie files, but + * we can ignore them. + */ + skip_this_path: if (!info->NextEntryOffset) break; @@ -493,6 +558,7 @@ clean_shutdown: int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { struct fsmonitor_daemon_backend_data *data; + char shortname[16]; /* a padded 8.3 buffer */ CALLOC_ARRAY(data, 1); @@ -523,6 +589,52 @@ int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) data->nr_listener_handles++; } + /* + * NEEDSWORK: Properly handle 8.3 shortnames. RDCW events can + * contain a shortname (if another application uses a + * shortname in a system call). We care about aliasing and + * the use of shortnames for: + * + * (1) ".git", + * -- if an external process deletes ".git" using "GIT~1", + * we need to catch that and shutdown. + * + * (2) our cookie files, + * -- if an external process deletes one of our cookie + * files using a shortname, we will get a shortname + * event for it. However, we should have already + * gotten a longname event for it when we created the + * cookie, so we can safely discard the shortname + * events for cookie files. + * + * (3) the spelling of modified files that we report to clients. + * -- we need to report the longname to the client because + * that is what they are expecting. Presumably, the + * client is going to lookup the paths that we report + * in their index and untracked-cache, so we should + * normalize the data for them. (Technically, they + * could adapt, so we could relax this maybe.) + * + * FOR NOW, while our CWD is at the root of the worktree we + * can easily get the spelling of the shortname of ".git" (if + * the volume has shortnames enabled). For most worktrees + * this value will be "GIT~1", but we don't want to assume + * that. + * + * Capture this so that we can handle (1). + * + * We leave (3) for a future effort. + */ + strbuf_init(&data->dot_git_shortname, 0); + GetShortPathNameA(".git", shortname, sizeof(shortname)); + if (!strcmp(".git", shortname)) + trace_printf_key(&trace_fsmonitor, "No shortname for '.git'"); + else { + trace_printf_key(&trace_fsmonitor, + "Shortname of '.git' is '%s'", shortname); + strbuf_addstr(&data->dot_git_shortname, shortname); + } + state->backend_data = data; return 0; diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index d1832702397..b166b4a0a31 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -113,6 +113,71 @@ test_expect_success 'implicit daemon stop (rename .git)' ' test_must_fail git -C test_implicit_2 fsmonitor--daemon status ' +# File systems on Windows may or may not have shortnames. +# This is a volume-specific setting on modern systems. +# "C:/" drives are required to have them enabled. Other +# hard drives default to disabled. +# +# This is a crude test to see if shortnames are enabled +# on the volume containing the test directory. It is +# crude, but it does not require elevation like `fsutil`. +# +test_lazy_prereq SHORTNAMES ' + mkdir .foo && + test -d "FOO~1" +' + +# Here we assume that the shortname of ".git" is "GIT~1". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s" && + + git init test_implicit_1s && + + start_daemon test_implicit_1s && + + # renaming the .git directory will implicitly stop the daemon. + # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + # this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}. + mv test_implicit_1s/.gitxyz test_implicit_1s/.git && + + test_must_fail git -C test_implicit_1s fsmonitor--daemon status +' + +# Here we first create a file with LONGNAME of "GIT~1" before +# we create the repo. This will cause the shortname of ".git" +# to be "GIT~2". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s2" && + + mkdir test_implicit_1s2 && + echo HELLO >test_implicit_1s2/GIT~1 && + git init test_implicit_1s2 && + + [ -f test_implicit_1s2/GIT~1 ] && + [ -d test_implicit_1s2/GIT~2 ] && + + start_daemon test_implicit_1s2 && + + # renaming the .git directory will implicitly stop the daemon. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git && + + test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status +' + test_expect_success 'cannot start multiple daemons' ' test_when_finished "stop_daemon_delete_repo test_multiple" && From patchwork Thu Jul 1 14:47:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12354357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 439E2C11F6A for ; Thu, 1 Jul 2021 14:48:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26F076141D for ; Thu, 1 Jul 2021 14:48:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233449AbhGAOu6 (ORCPT ); Thu, 1 Jul 2021 10:50:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233119AbhGAOuj (ORCPT ); Thu, 1 Jul 2021 10:50:39 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0076C0613E2 for ; Thu, 1 Jul 2021 07:48:07 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id l8so8544611wry.13 for ; Thu, 01 Jul 2021 07:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5/sMXoHyGg+3j8jh501vy22q2NIOR4Of40K9vWI1fSY=; b=oEU3GiNd+YmSC8nU1094l8MkH9kLX4rj6rZxI+eA8OA00DIeiFI1SE+NPJgf0p9Pcx g2R+36AaWBZ+HhtXY1kEHs+Snu5EgXpJOGw6GpPustWpAaer0Cz8AgpquMeJ98sLPY9y DoOVtOsZkWGkNVf2BQdSKra6/zDZ/HfrT8YLbVmVXu6PxDFyLtg6UtNlMd0SkO0y2FvL X2SCWLh0Le6JV4+PoC/Zto+ChlipmYgswDz5xkkdcpatR0Y9fffDJK1IjGM515wD58C1 041/uXdUyGCpmhxTe89e7aSlVuM6peJ+MGy+sGyAqQh/0qG6Sa4PasvL+cJ+nE3ja5pS cspg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5/sMXoHyGg+3j8jh501vy22q2NIOR4Of40K9vWI1fSY=; b=CGJnaA45DSkw/KeDUFBS+Ml1Vkh/bGlYoD0CnEpsDNSZbymB5695TuSdDI+mizEzMZ slVwBG+X1N4EwToa4i0S0f/7HIGd6SgKEmowVnGCw/sEYzgm7x/wx8OrjFiFcdrMZ/Z0 f6wUn36+ySmaALGYmS1Ej7qVkf/PQzmpUPwff293j3q7r8Z3nPwVznQwoP9D41aZdMdD ZE5kFKAZhFYT3HZ7BYdmwS/Iem8qk5yAJqOrBOLnE8EoAC0nDkV+YiceTIB6DUOcIR5K Ul6o5nt9mflF18uuMcJke3HEr6wQdEV9vX3ZJjzGsU6W2iuhA7DpsY+/Nh3C5ipUmavQ 2F3Q== X-Gm-Message-State: AOAM530f7hercwyrSHWHrGMpBHEuTpuDwoiAcEtSFg36y7nI0iwNJhu7 5iTcpmHF21mv1jfbRw0kI9DXrDk8OHc= X-Google-Smtp-Source: ABdhPJyP9Gs79IBdV5qj/HtNQom36Yu1+5ols2wk4w8oENAJemlaiKJfcZ9ultv4nJgRNmYd2U+M4Q== X-Received: by 2002:a5d:4003:: with SMTP id n3mr21579372wrp.147.1625150886480; Thu, 01 Jul 2021 07:48:06 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y3sm136510wru.78.2021.07.01.07.48.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jul 2021 07:48:06 -0700 (PDT) Message-Id: <627e27fe60bb543902fdcc4b2179c620403d9c38.1625150864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Jul 2021 14:47:44 +0000 Subject: [PATCH v3 34/34] t7527: test FS event reporing on MacOS WRT case and Unicode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes Schindelin , Jeff Hostetler , Derrick Stolee , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Confirm that MacOS FS events are reported with a normalized spelling. APFS (and/or HFS+) is case-insensitive. This means that case-independent lookups ( [ -d .git ] and [ -d .GIT ] ) should both succeed. But that doesn't tell us how FS events are reported if we try "rm -rf .git" versus "rm -rf .GIT". Are the events reported using the on-disk spelling of the pathname or in the spelling used by the command. NEEDSWORK: I was only able to test case. It would be nice to add tests that use different Unicode spellings/normalizations and understand the differences between APFS and HFS+ in this area. We should confirm that the spelling of the workdir paths that the daemon sends to clients are always properly normalized. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index b166b4a0a31..d2ff1bf6c49 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -178,6 +178,36 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status ' +# Confirm that MacOS hides all of the Unicode normalization and/or +# case folding from the FS events. That is, are the pathnames in the +# FS events reported using the spelling on the disk or in the spelling +# used by the other process. +# +# Note that we assume that the filesystem is set to case insensitive. +# +# NEEDSWORK: APFS handles Unicode and Unicode normalization +# differently than HFS+. I only have an APFS partition, so +# more testing here would be helpful. +# + +# Rename .git using alternate spelling and confirm that the daemon +# sees the event using the correct spelling and shutdown. +test_expect_success UTF8_NFD_TO_NFC 'MacOS event spelling (rename .GIT)' ' + test_when_finished "stop_daemon_delete_repo test_apfs" && + + git init test_apfs && + start_daemon test_apfs && + + [ -d test_apfs/.git ] && + [ -d test_apfs/.GIT ] && + + mv test_apfs/.GIT test_apfs/.FOO && + sleep 1 && + mv test_apfs/.FOO test_apfs/.git && + + test_must_fail git -C test_apfs fsmonitor--daemon status +' + test_expect_success 'cannot start multiple daemons' ' test_when_finished "stop_daemon_delete_repo test_multiple" &&