From patchwork Thu Oct 3 23:32:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Steadmon X-Patchwork-Id: 11173525 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05C33139A for ; Thu, 3 Oct 2019 23:33:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D836B215EA for ; Thu, 3 Oct 2019 23:33:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="U6vtP4GN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731682AbfJCXdF (ORCPT ); Thu, 3 Oct 2019 19:33:05 -0400 Received: from mail-pg1-f201.google.com ([209.85.215.201]:43000 "EHLO mail-pg1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727452AbfJCXdF (ORCPT ); Thu, 3 Oct 2019 19:33:05 -0400 Received: by mail-pg1-f201.google.com with SMTP id d3so3031548pgv.9 for ; Thu, 03 Oct 2019 16:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qVnqpH3h8oCKfyBF6a9bDdxgKtpHWtZSjCn/mKy7u3Y=; b=U6vtP4GNz8gSVkdobeU9NJazWvIVfTFvYc0OhVTNE79pwTNuuPpVLhu/OtFLLSm+8i hTOPXv7D84DeSppL3l4jPBLRyoks48FNP9M3Kfrf+8PmhxO5gdsB5MhZkZY4UNYMzRYo xPELOK9wSvoxxNDyn+Oum25gbE3tIzFVH6mki8NhtsrgJ7lueEwkHmCNihquFcGpqm2Y /t03E8L1kn3p2zx//xCN/IIn4Eb3gVI4gnK/k39HyLRpaEi+1ps9HkDkzAwORow8kmQH aFX66N7LIpuoErHYEjJEpwTEp6lh5j9TyAZbuREYua7vr6TAboEaGwQoh/S2K5gi72JZ 3VCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qVnqpH3h8oCKfyBF6a9bDdxgKtpHWtZSjCn/mKy7u3Y=; b=gkSEvbDDFt/3gmIW4A+R6fKkdjKceCVIJbHuIZj3rdmYILVleYezGSQ+eXO3cF98P5 HwgyGs9I9+M7R4taobOgDxNofF1IF/3K81/ybtq1kJaGEm1F0beSLEw1grVLFeS3zBNe rg39KNY/XcZyd22a+MwQaLrVkVvUVHXB5ADG26uIlc5jzJsv0FFZQ047DXVkKxaNzM5n sUTyjRtuR8WK/sxYkXaxBFkjGtwl4BUeaHS+w0nWMhoKap0d9NYsbRZJqYoZUMT13ZDj 8Al72Zaw0zY6eallePd0IGXnVm/fwYa4CVL1CQ8tU8B8fPVeqM/i+leC0I+2l4d6QfOh Q9Tw== X-Gm-Message-State: APjAAAXw/JbP3jq4pDKumyXdviLh9L80S0qkoZrSqLULQX3OGgq5DCpt h0AyWNDohhIPQXM7zRLInF573l9AbIqEGYc5ZwbAgUBvuf4w6a+7T+ToLmZP7RMFEbdHJsVfWK+ /HMbbSjOgtP4LQuiTJnUxI1lGmpjcnN5rLwbkjo5H7qD5/DPFzfJ4oO3UQic8IxQ= X-Google-Smtp-Source: APXvYqxWFfozsMWxY839NGq/mDmnYuVDfaNHN5byZhwl8IHR2wu5AQSeShXjklNZdKOjcP2kf1m4Ig8w4pvZ+Q== X-Received: by 2002:a63:211c:: with SMTP id h28mr12215821pgh.372.1570145584625; Thu, 03 Oct 2019 16:33:04 -0700 (PDT) Date: Thu, 3 Oct 2019 16:32:56 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <99e4a0fe409a236d210d95e54cd03fce61daa291.1564438745.git.steadmon@google.com> X-Mailer: git-send-email 2.23.0.581.g78d2f28ef7-goog Subject: [PATCH v4 1/4] docs: mention trace2 target-dir mode in git-config From: Josh Steadmon To: git@vger.kernel.org, git@jeffhostetler.com Cc: stolee@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Move the description of trace2's target-directory behavior into the shared trace2-target-values file so that it is included in both the git-config and api-trace2 docs. Leave the SID discussion only in api-trace2 since it's a technical detail. Signed-off-by: Josh Steadmon --- Documentation/technical/api-trace2.txt | 7 +++---- Documentation/trace2-target-values.txt | 4 +++- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt index 71eb081fed..80ffceada0 100644 --- a/Documentation/technical/api-trace2.txt +++ b/Documentation/technical/api-trace2.txt @@ -142,10 +142,9 @@ system or global config value to one of the following: include::../trace2-target-values.txt[] -If the target already exists and is a directory, the traces will be -written to files (one per process) underneath the given directory. They -will be named according to the last component of the SID (optionally -followed by a counter to avoid filename collisions). +When trace files are written to a target directory, they will be named according +to the last component of the SID (optionally followed by a counter to avoid +filename collisions). == Trace2 API diff --git a/Documentation/trace2-target-values.txt b/Documentation/trace2-target-values.txt index 27d3c64e66..3985b6d3c2 100644 --- a/Documentation/trace2-target-values.txt +++ b/Documentation/trace2-target-values.txt @@ -2,7 +2,9 @@ * `0` or `false` - Disables the target. * `1` or `true` - Writes to `STDERR`. * `[2-9]` - Writes to the already opened file descriptor. -* `` - Writes to the file in append mode. +* `` - Writes to the file in append mode. If the target +already exists and is a directory, the traces will be written to files (one +per process) underneath the given directory. * `af_unix:[:]` - Write to a Unix DomainSocket (on platforms that support them). Socket type can be either `stream` or `dgram`; if omitted Git will From patchwork Thu Oct 3 23:32:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Steadmon X-Patchwork-Id: 11173529 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0311214DB for ; Thu, 3 Oct 2019 23:33:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D64E82133F for ; Thu, 3 Oct 2019 23:33:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mXgEWndU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731730AbfJCXdJ (ORCPT ); Thu, 3 Oct 2019 19:33:09 -0400 Received: from mail-pl1-f201.google.com ([209.85.214.201]:47469 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731688AbfJCXdJ (ORCPT ); Thu, 3 Oct 2019 19:33:09 -0400 Received: by mail-pl1-f201.google.com with SMTP id v2so2713225plp.14 for ; Thu, 03 Oct 2019 16:33:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=wmkMdwf8Hqfqt0WUgC/qo+wg+wKZyeTo0aV9UurjgFA=; b=mXgEWndU+SghdL87LWpfoK0zzddHwNsRIvl7gKXvgbfVpI65SmO5TGUN6aGzEV/Yd5 LQgG8/z1FKRQB4glYMSaJzLv9sj+RmkfudwEMggCyxuMhCjPfmOilIXg04bOaA/HZ9Op ptafEPju1dMvrQb894K3Aow92Ydre/iDS/yXbwuJDq9ghXgK3beJZAVoe24no3PoQWVk So78Ur8qijQnuAUo01t/72olJJUoQ9JQsguFg6j9kzGw9XGYDkYvLmGHuJvhJT5otbg/ 7u5eYKsgSfFG6gHSESBmOLjJQfeQrya04BquVf846ZIza0xcEsK0mkcFmIuox+NOrsuq p94A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wmkMdwf8Hqfqt0WUgC/qo+wg+wKZyeTo0aV9UurjgFA=; b=UfunW4zSykRVRugwZ0Pkzg/a+d1/4UphqfX3bC1BcOmZFASUvvFKJp7Qfc75kgr1cA o1UewXpioDUOCtEUoDCoL532D3GgoeXeK0XcSDJ7kSko/e4205z88A5UmR2bYXCkl+yx 93L0wv8fBg90pHDsYuoTT4faxO3LhgZiXqy5ggntNYJBwMDbcAAeUoMkUSKwhwegG7ev qgTCJMx4m7LhGWSWBoghQkA3ESx0USFxRUW63IyR8DSqs6OHVPNEPFlQml2IBpVarW42 3Fmr2qX88VbUDj+tcaX7hTKSdZ4l/WSO/CLfNWsi9T5ldTkd8o8fOaHfrDtmhY7CGYDO yYHA== X-Gm-Message-State: APjAAAWEUUR7SaK4nSXWXs697Z7Gt3Fk+ooz+48Yw/cFEonQI4XhCSO3 40j3/X0oMyNx+kmU7B4itGLI80L9xukffx69lpPjYI2pIt7lu9LALYW4XMX5foTOKiKnLDMCtyj nY85er6Wz4tygSdhOqPWGE3UjgoDLlIxCo8vyHQm2JAqQvf9pNku+0pkCMNAn5jw= X-Google-Smtp-Source: APXvYqwDQzWg1vcvUUI3EWY+cWyjH7JkweAobJOUW9CsHrW0ox7qKw56rHbe2/TQWhklXlUfTwDHyOnu1UhKUQ== X-Received: by 2002:a63:358a:: with SMTP id c132mr12312484pga.32.1570145587012; Thu, 03 Oct 2019 16:33:07 -0700 (PDT) Date: Thu, 3 Oct 2019 16:32:57 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <99e4a0fe409a236d210d95e54cd03fce61daa291.1564438745.git.steadmon@google.com> X-Mailer: git-send-email 2.23.0.581.g78d2f28ef7-goog Subject: [PATCH v4 2/4] docs: clarify trace2 version invariants From: Josh Steadmon To: git@vger.kernel.org, git@jeffhostetler.com Cc: stolee@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Make it explicit that we always want trace2 "version" events to be the first event of any trace session. Also list the changes that would or would not cause the EVENT format version field to be incremented. Signed-off-by: Josh Steadmon --- Documentation/technical/api-trace2.txt | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt index 80ffceada0..18b79128fd 100644 --- a/Documentation/technical/api-trace2.txt +++ b/Documentation/technical/api-trace2.txt @@ -604,7 +604,13 @@ only present on the "start" and "atexit" events. ==== Event-Specific Key/Value Pairs `"version"`:: - This event gives the version of the executable and the EVENT format. + This event gives the version of the executable and the EVENT format. It + should always be the first event in a trace session. The EVENT format + version will be incremented if new event types are added, if existing + fields are removed, or if there are significant changes in + interpretation of existing events or fields. Smaller changes, such as + adding a new field to an existing event, will not require an increment + to the EVENT format version. + ------------ { From patchwork Thu Oct 3 23:32:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Steadmon X-Patchwork-Id: 11173531 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD418139A for ; Thu, 3 Oct 2019 23:33:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 904472133F for ; Thu, 3 Oct 2019 23:33:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oqcIKVcI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731734AbfJCXdL (ORCPT ); Thu, 3 Oct 2019 19:33:11 -0400 Received: from mail-qk1-f201.google.com ([209.85.222.201]:37042 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731688AbfJCXdL (ORCPT ); Thu, 3 Oct 2019 19:33:11 -0400 Received: by mail-qk1-f201.google.com with SMTP id o133so4500834qke.4 for ; Thu, 03 Oct 2019 16:33:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9E9hOYsrs89GMjPlB3G+AxpcVxldK5ktvSdyV9kFHvc=; b=oqcIKVcIjbFe8M4Og9t9kGrd+fuTysvBQzbcfzx1d7aB0Hr8iRFN1VZdBSdLXog7+i g3swJp9RfmlUXtKjYy+Pcjx2v4h7+htGopk4n3gK3a4aerbeEotE2Ikdt+U0t9EJMgRC PNECaO3ilLl3y2+KNA/0VGyIHG35cGy+nWeP7eqI7t8KB8vjrTHffNhiefF8zDCtzaxs jQmLjCKIxm3ijjMk4J1NeCFAYLqn2ki9zSVr88qe3DxVuyiYtR45X/5vyge/PGlyUu+d u59Ra0qSBni+n1rBtT9HNew1GWdQEeRq3T84c9Qz2xI/C/+cx+RxKoulWllpAtXCvgpk b4aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9E9hOYsrs89GMjPlB3G+AxpcVxldK5ktvSdyV9kFHvc=; b=LmMR32KAi13BBou7WjFjBRgOatLmaXbUpNchr46l+yb0Z0dg4QVhqN5/nZEMe9QA5x R8s+bGmzAhaweutUYSiHivMMgxb8mkbHx+Zt5tC97eyH+NZpchh6P5myKc2WJYdQpaH7 Wzl54HvKRjEbR29nWd8/ISD36g9gC0IlxtATCDORp6DrNkYKAzc/+k1GqOzeBqC/GFPE qHPHD1AOAG8GogiQjJLy0/xyZ1lOTg/Nt/HWWrXbo9nmrOO6+gfKKSeBVMCRqvgh9BcY +lcg/CIntqMtDKuzltn182GzSXszceQLMScU5s3QkUvBb3dwfrd++JWg5gZ7JDs/il73 fchQ== X-Gm-Message-State: APjAAAWCmPINFbs3waNQrva8De0Mxd8mcgRXLzZtKKkYpivVpODphg6j IbK87zPBj1vyZsi/XUC9wmrH6iPy4dE483jdnMEKguXP1gWxrZHDxWCDwLt6oRg5FC1ugh+WAWH crD+PEGKjX3gRowXRatBV9jAiyITuwLj0p69YoITS7AFDFhsqpPA0qX+ZFbXjCW4= X-Google-Smtp-Source: APXvYqwV6o061XjNaGVWMmwNI6y0rD9rs6pQ19b5XaA7DnTHel+0bi4lVgDW368PfSVWY8bBZ9lKGbPjEXuajA== X-Received: by 2002:ac8:33c3:: with SMTP id d3mr12919484qtb.41.1570145589825; Thu, 03 Oct 2019 16:33:09 -0700 (PDT) Date: Thu, 3 Oct 2019 16:32:58 -0700 In-Reply-To: Message-Id: <98a8440d3f0ef3cd3cdd0291051d976d4a659cc4.1570144820.git.steadmon@google.com> Mime-Version: 1.0 References: <99e4a0fe409a236d210d95e54cd03fce61daa291.1564438745.git.steadmon@google.com> X-Mailer: git-send-email 2.23.0.581.g78d2f28ef7-goog Subject: [PATCH v4 3/4] trace2: don't overload target directories From: Josh Steadmon To: git@vger.kernel.org, git@jeffhostetler.com Cc: stolee@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org trace2 can write files into a target directory. With heavy usage, this directory can fill up with files, causing difficulty for trace-processing systems. This patch adds a config option (trace2.maxFiles) to set a maximum number of files that trace2 will write to a target directory. The following behavior is enabled when the maxFiles is set to a positive integer: When trace2 would write a file to a target directory, first check whether or not the directory is overloaded. A directory is overloaded if there is a sentinel file declaring an overload, or if the number of files exceeds trace2.maxFiles. If the latter, create a sentinel file to speed up later overload checks. The assumption is that a separate trace-processing system is dealing with the generated traces; once it processes and removes the sentinel file, it should be safe to generate new trace files again. The default value for trace2.maxFiles is zero, which disables the overload check. The config can also be overridden with a new environment variable: GIT_TRACE2_MAX_FILES. Signed-off-by: Josh Steadmon --- Documentation/config/trace2.txt | 6 +++ t/t0212-trace2-event.sh | 17 +++++++ trace2/tr2_dst.c | 86 +++++++++++++++++++++++++++++++++ trace2/tr2_sysenv.c | 3 ++ trace2/tr2_sysenv.h | 2 + 5 files changed, 114 insertions(+) diff --git a/Documentation/config/trace2.txt b/Documentation/config/trace2.txt index 2edbfb02fe..4ce0b9a6d1 100644 --- a/Documentation/config/trace2.txt +++ b/Documentation/config/trace2.txt @@ -54,3 +54,9 @@ trace2.destinationDebug:: By default, these errors are suppressed and tracing is silently disabled. May be overridden by the `GIT_TRACE2_DST_DEBUG` environment variable. + +trace2.maxFiles:: + Integer. When writing trace files to a target directory, do not + write additional traces if we would exceed this many files. Instead, + write a sentinel file that will block further tracing to this + directory. Defaults to 0, which disables this check. diff --git a/t/t0212-trace2-event.sh b/t/t0212-trace2-event.sh index ff5b9cc729..2ff97e72da 100755 --- a/t/t0212-trace2-event.sh +++ b/t/t0212-trace2-event.sh @@ -265,4 +265,21 @@ test_expect_success JSON_PP 'using global config, event stream, error event' ' test_cmp expect actual ' +test_expect_success "don't overload target directory" ' + mkdir trace_target_dir && + test_when_finished "rm -r trace_target_dir" && + ( + GIT_TRACE2_MAX_FILES=5 && + export GIT_TRACE2_MAX_FILES && + cd trace_target_dir && + test_seq $GIT_TRACE2_MAX_FILES >../expected_filenames.txt && + xargs touch <../expected_filenames.txt && + cd .. && + GIT_TRACE2_EVENT="$(pwd)/trace_target_dir" test-tool trace2 001return 0 + ) && + echo git-trace2-overload >>expected_filenames.txt && + ls trace_target_dir >ls_output.txt && + test_cmp expected_filenames.txt ls_output.txt +' + test_done diff --git a/trace2/tr2_dst.c b/trace2/tr2_dst.c index 5dda0ca1cd..af3405f179 100644 --- a/trace2/tr2_dst.c +++ b/trace2/tr2_dst.c @@ -1,3 +1,5 @@ +#include + #include "cache.h" #include "trace2/tr2_dst.h" #include "trace2/tr2_sid.h" @@ -8,6 +10,19 @@ */ #define MAX_AUTO_ATTEMPTS 10 +/* + * Sentinel file used to detect when we're overloading a directory with too many + * trace files. + */ +#define OVERLOAD_SENTINEL_NAME "git-trace2-overload" + +/* + * When set to zero, disables directory overload checking. Otherwise, controls + * how many files we can write to a directory before entering overload mode. + * This can be overridden via the TR2_SYSENV_MAX_FILES setting. + */ +static int tr2env_max_files = 0; + static int tr2_dst_want_warning(void) { static int tr2env_dst_debug = -1; @@ -32,6 +47,67 @@ void tr2_dst_trace_disable(struct tr2_dst *dst) dst->need_close = 0; } +/* + * Check to make sure we're not overloading the target directory with too many + * files. First get the threshold (if present) from the config or envvar. If + * it's zero or unset, disable this check. Next check for the presence of a + * sentinel file, then check file count. If we are overloaded, create the + * sentinel file if it doesn't already exist. + * + * We expect that some trace processing system is gradually collecting files + * from the target directory; after it removes the sentinel file we'll start + * writing traces again. + */ +static int tr2_dst_overloaded(const char *tgt_prefix) +{ + int file_count = 0, max_files = 0, ret = 0; + const char *max_files_var; + DIR *dirp; + struct strbuf path = STRBUF_INIT, sentinel_path = STRBUF_INIT; + struct stat statbuf; + + /* Get the config or envvar and decide if we should continue this check */ + max_files_var = tr2_sysenv_get(TR2_SYSENV_MAX_FILES); + if (max_files_var && *max_files_var && ((max_files = atoi(max_files_var)) >= 0)) + tr2env_max_files = max_files; + + if (!tr2env_max_files) { + ret = 0; + goto cleanup; + } + + strbuf_addstr(&path, tgt_prefix); + if (!is_dir_sep(path.buf[path.len - 1])) { + strbuf_addch(&path, '/'); + } + + /* check sentinel */ + strbuf_addbuf(&sentinel_path, &path); + strbuf_addstr(&sentinel_path, OVERLOAD_SENTINEL_NAME); + if (!stat(sentinel_path.buf, &statbuf)) { + ret = 1; + goto cleanup; + } + + /* check file count */ + dirp = opendir(path.buf); + while (file_count < tr2env_max_files && dirp && readdir(dirp)) + file_count++; + if (dirp) + closedir(dirp); + + if (file_count >= tr2env_max_files) { + creat(sentinel_path.buf, 0666); + ret = 1; + goto cleanup; + } + +cleanup: + strbuf_release(&path); + strbuf_release(&sentinel_path); + return ret; +} + static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) { int fd; @@ -50,6 +126,16 @@ static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) strbuf_addstr(&path, sid); base_path_len = path.len; + if (tr2_dst_overloaded(tgt_prefix)) { + strbuf_release(&path); + if (tr2_dst_want_warning()) + warning("trace2: not opening %s trace file due to too " + "many files in target directory %s", + tr2_sysenv_display_name(dst->sysenv_var), + tgt_prefix); + return 0; + } + for (attempt_count = 0; attempt_count < MAX_AUTO_ATTEMPTS; attempt_count++) { if (attempt_count > 0) { strbuf_setlen(&path, base_path_len); diff --git a/trace2/tr2_sysenv.c b/trace2/tr2_sysenv.c index 5958cfc424..3c3792eca2 100644 --- a/trace2/tr2_sysenv.c +++ b/trace2/tr2_sysenv.c @@ -49,6 +49,9 @@ static struct tr2_sysenv_entry tr2_sysenv_settings[] = { "trace2.perftarget" }, [TR2_SYSENV_PERF_BRIEF] = { "GIT_TRACE2_PERF_BRIEF", "trace2.perfbrief" }, + + [TR2_SYSENV_MAX_FILES] = { "GIT_TRACE2_MAX_FILES", + "trace2.maxfiles" }, }; /* clang-format on */ diff --git a/trace2/tr2_sysenv.h b/trace2/tr2_sysenv.h index 8dd82a7a56..d4364a7b85 100644 --- a/trace2/tr2_sysenv.h +++ b/trace2/tr2_sysenv.h @@ -24,6 +24,8 @@ enum tr2_sysenv_variable { TR2_SYSENV_PERF, TR2_SYSENV_PERF_BRIEF, + TR2_SYSENV_MAX_FILES, + TR2_SYSENV_MUST_BE_LAST }; From patchwork Thu Oct 3 23:32:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josh Steadmon X-Patchwork-Id: 11173533 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C53D9139A for ; Thu, 3 Oct 2019 23:33:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9737E20862 for ; Thu, 3 Oct 2019 23:33:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bLeYjeQN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731752AbfJCXdP (ORCPT ); Thu, 3 Oct 2019 19:33:15 -0400 Received: from mail-pg1-f201.google.com ([209.85.215.201]:43002 "EHLO mail-pg1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731688AbfJCXdP (ORCPT ); Thu, 3 Oct 2019 19:33:15 -0400 Received: by mail-pg1-f201.google.com with SMTP id d3so3031738pgv.9 for ; Thu, 03 Oct 2019 16:33:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=QtPRDrbr7tyZLz0DSPk6C1u/MSfl4gNeCp+aPAjcZDw=; b=bLeYjeQNTbuW9dqc7tt7hzl8WKsGZ3yQOc0tOWU+u8f7k7lJwygLl7pbozZKy+3e13 TCLd5BPw61/FDFeZJpHtFLIajAvV+Qy/ONcro1ZlrvOCiMnMa1m22M5a7tnh2vc1WRBP E962pNzNE0xv4fCbFCocoYXI30wj5WjlprM+c9jsEWb9aUQ9fJqu+1/3QuhflFke+IjH PHjZ/xQrO8p2jVg7pDvK7J18/u0N/KOz76vtRNW93CfHNem6owU812J9RQd4A4aJPRJM U/JCKMGXQNlHZ+9Tdjpiews/nsECB2SovBPB0eYPniAF5LuKgkSRPtCRscFuJelJzN4Z zx9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=QtPRDrbr7tyZLz0DSPk6C1u/MSfl4gNeCp+aPAjcZDw=; b=uHc4Xd6Htg5ROI4zmxRgZ1PbqZPldzNcr8oYM2ZoQcze73CvidjfvYpwcqC0riQlHp Uld5oDRZY9ICqIdrapnZo6lqwOVHff+FEkoiJse6T518KcWGtTBe9sXWERQCD3m/Jrm9 /bXbnRRMP+qq8nDYrERIb+Dis8EwyrAL6lhKfzETIS6PT0mxYbYTPXizn5OR9D1k7+Cx fpAfcvC+UjtMXWjCk8XjOiqoUEbfeNewBKQ3f3TkjRT1YZFmcfNjFpqER0N4iZM7apIi v8IFwI8YfjFzdK8pYbeh1M1Z1j876b0fKfL8c+wmfgY40ai189g7QKn+oOVuqt7qC8Vt NOCQ== X-Gm-Message-State: APjAAAXqKBYUU56iFNofTRO7XGcu8FAQJsRcBQng6mUHPxg/WrV6BjU5 T/6yi0eKSRRKQ9CvGpsRMIHF2P0eOJkV21oA27PYW5xXlhu/m3c/cjTSmLNnxHrBXeuwK8zifNy awbp8CiDk1G2t5ujn1M3fxhKBw+ezMV/AUdOIucJdU1qlZz/767EC2+3jY0bs9KQ= X-Google-Smtp-Source: APXvYqzRK3JieFPgaXrdTcva6FpSPw4W+NghEc0UYs2HC3pm2jcA9tqxzO6PU9qE31jnTbSirKRjJbDejILZXA== X-Received: by 2002:a63:de12:: with SMTP id f18mr12525531pgg.453.1570145592588; Thu, 03 Oct 2019 16:33:12 -0700 (PDT) Date: Thu, 3 Oct 2019 16:32:59 -0700 In-Reply-To: Message-Id: <790c5ac95a8aa929f061fc68c89f53f95778e0f6.1570144820.git.steadmon@google.com> Mime-Version: 1.0 References: <99e4a0fe409a236d210d95e54cd03fce61daa291.1564438745.git.steadmon@google.com> X-Mailer: git-send-email 2.23.0.581.g78d2f28ef7-goog Subject: [PATCH v4 4/4] trace2: write overload message to sentinel files From: Josh Steadmon To: git@vger.kernel.org, git@jeffhostetler.com Cc: stolee@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a new "overload" event type for trace2 event destinations. When the trace2 overload feature creates a sentinel file, it will include the normal trace2 output in the sentinel, along with this new overload event. Writing this message into the sentinel file is useful for tracking how often the overload protection feature is triggered in practice. Bump up the event format version since we've added a new event type. Signed-off-by: Josh Steadmon --- Documentation/technical/api-trace2.txt | 15 ++++++-- t/t0212-trace2-event.sh | 4 ++- trace2/tr2_dst.c | 47 ++++++++++++++------------ trace2/tr2_dst.h | 1 + trace2/tr2_tgt_event.c | 31 +++++++++++++---- trace2/tr2_tgt_normal.c | 2 +- trace2/tr2_tgt_perf.c | 2 +- 7 files changed, 69 insertions(+), 33 deletions(-) diff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt index 18b79128fd..5afd643fa6 100644 --- a/Documentation/technical/api-trace2.txt +++ b/Documentation/technical/api-trace2.txt @@ -128,7 +128,7 @@ yields ------------ $ cat ~/log.event -{"event":"version","sid":"sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.620713Z","file":"common-main.c","line":38,"evt":"1","exe":"2.20.1.155.g426c96fcdb"} +{"event":"version","sid":"sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.620713Z","file":"common-main.c","line":38,"evt":"2","exe":"2.20.1.155.g426c96fcdb"} {"event":"start","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621027Z","file":"common-main.c","line":39,"t_abs":0.001173,"argv":["git","version"]} {"event":"cmd_name","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621122Z","file":"git.c","line":432,"name":"version","hierarchy":"version"} {"event":"exit","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621236Z","file":"git.c","line":662,"t_abs":0.001227,"code":0} @@ -616,11 +616,22 @@ only present on the "start" and "atexit" events. { "event":"version", ... - "evt":"1", # EVENT format version + "evt":"2", # EVENT format version "exe":"2.20.1.155.g426c96fcdb" # git version } ------------ +`"overload"`:: + This event is created in a sentinel file if we are overloading a target + trace directory (see the trace2.maxFiles config option). ++ +------------ +{ + "event":"overload", + ... +} +------------ + `"start"`:: This event contains the complete argv received by main(). + diff --git a/t/t0212-trace2-event.sh b/t/t0212-trace2-event.sh index 2ff97e72da..7081153a11 100755 --- a/t/t0212-trace2-event.sh +++ b/t/t0212-trace2-event.sh @@ -279,7 +279,9 @@ test_expect_success "don't overload target directory" ' ) && echo git-trace2-overload >>expected_filenames.txt && ls trace_target_dir >ls_output.txt && - test_cmp expected_filenames.txt ls_output.txt + test_cmp expected_filenames.txt ls_output.txt && + head -n1 trace_target_dir/git-trace2-overload | grep \"event\":\"version\" && + head -n2 trace_target_dir/git-trace2-overload | tail -n1 | grep \"event\":\"overload\" ' test_done diff --git a/trace2/tr2_dst.c b/trace2/tr2_dst.c index af3405f179..eedc5a332f 100644 --- a/trace2/tr2_dst.c +++ b/trace2/tr2_dst.c @@ -50,15 +50,19 @@ void tr2_dst_trace_disable(struct tr2_dst *dst) /* * Check to make sure we're not overloading the target directory with too many * files. First get the threshold (if present) from the config or envvar. If - * it's zero or unset, disable this check. Next check for the presence of a - * sentinel file, then check file count. If we are overloaded, create the - * sentinel file if it doesn't already exist. + * it's zero or unset, disable this check. Next check for the presence of a + * sentinel file, then check file count. + * + * Returns 0 if tracing should proceed as normal. Returns 1 if the sentinel file + * already exists, which means tracing should be disabled. Returns -1 if we are + * overloaded but there was no sentinel file, which means we have created and + * should write traces to the sentinel file. * * We expect that some trace processing system is gradually collecting files * from the target directory; after it removes the sentinel file we'll start * writing traces again. */ -static int tr2_dst_overloaded(const char *tgt_prefix) +static int tr2_dst_overloaded(struct tr2_dst *dst, const char *tgt_prefix) { int file_count = 0, max_files = 0, ret = 0; const char *max_files_var; @@ -97,8 +101,9 @@ static int tr2_dst_overloaded(const char *tgt_prefix) closedir(dirp); if (file_count >= tr2env_max_files) { - creat(sentinel_path.buf, 0666); - ret = 1; + dst->overloaded = 1; + dst->fd = open(sentinel_path.buf, O_WRONLY | O_CREAT | O_EXCL, 0666); + ret = -1; goto cleanup; } @@ -110,7 +115,7 @@ static int tr2_dst_overloaded(const char *tgt_prefix) static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) { - int fd; + int overloaded; const char *last_slash, *sid = tr2_sid_get(); struct strbuf path = STRBUF_INIT; size_t base_path_len; @@ -126,7 +131,19 @@ static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) strbuf_addstr(&path, sid); base_path_len = path.len; - if (tr2_dst_overloaded(tgt_prefix)) { + overloaded = tr2_dst_overloaded(dst, tgt_prefix); + if (!overloaded) { + for (attempt_count = 0; attempt_count < MAX_AUTO_ATTEMPTS; attempt_count++) { + if (attempt_count > 0) { + strbuf_setlen(&path, base_path_len); + strbuf_addf(&path, ".%d", attempt_count); + } + + dst->fd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, 0666); + if (dst->fd != -1) + break; + } + } else if (overloaded == 1) { strbuf_release(&path); if (tr2_dst_want_warning()) warning("trace2: not opening %s trace file due to too " @@ -136,18 +153,7 @@ static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) return 0; } - for (attempt_count = 0; attempt_count < MAX_AUTO_ATTEMPTS; attempt_count++) { - if (attempt_count > 0) { - strbuf_setlen(&path, base_path_len); - strbuf_addf(&path, ".%d", attempt_count); - } - - fd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, 0666); - if (fd != -1) - break; - } - - if (fd == -1) { + if (dst->fd == -1) { if (tr2_dst_want_warning()) warning("trace2: could not open '%.*s' for '%s' tracing: %s", (int) base_path_len, path.buf, @@ -161,7 +167,6 @@ static int tr2_dst_try_auto_path(struct tr2_dst *dst, const char *tgt_prefix) strbuf_release(&path); - dst->fd = fd; dst->need_close = 1; dst->initialized = 1; diff --git a/trace2/tr2_dst.h b/trace2/tr2_dst.h index 3adf3bac13..e8abe1e490 100644 --- a/trace2/tr2_dst.h +++ b/trace2/tr2_dst.h @@ -9,6 +9,7 @@ struct tr2_dst { int fd; unsigned int initialized : 1; unsigned int need_close : 1; + unsigned int overloaded : 1; }; /* diff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c index c2852d1bd2..f27bc572c5 100644 --- a/trace2/tr2_tgt_event.c +++ b/trace2/tr2_tgt_event.c @@ -10,16 +10,17 @@ #include "trace2/tr2_tgt.h" #include "trace2/tr2_tls.h" -static struct tr2_dst tr2dst_event = { TR2_SYSENV_EVENT, 0, 0, 0 }; +static struct tr2_dst tr2dst_event = { TR2_SYSENV_EVENT, 0, 0, 0, 0 }; /* - * The version number of the JSON data generated by the EVENT target - * in this source file. Update this if you make a significant change - * to the JSON fields or message structure. You probably do not need - * to update this if you just add another call to one of the existing - * TRACE2 API methods. + * The version number of the JSON data generated by the EVENT target in this + * source file. The version should be incremented if new event types are added, + * if existing fields are removed, or if there are significant changes in + * interpretation of existing events or fields. Smaller changes, such as adding + * a new field to an existing event, do not require an increment to the EVENT + * format version. */ -#define TR2_EVENT_VERSION "1" +#define TR2_EVENT_VERSION "2" /* * Region nesting limit for messages written to the event target. @@ -107,6 +108,19 @@ static void event_fmt_prepare(const char *event_name, const char *file, jw_object_intmax(jw, "repo", repo->trace2_repo_id); } +static void fn_overload_fl(const char *file, int line) +{ + const char *event_name = "overload"; + struct json_writer jw = JSON_WRITER_INIT; + + jw_object_begin(&jw, 0); + event_fmt_prepare(event_name, file, line, NULL, &jw); + jw_end(&jw); + + tr2_dst_write_line(&tr2dst_event, &jw.json); + jw_release(&jw); +} + static void fn_version_fl(const char *file, int line) { const char *event_name = "version"; @@ -120,6 +134,9 @@ static void fn_version_fl(const char *file, int line) tr2_dst_write_line(&tr2dst_event, &jw.json); jw_release(&jw); + + if (tr2dst_event.overloaded) + fn_overload_fl(file, line); } static void fn_start_fl(const char *file, int line, diff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c index 00b116d797..e91633468d 100644 --- a/trace2/tr2_tgt_normal.c +++ b/trace2/tr2_tgt_normal.c @@ -9,7 +9,7 @@ #include "trace2/tr2_tgt.h" #include "trace2/tr2_tls.h" -static struct tr2_dst tr2dst_normal = { TR2_SYSENV_NORMAL, 0, 0, 0 }; +static struct tr2_dst tr2dst_normal = { TR2_SYSENV_NORMAL, 0, 0, 0, 0 }; /* * Use the TR2_SYSENV_NORMAL_BRIEF setting to omit the "