From patchwork Thu Jul 20 16:30:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90DDDEB64DC for ; Thu, 20 Jul 2023 16:31:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DBF728012F; Thu, 20 Jul 2023 12:31:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1654E28004C; Thu, 20 Jul 2023 12:31:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED24D28012F; Thu, 20 Jul 2023 12:31:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D6D7A28004C for ; Thu, 20 Jul 2023 12:31:57 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7C591B0EA3 for ; Thu, 20 Jul 2023 16:31:57 +0000 (UTC) X-FDA: 81032531874.22.A1586E4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 923D74000D for ; Thu, 20 Jul 2023 16:31:55 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LhrWsx++; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870715; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FgooZspr4TQIjuc/tHX/XT4D8n/wo2nwvhkclMi4+iU=; b=bNwd+hHITwFKQQf+3FOZcj1EDY7Wv+DdLICfwT3TEA1V2nufLlRnCeYbpLSnCZr4GCPT2r d7mdZPQu4U6ycfmC9XBeLMhgsCzIUlrQBnuBTh1LjUu4atKeZMt4M97E3ul6LFdF4TZs6z UHsSpCN6TpZ7nibPizC37b8eGItBq90= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LhrWsx++; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870715; a=rsa-sha256; cv=none; b=eiFPWXf3VVmAku3oarG2BFVyrfe1uHdnBLp8OGn1xAiQRBgV4NcWGGydxsQ5Bo2CVwZQHe nWi1Zb35nyiqll4zTs5mJTd6ifPkAO9ymjqOGomuHKsge1DVdkil5RZ17KqhekEN9LNfLs AUKpNPgDTDC3d8bCdHui1Vln2nvgFrs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FgooZspr4TQIjuc/tHX/XT4D8n/wo2nwvhkclMi4+iU=; b=LhrWsx++trxf0wDP9Oo+0WuzdhIrPKJZa2A2ROrJP9nIC8exZpk5F5wfhfq9qtW/pwRtni OL2MCauiwYPCkdacwkcByKoDtk4UGjIGjCiYGOfoI0npMrEfO79eFD0nCe2N0FGQ6Y9SCT hJPpGkE7lACQOFhzAFsyzmlCfmIhAsw= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-558-fsF-jKQQP5yEcGDp2gQ-Yw-1; Thu, 20 Jul 2023 12:31:53 -0400 X-MC-Unique: fsF-jKQQP5yEcGDp2gQ-Yw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 986C22812951; Thu, 20 Jul 2023 16:31:49 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6C05640C206F; Thu, 20 Jul 2023 16:31:40 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 01/20] tracing/filters: Dynamically allocate filter_pred.regex Date: Thu, 20 Jul 2023 17:30:37 +0100 Message-Id: <20230720163056.2564824-2-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 923D74000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1jj96rt4x1eekw7ucnwu5i5okkk9suu7 X-HE-Tag: 1689870715-31811 X-HE-Meta: U2FsdGVkX1/NcvHAJHSTTzElbzH2E39TAhZG4/lai0G51/0HaAfHpR1fTL0Xmm02GpYj5ArujmgxDv5IoBRjFAseiHF3hPfqulYPCXOqDpMOQ2iUfQolRcQxyhf0piWNzln6h4AiFczm15cROpVhNEC/Hyp1+k8AmPuXB5sprmsypzzvU0cfPhxT3etd2UM3mrhxPfPWQVO7R2upsvOgtkmeDxokz97YgCHmmi8hwH79s2IVVOZfAy8MWcQUB0nP9dQgoTUIT1jof6bZ6zBhW06nPeXEvBMDo6GlErtsmr6Ux19bk/YkIYqQxQgH9WxwHwQyuA7aHDUMrx+UbiT4qq7iUCb0T0pAjXP64XO8N5bvkEY7CALZxLkK7GDCaR8qNZClaHpRksXnRkpd0Za0zBMkoA8rrE3xpjS/tRJveY6rbKQSeI5KXjYLUlZ9yyuFZi+KqWWmWgKcmNWp9fLI8CR/PjmyIgZ0OHEROCfM1jAQ9oZ3hRCCDXPWbnxOiAfGUL/7WDemqxvZltEzy015UCoFUbBPXyEpy1Qy9FpGNxlcnJBtmJPlXzF7Hx/vc0IjBAXGP4M6Da+YFFC0/zi8yYlqCXi0IJg+EmJoJxmVVbWNqcIL6ff9kelcLpvLSgcFvWyfxAd+5xgED6D0+Qu5zVrJhO9Cqkxdo5qMFtJtmSlbcZL7w2lwoy5sj4QqtF4ARixqUXUzjHCibrKX/qAD7AvlK6Zl2ixT6mdNkko2VvGCyZJKhrDkcH6UjLJboYnLpNx24M1Cei1iH/T1B5+/AvBa/ScGIdNgh/I4ySGBdkJyAgCqGuwJNtOfRD2pxsCDapF6BiUECrsU+nEc77GQBbvQL5GNTaRx0zRNi7EibtHTh5FYYyLMFmMXDaBIk/22iyNTTA+URp6USrFm9EorQAS1kECvihUip3Zyhe1tGD5a41lBGTto/pltE+m/0QJM63v4P2XnXSKcIqUkXoL 6ZyNB6Hx VB7zwTq3AhSoxTq+DDsa7/3FHha5l4l4DQ9+9hz2MMW1mPY8/MxHVLuKYnmo3P67+HKeIV7ZcnyZ82YCW1/Zz5zLa9QTEt4ek86ur5b4D3perxoq5QmegxUe6AF1MI3Fo2ULoBraMV99GgIyspxv2kR93p2AXJ2VSRKKdh9s8J/eExC9MW5w5/BR/G9Z3YUFX4tSGjbiE/40aY1LrthJ4m8BHe2MfgTOtMu+oVwje2EGknl+kk1zG34q8Nwrngi6YRgY0gg8V1BX1Zz4+NH5k89Det5m3vAx12PzB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Every predicate allocation includes a MAX_FILTER_STR_VAL (256) char array in the regex field, even if the predicate function does not use the field. A later commit will introduce a dynamically allocated cpumask to struct filter_pred, which will require a dedicated freeing function. Bite the bullet and make filter_pred.regex dynamically allocated. While at it, reorder the fields of filter_pred to fill in the byte holes. The struct now fits on a single cacheline. No change in behaviour intended. The kfree()'s were patched via Coccinelle: @@ struct filter_pred *pred; @@ -kfree(pred); +free_predicate(pred); Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 64 ++++++++++++++++++------------ 1 file changed, 39 insertions(+), 25 deletions(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 1dad64267878c..91fc9990107f1 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -70,15 +70,15 @@ enum filter_pred_fn { }; struct filter_pred { - enum filter_pred_fn fn_num; - u64 val; - u64 val2; - struct regex regex; + struct regex *regex; unsigned short *ops; struct ftrace_event_field *field; - int offset; + u64 val; + u64 val2; + enum filter_pred_fn fn_num; + int offset; int not; - int op; + int op; }; /* @@ -186,6 +186,14 @@ enum { PROCESS_OR = 4, }; +static void free_predicate(struct filter_pred *pred) +{ + if (pred) { + kfree(pred->regex); + kfree(pred); + } +} + /* * Without going into a formal proof, this explains the method that is used in * parsing the logical expressions. @@ -623,7 +631,7 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, kfree(inverts); if (prog_stack) { for (i = 0; prog_stack[i].pred; i++) - kfree(prog_stack[i].pred); + free_predicate(prog_stack[i].pred); kfree(prog_stack); } return ERR_PTR(ret); @@ -750,7 +758,7 @@ static int filter_pred_string(struct filter_pred *pred, void *event) char *addr = (char *)(event + pred->offset); int cmp, match; - cmp = pred->regex.match(addr, &pred->regex, pred->regex.field_len); + cmp = pred->regex->match(addr, pred->regex, pred->regex->field_len); match = cmp ^ pred->not; @@ -763,7 +771,7 @@ static __always_inline int filter_pchar(struct filter_pred *pred, char *str) int len; len = strlen(str) + 1; /* including tailing '\0' */ - cmp = pred->regex.match(str, &pred->regex, len); + cmp = pred->regex->match(str, pred->regex, len); match = cmp ^ pred->not; @@ -813,7 +821,7 @@ static int filter_pred_strloc(struct filter_pred *pred, void *event) char *addr = (char *)(event + str_loc); int cmp, match; - cmp = pred->regex.match(addr, &pred->regex, str_len); + cmp = pred->regex->match(addr, pred->regex, str_len); match = cmp ^ pred->not; @@ -836,7 +844,7 @@ static int filter_pred_strrelloc(struct filter_pred *pred, void *event) char *addr = (char *)(&item[1]) + str_loc; int cmp, match; - cmp = pred->regex.match(addr, &pred->regex, str_len); + cmp = pred->regex->match(addr, pred->regex, str_len); match = cmp ^ pred->not; @@ -874,7 +882,7 @@ static int filter_pred_comm(struct filter_pred *pred, void *event) { int cmp; - cmp = pred->regex.match(current->comm, &pred->regex, + cmp = pred->regex->match(current->comm, pred->regex, TASK_COMM_LEN); return cmp ^ pred->not; } @@ -1004,7 +1012,7 @@ enum regex_type filter_parse_regex(char *buff, int len, char **search, int *not) static void filter_build_regex(struct filter_pred *pred) { - struct regex *r = &pred->regex; + struct regex *r = pred->regex; char *search; enum regex_type type = MATCH_FULL; @@ -1169,7 +1177,7 @@ static void free_prog(struct event_filter *filter) return; for (i = 0; prog[i].pred; i++) - kfree(prog[i].pred); + free_predicate(prog[i].pred); kfree(prog); } @@ -1553,9 +1561,12 @@ static int parse_pred(const char *str, void *data, goto err_free; } - pred->regex.len = len; - strncpy(pred->regex.pattern, str + s, len); - pred->regex.pattern[len] = 0; + pred->regex = kzalloc(sizeof(*pred->regex), GFP_KERNEL); + if (!pred->regex) + goto err_mem; + pred->regex->len = len; + strncpy(pred->regex->pattern, str + s, len); + pred->regex->pattern[len] = 0; /* This is either a string, or an integer */ } else if (str[i] == '\'' || str[i] == '"') { @@ -1597,9 +1608,12 @@ static int parse_pred(const char *str, void *data, goto err_free; } - pred->regex.len = len; - strncpy(pred->regex.pattern, str + s, len); - pred->regex.pattern[len] = 0; + pred->regex = kzalloc(sizeof(*pred->regex), GFP_KERNEL); + if (!pred->regex) + goto err_mem; + pred->regex->len = len; + strncpy(pred->regex->pattern, str + s, len); + pred->regex->pattern[len] = 0; filter_build_regex(pred); @@ -1608,7 +1622,7 @@ static int parse_pred(const char *str, void *data, } else if (field->filter_type == FILTER_STATIC_STRING) { pred->fn_num = FILTER_PRED_FN_STRING; - pred->regex.field_len = field->size; + pred->regex->field_len = field->size; } else if (field->filter_type == FILTER_DYN_STRING) { pred->fn_num = FILTER_PRED_FN_STRLOC; @@ -1691,10 +1705,10 @@ static int parse_pred(const char *str, void *data, return i; err_free: - kfree(pred); + free_predicate(pred); return -EINVAL; err_mem: - kfree(pred); + free_predicate(pred); return -ENOMEM; } @@ -2287,8 +2301,8 @@ static int ftrace_function_set_filter_pred(struct filter_pred *pred, return ret; return __ftrace_function_set_filter(pred->op == OP_EQ, - pred->regex.pattern, - pred->regex.len, + pred->regex->pattern, + pred->regex->len, data); } From patchwork Thu Jul 20 16:30:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 033D5EB64DC for ; Thu, 20 Jul 2023 16:32:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A88F280130; Thu, 20 Jul 2023 12:32:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7310928004C; Thu, 20 Jul 2023 12:32:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AA44280130; Thu, 20 Jul 2023 12:32:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 480AC28004C for ; Thu, 20 Jul 2023 12:32:17 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 20485160216 for ; Thu, 20 Jul 2023 16:32:17 +0000 (UTC) X-FDA: 81032532714.29.D9D9AE1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 520934000D for ; Thu, 20 Jul 2023 16:32:15 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UPvUlz5P; spf=pass (imf01.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870735; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C96oWrbX6n5TlLE7O7XWEwueAW9U6q6ni5+a3A4c/ko=; b=s0yYHqjcuFBZoEyBJ8BvELu9BjjoB+aA+mBOU6g4Vfd+jfZPeHX46KGPOovS27C9xGArio 4cnSzPHdXZwhhDabpRtZz+WxnSPgoPOQ0ZinHJMIkUFIDj4YXuzbRARt9ZfKV8H2wX40Tw jeESrrIm69U24xPoBMy20zPxBVFNkpA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870735; a=rsa-sha256; cv=none; b=Dv/6qYs+jZPveXMArBm1+otuZwpK7N2syi6VYmtzBefWi1wsMSIPV6ifV80BXMDVm2svf8 DJ84RPQzsSGErVkt+PCEg2cFSsCUmnnOk5orw4t5dsqA4c8ZhnrZMrMIX6kOJRl+TkCwNR 5MYJmTwvtNNbSVMdW7AZt3kXFbRG460= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UPvUlz5P; spf=pass (imf01.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C96oWrbX6n5TlLE7O7XWEwueAW9U6q6ni5+a3A4c/ko=; b=UPvUlz5PioKud+LCsIM+pWBk3efICAb7bIQZZemCybVmnjLcwuNdaoNMZwgIKb+bgrznjq ThGHDmbgMI0T1Hy2nkVaQ2sZ0dJVTamAERqqkk4dY139XUQEGxpnNpJcCEpOwZ4mw2rA0e 3fkJemkaeyKbwaGEYi10oODcXx0+6Go= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-649-TP5rCbtOO8Oz4kDb2HxfDg-1; Thu, 20 Jul 2023 12:32:11 -0400 X-MC-Unique: TP5rCbtOO8Oz4kDb2HxfDg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3FC5B1C05EA8; Thu, 20 Jul 2023 16:32:08 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C8AC440C206F; Thu, 20 Jul 2023 16:31:49 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 02/20] tracing/filters: Enable filtering a cpumask field by another cpumask Date: Thu, 20 Jul 2023 17:30:38 +0100 Message-Id: <20230720163056.2564824-3-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Stat-Signature: rr4gpicqd55jeurjppi7gzr9z6yf157y X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 520934000D X-Rspam-User: X-HE-Tag: 1689870735-546168 X-HE-Meta: U2FsdGVkX18x8ugb6u388LcjCz1wncLPGAikfP17wUGaBa9UMmOcDATIfOqdPGT0EODWnyefLbnuiu5grKzeckL0qRaZ2CwUTDbwINmE9ALR/pa8CqcC5TFx7ijN8R+BY9qUJ4gov8iIPtakeONGDPNq2+Oib0DNEd31gNz5w3doMOyx5POG56j4UTv+cKkC/1QXdFiuKdFyxt0nj+1D6pX4lgyd/kxzlyPKVe+LlOTnJElo+OHT9FxPEY+pXJIwc5K+Ot2VjHNkITfd8eivrhTqRTUTay/IHiJS5zTzhNuSttTAASz26Jwj9OyGv6s+hF2tsR2ns4suA79m2Gg/u64xvRyBmUM+EMkliNMjCyIsPQ34j8vqChXjfRgvk16McViNa3+ZlAui+zRDn9dZ5AGXFpdcM3g77COYMtpiKQm9gZbooXnJAgaZsGh8rI2ucGbwA2lsJCfxE0ONC6JurjT1V8vMzLHxwwAzopGdwwG8/z8ByHP+CWMmd/Wr8gff1clcG9b7WfJZdX5+Sh4/LgTlhEFJrRIT2Tr4IkObgb35BXgALsf4UF7SXEm/PmpHH0hQ+CY2qM5RA81L5uAlTiD11/nn/tWh4tCg9XoGeA0s7Tq0jkaUtsg6ZZCwQXLU8oYu5rgoE6mOJ3GjKQ+X9gad3uvdI13t7p0wI6GdLWfd0dM3mqXbRqSPkgfk7FyyWjadnTKzkSRv9DbZ0fh1D7XoLdPvrjFV/ruAbTMkYfDjkFH5rEOj5eYjbOenW6pTZeeUy6BCRy3N+if2plTtlOQFTgQ3STd4ewvbP0ZMl0fk7hpi98wssbuXZf3uedlu31QmbIaN91fTHl2abfg9K5T31ClypkybgfEpB+sBJwphUtK0e9ZSnhcpIxaQeWriB6oEHlTbjDqGjFmkiLeNnSRBKs+umECONfStRIAqba5W2xKAuKQJeEGO4HpXbRFg8D9Z5DCiw/l3m0tKOQZ TdX/bXXB nlFOJSTn1FoTOs7JGi0HArLLwkPw7FvooQwnQ2er3UtExfoCDTjBYwI545KBnT75tkUR7dryFFwzKsJPOPWExDFgDwdbnyAdN2tVhXSOQ3Losl5/6SLa8tWAOXMgvSYEngUKlcawLCgqCovVmoyWduziYWWTxa8oFUx2lamZxYq+MXN8cNwTsDVMS0GBTLIz9lkanbjeIdyUHDhXOrAqpjJY2h/E3gpGMuYuWE7pQ/NwmCYZjfggOYrxRnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The recently introduced ipi_send_cpumask trace event contains a cpumask field, but it currently cannot be used in filter expressions. Make event filtering aware of cpumask fields, and allow these to be filtered by a user-provided cpumask. The user-provided cpumask is to be given in cpulist format and wrapped as: "CPUS{$cpulist}". The use of curly braces instead of parentheses is to prevent predicate_parse() from parsing the contents of CPUS{...} as a full-fledged predicate subexpression. This enables e.g.: $ trace-cmd record -e 'ipi_send_cpumask' -f 'cpumask & CPUS{2,4,6,8-32}' Signed-off-by: Valentin Schneider --- include/linux/trace_events.h | 1 + kernel/trace/trace_events_filter.c | 97 +++++++++++++++++++++++++++++- 2 files changed, 96 insertions(+), 2 deletions(-) diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h index 3930e676436c9..302be73918336 100644 --- a/include/linux/trace_events.h +++ b/include/linux/trace_events.h @@ -807,6 +807,7 @@ enum { FILTER_RDYN_STRING, FILTER_PTR_STRING, FILTER_TRACE_FN, + FILTER_CPUMASK, FILTER_COMM, FILTER_CPU, FILTER_STACKTRACE, diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 91fc9990107f1..cb1863dfa280b 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -64,6 +64,7 @@ enum filter_pred_fn { FILTER_PRED_FN_PCHAR_USER, FILTER_PRED_FN_PCHAR, FILTER_PRED_FN_CPU, + FILTER_PRED_FN_CPUMASK, FILTER_PRED_FN_FUNCTION, FILTER_PRED_FN_, FILTER_PRED_TEST_VISITED, @@ -71,6 +72,7 @@ enum filter_pred_fn { struct filter_pred { struct regex *regex; + struct cpumask *mask; unsigned short *ops; struct ftrace_event_field *field; u64 val; @@ -94,6 +96,8 @@ struct filter_pred { C(TOO_MANY_OPEN, "Too many '('"), \ C(TOO_MANY_CLOSE, "Too few '('"), \ C(MISSING_QUOTE, "Missing matching quote"), \ + C(MISSING_BRACE_OPEN, "Missing '{'"), \ + C(MISSING_BRACE_CLOSE, "Missing '}'"), \ C(OPERAND_TOO_LONG, "Operand too long"), \ C(EXPECT_STRING, "Expecting string field"), \ C(EXPECT_DIGIT, "Expecting numeric field"), \ @@ -103,6 +107,7 @@ struct filter_pred { C(BAD_SUBSYS_FILTER, "Couldn't find or set field in one of a subsystem's events"), \ C(TOO_MANY_PREDS, "Too many terms in predicate expression"), \ C(INVALID_FILTER, "Meaningless filter expression"), \ + C(INVALID_CPULIST, "Invalid cpulist"), \ C(IP_FIELD_ONLY, "Only 'ip' field is supported for function trace"), \ C(INVALID_VALUE, "Invalid value (did you forget quotes)?"), \ C(NO_FUNCTION, "Function not found"), \ @@ -190,6 +195,7 @@ static void free_predicate(struct filter_pred *pred) { if (pred) { kfree(pred->regex); + kfree(pred->mask); kfree(pred); } } @@ -877,6 +883,26 @@ static int filter_pred_cpu(struct filter_pred *pred, void *event) } } +/* Filter predicate for cpumask field vs user-provided cpumask */ +static int filter_pred_cpumask(struct filter_pred *pred, void *event) +{ + u32 item = *(u32 *)(event + pred->offset); + int loc = item & 0xffff; + const struct cpumask *mask = (event + loc); + const struct cpumask *cmp = pred->mask; + + switch (pred->op) { + case OP_EQ: + return cpumask_equal(mask, cmp); + case OP_NE: + return !cpumask_equal(mask, cmp); + case OP_BAND: + return cpumask_intersects(mask, cmp); + default: + return 0; + } +} + /* Filter predicate for COMM. */ static int filter_pred_comm(struct filter_pred *pred, void *event) { @@ -1244,8 +1270,12 @@ static void filter_free_subsystem_filters(struct trace_subsystem_dir *dir, int filter_assign_type(const char *type) { - if (strstr(type, "__data_loc") && strstr(type, "char")) - return FILTER_DYN_STRING; + if (strstr(type, "__data_loc")) { + if (strstr(type, "char")) + return FILTER_DYN_STRING; + if (strstr(type, "cpumask_t")) + return FILTER_CPUMASK; + } if (strstr(type, "__rel_loc") && strstr(type, "char")) return FILTER_RDYN_STRING; @@ -1357,6 +1387,8 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) return filter_pred_pchar(pred, event); case FILTER_PRED_FN_CPU: return filter_pred_cpu(pred, event); + case FILTER_PRED_FN_CPUMASK: + return filter_pred_cpumask(pred, event); case FILTER_PRED_FN_FUNCTION: return filter_pred_function(pred, event); case FILTER_PRED_TEST_VISITED: @@ -1568,6 +1600,67 @@ static int parse_pred(const char *str, void *data, strncpy(pred->regex->pattern, str + s, len); pred->regex->pattern[len] = 0; + } else if (!strncmp(str + i, "CPUS", 4)) { + unsigned int maskstart; + char *tmp; + + switch (field->filter_type) { + case FILTER_CPUMASK: + break; + default: + parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i); + goto err_free; + } + + switch (op) { + case OP_EQ: + case OP_NE: + case OP_BAND: + break; + default: + parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i); + goto err_free; + } + + /* Skip CPUS */ + i += 4; + if (str[i++] != '{') { + parse_error(pe, FILT_ERR_MISSING_BRACE_OPEN, pos + i); + goto err_free; + } + maskstart = i; + + /* Walk the cpulist until closing } */ + for (; str[i] && str[i] != '}'; i++); + if (str[i] != '}') { + parse_error(pe, FILT_ERR_MISSING_BRACE_CLOSE, pos + i); + goto err_free; + } + + if (maskstart == i) { + parse_error(pe, FILT_ERR_INVALID_CPULIST, pos + i); + goto err_free; + } + + /* Copy the cpulist between { and } */ + tmp = kmalloc((i - maskstart) + 1, GFP_KERNEL); + strscpy(tmp, str + maskstart, (i - maskstart) + 1); + + pred->mask = kzalloc(cpumask_size(), GFP_KERNEL); + if (!pred->mask) + goto err_mem; + + /* Now parse it */ + if (cpulist_parse(tmp, pred->mask)) { + parse_error(pe, FILT_ERR_INVALID_CPULIST, pos + i); + goto err_free; + } + + /* Move along */ + i++; + if (field->filter_type == FILTER_CPUMASK) + pred->fn_num = FILTER_PRED_FN_CPUMASK; + /* This is either a string, or an integer */ } else if (str[i] == '\'' || str[i] == '"') { char q = str[i]; From patchwork Thu Jul 20 16:30:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 097B3EB64DC for ; Thu, 20 Jul 2023 16:32:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F556280131; Thu, 20 Jul 2023 12:32:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77EFF28004C; Thu, 20 Jul 2023 12:32:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AAA7280131; Thu, 20 Jul 2023 12:32:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 46BB628004C for ; Thu, 20 Jul 2023 12:32:24 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 000811401CA for ; Thu, 20 Jul 2023 16:32:23 +0000 (UTC) X-FDA: 81032532966.28.07AA78E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 38D7A40013 for ; Thu, 20 Jul 2023 16:32:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Iqa7XZfA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870742; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D/8mq5c0cbFoNC4kMb4OaVtFnN/bUBn9qmubvCUjumA=; b=fwlBUMXhfAifUnon6sK/kBjA/Qi/C/66fMgEqPkB0wSxNb2TSmwohR9qvQNvEgXHsd2Q3a DM15vYXd3W0ay5+op9Fdd9+Oy0J9gwmq19Q1msDqOBmpoFXKEInpPokiiBXOUQUfOB0hGt 2D5EfzhOATU6gMOEVxkXLVVnacBjsPs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Iqa7XZfA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870742; a=rsa-sha256; cv=none; b=alzCKaLeouqEGGIor+BFDAqdWXk8N4N3+bvhjaLcCdlRONacBkvt8LGn7HJIiviyjiet9S GGnYqzD1dN6aEYLExrGakgbHQVmJZ2yjzwH1hb0zf2A6/ZZrTP8+22JXII+xCBxbNHaGlR gAd7T+D+E29EACQa0VwQy6+HbKTOgdQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D/8mq5c0cbFoNC4kMb4OaVtFnN/bUBn9qmubvCUjumA=; b=Iqa7XZfAbrPeqeIHzZdxZrC8conlXzvzc+DavlGijcQ6qvfMzkxZTMpfQQ/l7woa+R/0YI b43y0Dtnx8Z5ziWnJZ9legigCfrecTGZm2P1dL5Ik3evanDom/jeK2NzeVov2J4mvrR4vN Hq6yfq5m6WNAvGU5DgZzmEwQly5Hb0w= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-99-y6uyriohNKyYlI_N0r9yPA-1; Thu, 20 Jul 2023 12:32:19 -0400 X-MC-Unique: y6uyriohNKyYlI_N0r9yPA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 84E3A1C05EAE; Thu, 20 Jul 2023 16:32:16 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 83E5540C2070; Thu, 20 Jul 2023 16:32:08 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 03/20] tracing/filters: Enable filtering a scalar field by a cpumask Date: Thu, 20 Jul 2023 17:30:39 +0100 Message-Id: <20230720163056.2564824-4-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Stat-Signature: fudtsypj679ceexqms7gnsrtjga6k5ry X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 38D7A40013 X-HE-Tag: 1689870741-345958 X-HE-Meta: U2FsdGVkX1+DHhEgdk5Jklr5+EKX5t3l1UaGGeMEx5iOVhzDGjO28sHTVvustAVuRq1IZZ89DyKuFFoPuIsYFF9jY+C9G/Ob9JJ7qUFmDaUiuPfbOW94jhe4Wsu8SeZDUNYNDrqsPUjpC+s2UMHOxg1okxThfukQoGh4Dr9KDQANDz6NhI45IX/WBuoH00pL1/CLb6fLkBNvuQFmBxHORN9oUMCWJGQkvj1jZT2VJpIajn65sVhW2KCaflPFJnwzWRrHm8BAfQPrPwktHsAeenL2ZZoNKtFVgSpOx6r3EiMK0PdCcVPabdVtWxj98s+Z19g9W2icuOAexewu9Eken7p4psVO9qgTECb4/3CeVgpJuFI6jWvT4u2nU+hD5B2iRX2PInmPu5sAg4LELooomLUvIlfHwKzjg1gZsuBmZ0irA6QUXvbRzsIPuKz1kNeCD7x9KqcaUtv+3Vf5PGR3KQJeZSPCrVmf4nWLGwoQPhchIPXkXYyjHANM3zCRYEVpX621sEnqTZmo8BLgTpxuUamCQnRcDzd53dVSweBRs6KOkZEXu2kiBV+lPmnxjrbAGTXI8sBVFdbLPxVVrqGtN3M9sH0R9h+Tv1KyjjxkBklWN9uXSwf+OYsSaPo6d/XVAAWd/+tbamtC70gnzXKWp97xCizkDqNAaTX0arHZuMKC66TJ5BnYpCxiC02sMtcRQzHh3NuUJ3BwmLMtb3zEaGqosuJ2dtQMuYluHsBlin4CNzSQYdxQz9FBTHKXnKvP28fr0UUK8mRdQk0a+WzJ/FPmspU/yOM6++htJDrA5Zik2MNtJ/fXQm6hqU2mtNhIt7QkWh7JDyAhTnWJEpoiJK8UqTbc8/XwUUJ22NZDMxGLnntpIeeyqPMJ8ZJoLm+e5RtNrbjpKfztZdxw1q//zZ2BI3OzW7hpzvsRSJ2ASAEv5zx4ZbTlI6u2LtpYLP2UKt+bynTX0HgLsy+hsXP MQM6UvMD CMsm0VqylGgH75G/RKR4p3LMeHpHfKsluXMCZ0yMVuwsBTVjmoC43p8G/FzEEO/XgTx96MH9dhBQWz7Lx1QkCtaq6Rgw4BIftmT+HzpwT7xyTaJHAN/55ZfRWzDVqwYAUeZb9VU92uwTu1Hicj9jtazqhbG6yp50Daz8iwl+06PT23ysOy4ush2R8UzbgOt28wTVK/MIwVJn9lknr2gwJjYKlrOMaDruTRHQUyDiivDGiZ6YG5hKpN9o4sA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Several events use a scalar field to denote a CPU: o sched_wakeup.target_cpu o sched_migrate_task.orig_cpu,dest_cpu o sched_move_numa.src_cpu,dst_cpu o ipi_send_cpu.cpu o ... Filtering these currently requires using arithmetic comparison functions, which can be tedious when dealing with interleaved SMT or NUMA CPU ids. Allow these to be filtered by a user-provided cpumask, which enables e.g.: $ trace-cmd record -e 'sched_wakeup' -f 'target_cpu & CPUS{2,4,6,8-32}' Signed-off-by: Valentin Schneider --- NOTE: I went with an implicit cpumask conversion of the event field, as AFAICT predicate_parse() does not support parsing the application of a function to a field (e.g. 'CPUS(target_cpu) & CPUS{2,4,6,8-32}') --- kernel/trace/trace_events_filter.c | 92 ++++++++++++++++++++++++++---- 1 file changed, 81 insertions(+), 11 deletions(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index cb1863dfa280b..1e14f801685a8 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -46,15 +46,19 @@ static const char * ops[] = { OPS }; enum filter_pred_fn { FILTER_PRED_FN_NOP, FILTER_PRED_FN_64, + FILTER_PRED_FN_64_CPUMASK, FILTER_PRED_FN_S64, FILTER_PRED_FN_U64, FILTER_PRED_FN_32, + FILTER_PRED_FN_32_CPUMASK, FILTER_PRED_FN_S32, FILTER_PRED_FN_U32, FILTER_PRED_FN_16, + FILTER_PRED_FN_16_CPUMASK, FILTER_PRED_FN_S16, FILTER_PRED_FN_U16, FILTER_PRED_FN_8, + FILTER_PRED_FN_8_CPUMASK, FILTER_PRED_FN_S8, FILTER_PRED_FN_U8, FILTER_PRED_FN_COMM, @@ -643,6 +647,39 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, return ERR_PTR(ret); } +static inline int +do_filter_cpumask(int op, const struct cpumask *mask, const struct cpumask *cmp) +{ + switch (op) { + case OP_EQ: + return cpumask_equal(mask, cmp); + case OP_NE: + return !cpumask_equal(mask, cmp); + case OP_BAND: + return cpumask_intersects(mask, cmp); + default: + return 0; + } +} + +/* Optimisation of do_filter_cpumask() for scalar fields */ +static inline int +do_filter_scalar_cpumask(int op, unsigned int cpu, const struct cpumask *mask) +{ + switch (op) { + case OP_EQ: + return cpumask_test_cpu(cpu, mask) && + cpumask_nth(1, mask) >= nr_cpu_ids; + case OP_NE: + return !cpumask_test_cpu(cpu, mask) || + cpumask_nth(1, mask) < nr_cpu_ids; + case OP_BAND: + return cpumask_test_cpu(cpu, mask); + default: + return 0; + } +} + enum pred_cmp_types { PRED_CMP_TYPE_NOP, PRED_CMP_TYPE_LT, @@ -686,6 +723,18 @@ static int filter_pred_##type(struct filter_pred *pred, void *event) \ } \ } +#define DEFINE_CPUMASK_COMPARISON_PRED(size) \ +static int filter_pred_##size##_cpumask(struct filter_pred *pred, void *event) \ +{ \ + u##size *addr = (u##size *)(event + pred->offset); \ + unsigned int cpu = *addr; \ + \ + if (cpu >= nr_cpu_ids) \ + return 0; \ + \ + return do_filter_scalar_cpumask(pred->op, cpu, pred->mask); \ +} + #define DEFINE_EQUALITY_PRED(size) \ static int filter_pred_##size(struct filter_pred *pred, void *event) \ { \ @@ -707,6 +756,11 @@ DEFINE_COMPARISON_PRED(u16); DEFINE_COMPARISON_PRED(s8); DEFINE_COMPARISON_PRED(u8); +DEFINE_CPUMASK_COMPARISON_PRED(64); +DEFINE_CPUMASK_COMPARISON_PRED(32); +DEFINE_CPUMASK_COMPARISON_PRED(16); +DEFINE_CPUMASK_COMPARISON_PRED(8); + DEFINE_EQUALITY_PRED(64); DEFINE_EQUALITY_PRED(32); DEFINE_EQUALITY_PRED(16); @@ -891,16 +945,7 @@ static int filter_pred_cpumask(struct filter_pred *pred, void *event) const struct cpumask *mask = (event + loc); const struct cpumask *cmp = pred->mask; - switch (pred->op) { - case OP_EQ: - return cpumask_equal(mask, cmp); - case OP_NE: - return !cpumask_equal(mask, cmp); - case OP_BAND: - return cpumask_intersects(mask, cmp); - default: - return 0; - } + return do_filter_cpumask(pred->op, mask, cmp); } /* Filter predicate for COMM. */ @@ -1351,24 +1396,32 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) switch (pred->fn_num) { case FILTER_PRED_FN_64: return filter_pred_64(pred, event); + case FILTER_PRED_FN_64_CPUMASK: + return filter_pred_64_cpumask(pred, event); case FILTER_PRED_FN_S64: return filter_pred_s64(pred, event); case FILTER_PRED_FN_U64: return filter_pred_u64(pred, event); case FILTER_PRED_FN_32: return filter_pred_32(pred, event); + case FILTER_PRED_FN_32_CPUMASK: + return filter_pred_32_cpumask(pred, event); case FILTER_PRED_FN_S32: return filter_pred_s32(pred, event); case FILTER_PRED_FN_U32: return filter_pred_u32(pred, event); case FILTER_PRED_FN_16: return filter_pred_16(pred, event); + case FILTER_PRED_FN_16_CPUMASK: + return filter_pred_16_cpumask(pred, event); case FILTER_PRED_FN_S16: return filter_pred_s16(pred, event); case FILTER_PRED_FN_U16: return filter_pred_u16(pred, event); case FILTER_PRED_FN_8: return filter_pred_8(pred, event); + case FILTER_PRED_FN_8_CPUMASK: + return filter_pred_8_cpumask(pred, event); case FILTER_PRED_FN_S8: return filter_pred_s8(pred, event); case FILTER_PRED_FN_U8: @@ -1606,6 +1659,7 @@ static int parse_pred(const char *str, void *data, switch (field->filter_type) { case FILTER_CPUMASK: + case FILTER_OTHER: break; default: parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i); @@ -1658,8 +1712,24 @@ static int parse_pred(const char *str, void *data, /* Move along */ i++; - if (field->filter_type == FILTER_CPUMASK) + if (field->filter_type == FILTER_CPUMASK) { pred->fn_num = FILTER_PRED_FN_CPUMASK; + } else { + switch (field->size) { + case 8: + pred->fn_num = FILTER_PRED_FN_64_CPUMASK; + break; + case 4: + pred->fn_num = FILTER_PRED_FN_32_CPUMASK; + break; + case 2: + pred->fn_num = FILTER_PRED_FN_16_CPUMASK; + break; + case 1: + pred->fn_num = FILTER_PRED_FN_8_CPUMASK; + break; + } + } /* This is either a string, or an integer */ } else if (str[i] == '\'' || str[i] == '"') { From patchwork Thu Jul 20 16:30:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97B1BEB64DA for ; Thu, 20 Jul 2023 16:32:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2ED8F280132; Thu, 20 Jul 2023 12:32:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 275C628004C; Thu, 20 Jul 2023 12:32:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C882280132; Thu, 20 Jul 2023 12:32:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EEDA428004C for ; Thu, 20 Jul 2023 12:32:40 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AB58F1201DC for ; Thu, 20 Jul 2023 16:32:40 +0000 (UTC) X-FDA: 81032533680.29.2F77E9A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 34D7F1C0031 for ; Thu, 20 Jul 2023 16:32:37 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nd8+fnSy; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870758; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ru1xpHoly8RVJpDGIBIPQWc2v6Ebq5LepRuPaIIUDyE=; b=GMdh7jPgGDdPpoqx5FIr4QS/HwQ3UTsi/pNSgoSdsP2e3fPrPwkuW4TW1L6FaB1k8Pl4LT 8KMYulJJ83I1gG59vX1aO0Xlw6TpVxO+PpyBtAdD/NcZrLaRLYzUztnH0wJ+qqPd0BDXVw pOPnjivBKfeTpisMwCGPFAGYRTiTW0k= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nd8+fnSy; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870758; a=rsa-sha256; cv=none; b=zyFrlIAqBkeET2nKpHa0t1IshjX34KJX+DWpcg7oL7HUsXjiJjkJDrWze0a9eJQgFpUUQo vAtqwRcYJyBDuk+t/X5f0+VMMG+0L44FI54EUlgf2qkQX/M3X/QbHLruZ9ieavV5WqKlVC HVeToCJ34yNoGnukpFklTq3qCOx0UzM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870757; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ru1xpHoly8RVJpDGIBIPQWc2v6Ebq5LepRuPaIIUDyE=; b=Nd8+fnSyrqK5jkh5DSLtui61/xLP/CZtRAon3lWFGAhfQHQN6Ztp8cECi4f118o/u7BXE/ qrrW71K3/S0RNkB67KJA9CN/lUD36gOIDKacJAZG0ftzxu37rl2tdqpKdoYFUttcWW5ecg aSvbJi0peh3Q+VY5AUuljloJlZzcMPg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-468-L0PDvKruOe6zKXZsgB5siQ-1; Thu, 20 Jul 2023 12:32:33 -0400 X-MC-Unique: L0PDvKruOe6zKXZsgB5siQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B209B1044590; Thu, 20 Jul 2023 16:32:30 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B4F2E40C206F; Thu, 20 Jul 2023 16:32:16 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 04/20] tracing/filters: Enable filtering the CPU common field by a cpumask Date: Thu, 20 Jul 2023 17:30:40 +0100 Message-Id: <20230720163056.2564824-5-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 34D7F1C0031 X-Stat-Signature: idntrwze3ugah5jp19cz4uuib7j1qx6n X-Rspam-User: X-HE-Tag: 1689870757-456753 X-HE-Meta: U2FsdGVkX1+a+tdgQOYwePmgVV7qtWMbGEF4i2alo7VDVrQptqxQr4MpsRE021k0Ey6q7EmNI6xucY9wCNsAQcOkybF8e1iIQ7ASdbfBPe8g0OYQSTHWBPZcbZjI0PURJX+V18QJ5KbqWO3tKw/7sta26ZhNLEK2UnwLIkbhCcPZiFGQxYxqmlzteCIHjNBurKPEmjac5k5mkpSyFQOwOs68nw1ybtF5WotmVlWGZdspLNasyVpPEBPz96AVJ95myQqXdhUPCPUSWPimlodj7asDZg+FuYDnONk11ub9XpiY2Zh0LEFwtsCIujwHnaBtkLOBepsHhi8YehTfk2ce/FwHz2AA+UNzaVoTQ7FOque+DuJoVXMWL70WgK7wwE1fRlnrFjvybMu/S3UjBaX30nG88MD4ad5Puuj+EYpw+0OnACEfruPkgniQmhqIVotdxoOPvRjS9KdwlHpPb9/rZe7f/GLdT5/9xf+VFmkHPUIM9tPtzqRE4v9o5egIw8wulBxjBWCLO99SfsG7EnY8/8sTNnKl8ji3rbNIGc1t6HWokJpQv4jyz1W832/WAjfBTmhK5Mv/Lu7UTtno8sIjPXz8gcusuYEYZDtRDR0X4tsiFO8Xd3hKeNl5MVXQPBakLW6sZLAAsQLjYXyCnswzp97Xz+ErPKW+jBar/WoWeufkMcXBXF48OukyOSVy9STNstIJdYuZlaH5EPxVA+wk+9xxqVYQstoPn/0XbRWBHC6uZ8O2jeVaTAF4MYgl0X7ycgB+/llDxpGWoFVfqwibaZlmF62zfK1FMY3GqnrHaOjz/3X+1iqyCo6W5fhpvXH5WdSr9ESy9VMQygEJyf1GgtGbqW9FktII0DGeCGUdDV0HSjRhuBRVbjfPmfxrh/zKYtO6WJXC5BAJNbKlb7v4N4yhHKbybxDGOAtawdf7j6TtzzT9Z4ZhslWhR9zHw7yqojqxb3RqKebAS7Bi3nP O5vbAKMB /lp4tqvmpVZq8NBxknUbLABLcGCYRQ30uhPg4u8d+O/eqhVfX1d5uoLxjd8SGbQqCiHbPQMcE9Uxo5WWF4MNnqXVlCmIFC3mOIuGbIzPrgXC7VX3i8WTXf87eV2bRzCVB+j/2ddoq5nR04urXX8yBVhd8mgW4JqWsiMifXU8L4GwcFLaMRvq8Jju+Ksh4Gr8TE9GggjUFr80kwQZnIxh51gVK2lJkkZPXwPGXBYwBn3xGXCQWBrwhvn9F4g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The tracing_cpumask lets us specify which CPUs are traced in a buffer instance, but doesn't let us do this on a per-event basis (unless one creates an instance per event). A previous commit added filtering scalar fields by a user-given cpumask, make this work with the CPU common field as well. This enables doing things like $ trace-cmd record -e 'sched_switch' -f 'CPU & CPUS{12-52}' \ -e 'sched_wakeup' -f 'target_cpu & CPUS{12-52}' Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 1e14f801685a8..3009d0c61b532 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -68,6 +68,7 @@ enum filter_pred_fn { FILTER_PRED_FN_PCHAR_USER, FILTER_PRED_FN_PCHAR, FILTER_PRED_FN_CPU, + FILTER_PRED_FN_CPU_CPUMASK, FILTER_PRED_FN_CPUMASK, FILTER_PRED_FN_FUNCTION, FILTER_PRED_FN_, @@ -937,6 +938,14 @@ static int filter_pred_cpu(struct filter_pred *pred, void *event) } } +/* Filter predicate for current CPU vs user-provided cpumask */ +static int filter_pred_cpu_cpumask(struct filter_pred *pred, void *event) +{ + int cpu = raw_smp_processor_id(); + + return do_filter_scalar_cpumask(pred->op, cpu, pred->mask); +} + /* Filter predicate for cpumask field vs user-provided cpumask */ static int filter_pred_cpumask(struct filter_pred *pred, void *event) { @@ -1440,6 +1449,8 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) return filter_pred_pchar(pred, event); case FILTER_PRED_FN_CPU: return filter_pred_cpu(pred, event); + case FILTER_PRED_FN_CPU_CPUMASK: + return filter_pred_cpu_cpumask(pred, event); case FILTER_PRED_FN_CPUMASK: return filter_pred_cpumask(pred, event); case FILTER_PRED_FN_FUNCTION: @@ -1659,6 +1670,7 @@ static int parse_pred(const char *str, void *data, switch (field->filter_type) { case FILTER_CPUMASK: + case FILTER_CPU: case FILTER_OTHER: break; default: @@ -1714,6 +1726,8 @@ static int parse_pred(const char *str, void *data, i++; if (field->filter_type == FILTER_CPUMASK) { pred->fn_num = FILTER_PRED_FN_CPUMASK; + } else if (field->filter_type == FILTER_CPU) { + pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK; } else { switch (field->size) { case 8: From patchwork Thu Jul 20 16:30:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 326D5EB64DD for ; Thu, 20 Jul 2023 16:32:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B50A6280133; Thu, 20 Jul 2023 12:32:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ADA4E28004C; Thu, 20 Jul 2023 12:32:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95475280133; Thu, 20 Jul 2023 12:32:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8463E28004C for ; Thu, 20 Jul 2023 12:32:48 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5598D8020D for ; Thu, 20 Jul 2023 16:32:48 +0000 (UTC) X-FDA: 81032534016.06.FB63F10 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 408CD8001B for ; Thu, 20 Jul 2023 16:32:46 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MPh03rZW; spf=pass (imf02.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=07XsHvqnmvZKa7o2IK/zebs5uLAhltI+FwSlNCiWefo=; b=k6TLtE2Cx3WzzBWg5hu9jaA7ku4w5BewJDtlzqBHTLSTtfxRFfbd+U6pAsN5U7ilq4OsBE F2k/04a6Hj//wRe7MUDaCXyk+HyKjdGbMHJ5W3UvBe/d0xeYyFDdV5ykMz8k37wDPlVxvb MYEfYBpKUxxDkdSRJO13Dq2Lc8AMdXs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MPh03rZW; spf=pass (imf02.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870766; a=rsa-sha256; cv=none; b=gdqMcFHTbb6W6O5xJb0oGvSCULJFD9H+QD29bFbA+uVR5lws4qICKIIkWr4DOt5IyRjBJD 6xevxHb5Hf4liGKyQoy7Ix29I5QeA2MbVlZWfYF4M2kscdW+bEbm3rdS8DBbN7VCCTFfH2 2Njl2QNkfNs6BtFT7x9RWWlI9ndlhVA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=07XsHvqnmvZKa7o2IK/zebs5uLAhltI+FwSlNCiWefo=; b=MPh03rZWAXlcovDkT2RqEjie7KjpVJMynbLjTazJ7IQBbZi0yni8tpgIEo8x9s1vv/OopJ Tuij0cLeKwVK5I6X0WrxljxkGFyjDMVjRfZ5VbKDBMwsIuJGswFy9qKgxZxIJneGG1qcvS 30OoKDG3SNS1Ha1xZ6UrY6rEheMeQkk= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-171-KQfh-86JMfWenLLKyFLDqQ-1; Thu, 20 Jul 2023 12:32:41 -0400 X-MC-Unique: KQfh-86JMfWenLLKyFLDqQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A9EAF2812951; Thu, 20 Jul 2023 16:32:38 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E955440C2070; Thu, 20 Jul 2023 16:32:30 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 05/20] tracing/filters: Optimise cpumask vs cpumask filtering when user mask is a single CPU Date: Thu, 20 Jul 2023 17:30:41 +0100 Message-Id: <20230720163056.2564824-6-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 408CD8001B X-Rspam-User: X-Stat-Signature: m1145do87ot1e7ztqyaajx1oqbziuh15 X-Rspamd-Server: rspam01 X-HE-Tag: 1689870766-777917 X-HE-Meta: U2FsdGVkX18jLPX6w3o1uJiO9/iMb7vcRN3C9hZ3CDUs7nC2kbsN3vg6K2HLCQR1Fwdyxsq0fet5356Z6uswADXiHDG0uU3UwppoxdJ8OtsR1JRAuDuamflYozEmxRtTrCMD8vXPTiU2wQo5IH88AL/dUGRx9rIQkKKq6SNthV9rYidpyI4Hi/gqUk+2jgM6ozvOlVUNFkbpxcSG5i57IgsHQyjhqgk0wxZM0f/sLEkZq20M4yUSuHb8s6i/N1X4gZbwB+84pdNOwp2rjIYQonZmBbnPXA5Dy1t1vipCmoMbf+7KGlQIg4bSfHNEecNyy9ESl19uNgBAFMNrFq9JRYrZIwnzqchGNtrZ2jI7K7erJkIhW3OOfOJYZ2hirVCFk76QFOhnN0gZrJ5SvDN7N40bydvjXIr907qvkeTfoLxg8ROJ7OhCwOSXRkVkbDo/6zflrbX1Vi82FtbhW25Awx71/xSCcTzReAxFjfKKPdGsbMxT84flLynPT3FevR6TSK1XLL6bj5NCRI1QHHNH/4fe/Im8uAgX0zgvb4H+eQbMnpubSkyyq2p2+Z7CI1hla/BcPcba6lK0N3Ng/YjIbcWzLEaY9AbE11GoXOMypPoqA3TJ9vIOCKgdEWcMpP/8g32rvgnquJLXLPWIRhsL17nLCE3aOsOC0f5YRNrSnt3UaFNT5CU6oeKQp7fGTyXP0HkOWwG0UARBEIHCJhztpG1szLQ89FBCx3H0DIAuIKZU3eTV9y1bcOvfCwLUkCqHOkzLVcVIXrb0Xh2FoLi29LyfcveXV/YN9Aec+LpfzbSI+1InK9z9QW2sz9JeeRop5Qct9knfirjq49uJXFlzYBt7QY5li0NiqV/PfZWI5QgJwGUFjFtMovfIucXUJESl+1OOIKKSutFwQMN1xBfqWAZuqmF0o32JHlc4dbox52QEa1nHSvrpy7dBudNZwfGCHtxLH+H99RzZ1FLMKZR QD764Xy4 9gsWGy37KYQm4jvNgH8oz4FPzAShhqP4ESs9+/YB9wPy6yoF5WaooOi0GTl2oNu4N2DZ6aNwKXwpwxW8ELX0oJ1QtGvPj8CNPUL1j3SMyxAPWxrl6My3n5oNCylm6AI0CMFqriX0Xor75j6S1kih3ETyPiW9CzgrlIiR2RV8T3qOLrw/kKnjVOjhWidahz7823J5bFC64BEABdj8OhDbHneBZm8JV1F3b1BL/EGdDwWptzIr+5G4ahxOLgJCEl3Go7G3+VeBcIY8hCQQ/tPq+GmgKHjS8233OwFCqZ0RfrwxWv+U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Steven noted that when the user-provided cpumask contains a single CPU, then the filtering function can use a scalar as input instead of a full-fledged cpumask. Reuse do_filter_scalar_cpumask() when the input mask has a weight of one. Suggested-by: Steven Rostedt Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 35 +++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 3009d0c61b532..2fe65ddeb34ef 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -70,6 +70,7 @@ enum filter_pred_fn { FILTER_PRED_FN_CPU, FILTER_PRED_FN_CPU_CPUMASK, FILTER_PRED_FN_CPUMASK, + FILTER_PRED_FN_CPUMASK_CPU, FILTER_PRED_FN_FUNCTION, FILTER_PRED_FN_, FILTER_PRED_TEST_VISITED, @@ -957,6 +958,22 @@ static int filter_pred_cpumask(struct filter_pred *pred, void *event) return do_filter_cpumask(pred->op, mask, cmp); } +/* Filter predicate for cpumask field vs user-provided scalar */ +static int filter_pred_cpumask_cpu(struct filter_pred *pred, void *event) +{ + u32 item = *(u32 *)(event + pred->offset); + int loc = item & 0xffff; + const struct cpumask *mask = (event + loc); + unsigned int cpu = pred->val; + + /* + * This inverts the usual usage of the function (field is first element, + * user parameter is second), but that's fine because the (scalar, mask) + * operations used are symmetric. + */ + return do_filter_scalar_cpumask(pred->op, cpu, mask); +} + /* Filter predicate for COMM. */ static int filter_pred_comm(struct filter_pred *pred, void *event) { @@ -1453,6 +1470,8 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) return filter_pred_cpu_cpumask(pred, event); case FILTER_PRED_FN_CPUMASK: return filter_pred_cpumask(pred, event); + case FILTER_PRED_FN_CPUMASK_CPU: + return filter_pred_cpumask_cpu(pred, event); case FILTER_PRED_FN_FUNCTION: return filter_pred_function(pred, event); case FILTER_PRED_TEST_VISITED: @@ -1666,6 +1685,7 @@ static int parse_pred(const char *str, void *data, } else if (!strncmp(str + i, "CPUS", 4)) { unsigned int maskstart; + bool single; char *tmp; switch (field->filter_type) { @@ -1724,8 +1744,21 @@ static int parse_pred(const char *str, void *data, /* Move along */ i++; + + /* + * Optimisation: if the user-provided mask has a weight of one + * then we can treat it as a scalar input. + */ + single = cpumask_weight(pred->mask) == 1; + if (single && field->filter_type == FILTER_CPUMASK) { + pred->val = cpumask_first(pred->mask); + kfree(pred->mask); + } + if (field->filter_type == FILTER_CPUMASK) { - pred->fn_num = FILTER_PRED_FN_CPUMASK; + pred->fn_num = single ? + FILTER_PRED_FN_CPUMASK_CPU : + FILTER_PRED_FN_CPUMASK; } else if (field->filter_type == FILTER_CPU) { pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK; } else { From patchwork Thu Jul 20 16:30:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8299CEB64DA for ; Thu, 20 Jul 2023 16:32:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B3C9280134; Thu, 20 Jul 2023 12:32:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13D4E28004C; Thu, 20 Jul 2023 12:32:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED1F9280134; Thu, 20 Jul 2023 12:32:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DC66A28004C for ; Thu, 20 Jul 2023 12:32:58 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 878C31A01DC for ; Thu, 20 Jul 2023 16:32:58 +0000 (UTC) X-FDA: 81032534436.03.954668E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 807E140021 for ; Thu, 20 Jul 2023 16:32:56 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MYaenlVL; spf=pass (imf11.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Aip5VpUJA7qveL6maeQFBeNCXZHRxiQ1qBeFLUBOMzY=; b=bOTmlnxZCpZZQTin/+kcYK+OoC6xZK0BC+Wf4+zCQSS08qSCuJnRhavezcVBpGSgzIgO77 Yt4V26LamLEHSLmKBQojQya/8hD9/+M8yDPGfh0stFUEJlrU/fpcA/flEa2bze+y3WAeJr kWo5sAANl8IEWGXMKkOChvpoBE5Iblc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870776; a=rsa-sha256; cv=none; b=7bRu3521Ed+ZnbCrLLySaEp80IOgBm3ky1RiFziy6zAmApWO8Ux9vkeRwIIMYpGIyF348p hNvIJL1xlSw+XiUen3awBG3RA5aKKKk1V2JbMptUhs6wrt31bd21G22EVDE6b4xhiWsn7T niuTQ5Snc48XLm+uHvfz/mtup8KW1ws= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MYaenlVL; spf=pass (imf11.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Aip5VpUJA7qveL6maeQFBeNCXZHRxiQ1qBeFLUBOMzY=; b=MYaenlVLJkgcNBDivtjWTMItStcvi/LlnUuEw8Ha4HMnoBJ+tYi6Zzb3czG3xCXbIYRe8N jZVvIm9XFJfHBF3ViE4LwzuC0I7Gc5XS5M5U09cWBWKjmRYAlHNDyDb36PuFNbvaHWqVsv Rq+UC1gypSGq3gPgcolG/Y1s+9eT3mM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-660-hS_t_EPEMJWOi3NXCeMnHQ-1; Thu, 20 Jul 2023 12:32:51 -0400 X-MC-Unique: hS_t_EPEMJWOi3NXCeMnHQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 91B94901848; Thu, 20 Jul 2023 16:32:46 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D8DB640C206F; Thu, 20 Jul 2023 16:32:38 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 06/20] tracing/filters: Optimise scalar vs cpumask filtering when the user mask is a single CPU Date: Thu, 20 Jul 2023 17:30:42 +0100 Message-Id: <20230720163056.2564824-7-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 807E140021 X-Rspam-User: X-Stat-Signature: jfjetgg1fn5kcoggtscuokefgwsc1a6e X-Rspamd-Server: rspam03 X-HE-Tag: 1689870776-21522 X-HE-Meta: U2FsdGVkX195fagqgsKGJl6iZ2ntpMggOe/VinDgMSFcE90zRi9Ykc/pt/doITaZZ3F6IdQ5Y5WsO954JNPS99wytq/a+qmyb9Tu8aSK/wuJBM5iHGA3SV7Ci/c4Xz+0ASC2SJmTr1BwdZzS8USjXVMnf94Ebvsc6z2leLYlc3DwB1nbge4/k2e0+MhH02Z174Yhe5erJXJp96faLh6nYxL+917A83t1ulz4LeUUbcFbjVwZmSww9Wim0E0Nf28yhp0UsL6YNPIEJzhBi1PToRpInDQ8QBWPncghqbaRTEM2uvJFh0JxbYNDueBg6p+CCe3FiQkhyZmnP9QlK6tp4WZjziPzkls/XRNzMPaDcio1zBjdfccUI98cR/e63/MhjGsRVkL4zHBy81/Xkit2s/kpStNrwH75n1WjVN7IHXSUgcFEDYPr/JbDBdL1kMvdjTd5+K3fO6iq+X0uDlsicFo+1/DnN2ZM1U6QO3+rBTozPxQxuy79CYnYYbtVdiZFUfE+Ixyku3VeSKqdGQE4U2chh3yiDAOYDf8Ctaj9bNE/Vbd/lFi7uQEdpD/fwYG+COM8fHnCOxRz41xKxq2JkyFeNkhWzzbdrCMIFOgbTkOvCNgReT4w92OVtEZRzWZNmt0b2I6lipkLffN6XfpNEZlcTs+HIEupvvm1K87qISoduyJ9kQg6cqHV0JvmNiAS6rubNKbvcOQ1UM97X8JrmYffwUyfUEp0d++2VoiFNaeNULR4sQWUbqy3YzVzwmhWRsYmM2SRS8VcltvAa5UDgCpdC7plsXAxwc2Gi4V+kxpXz+/LkhGWMzIgTHDGJVfJ4X9eeQkQ3EvkHBCkxjTl76UDZ2ixgmUARyhQqRY+uuGSruUrN+V/SXKnmsIyHNOJGhcU7EKoQuybeDCCfxx3/ERkbK3jkPtU9pVy344PqnPtWAf9BhoTY8+QZ2B1T7Lrra3xpRMAQKTn6BpFa5c XgC5dP+R xmX8C6t1ix8QAdKPO0vi0Xa2W7/XpPyEJ7CDuqEdYK9R8FjDGeBRavLSD1Zfqu9LSViCvSjZr2svk5fANaPoReaMTI+4UdwlBCzEJUCT9772X+ix52VHKh6BbH4uHYTajy6q3nWmpUQnlRf5Oataes/VtiBgIaq8694i67Tix/Y0Vi0UnSkc8tuURG1+51w2K3ld6ibpk473lKU+tJYBuYdbPZ8lkJXzQ66sG1lh2titED1AdtzgQTJlZP/pSfwypB2gM47MgiFR6zJr6m7vEU28Wp2EwzlkKllcqgQGiYwbHR+5al9DcizktMoJUUIxJCv/ebIqLreSMeOlyK3mxkQaXJUfGiKZC+dTVJawIsRGkjO5BppUFczBRwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Steven noted that when the user-provided cpumask contains a single CPU, then the filtering function can use a scalar as input instead of a full-fledged cpumask. When the mask contains a single CPU, directly re-use the unsigned field predicate functions. Transform '&' into '==' beforehand. Suggested-by: Steven Rostedt Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 2fe65ddeb34ef..54d642fabb7f1 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -1750,7 +1750,7 @@ static int parse_pred(const char *str, void *data, * then we can treat it as a scalar input. */ single = cpumask_weight(pred->mask) == 1; - if (single && field->filter_type == FILTER_CPUMASK) { + if (single && field->filter_type != FILTER_CPU) { pred->val = cpumask_first(pred->mask); kfree(pred->mask); } @@ -1761,6 +1761,11 @@ static int parse_pred(const char *str, void *data, FILTER_PRED_FN_CPUMASK; } else if (field->filter_type == FILTER_CPU) { pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK; + } else if (single) { + pred->op = pred->op == OP_BAND ? OP_EQ : pred->op; + pred->fn_num = select_comparison_fn(pred->op, field->size, false); + if (pred->op == OP_NE) + pred->not = 1; } else { switch (field->size) { case 8: From patchwork Thu Jul 20 16:30:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C148FEB64DA for ; Thu, 20 Jul 2023 16:33:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64791280135; Thu, 20 Jul 2023 12:33:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D0A828004C; Thu, 20 Jul 2023 12:33:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4237C280135; Thu, 20 Jul 2023 12:33:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2D1A928004C for ; Thu, 20 Jul 2023 12:33:06 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C9E87160210 for ; Thu, 20 Jul 2023 16:33:05 +0000 (UTC) X-FDA: 81032534730.03.708B7E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id E6F70A0025 for ; Thu, 20 Jul 2023 16:33:03 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HydDn72C; spf=pass (imf15.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eoYdZ3j5mn2MTw1iUVYlPJSnwNWgHj3M8L9iz2kPLwU=; b=eI8c9v4AXJmUnxSKzphLh4heUXZvZKotRHiRq4sbioLO510RKbqycovZCI8MnnITDhB7Ex S80Z2sBVGzSJHnkexsqcWsnyN1SURafJ7ql48X6KqcL38qdxF0kuq0rTtUU6/K12BbyIta EG5DdqYJfULIJZ6qPuToh29JjIjn/2k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870784; a=rsa-sha256; cv=none; b=M3/4igfOIo68isEM7HN1dq0PJszeX1EGYyZbYfy48txd2k3V0wrz5KJoATYno/eDjCUh18 EYBcicmZC56VsOeM/9j+KdJM0rKyM1RAieVGU2PdfpAXe5E/haYr4m/9OLrYFr2VxT+AT3 Mqi5VHIjQ5+29s6qHckWu+ILivxDIn4= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HydDn72C; spf=pass (imf15.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eoYdZ3j5mn2MTw1iUVYlPJSnwNWgHj3M8L9iz2kPLwU=; b=HydDn72CehzrZvIyJPDBlvtdLiM6MpixhN2qMqtTinigwG/7HU4e0olqgpIn9cuSSQLoGb C5MmBi5fFL+l/9EY0pJ/fkQA5rt04R8VqJnFuIe/daNQRYTS1LLMjJblGUPa7xwlvT4lR0 JpOWIcbZIvhcQu2fXLAfo6Du0t/UFCc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-xF48V0OiM2SJcrNgAEdCkA-1; Thu, 20 Jul 2023 12:32:57 -0400 X-MC-Unique: xF48V0OiM2SJcrNgAEdCkA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C1AF3936D28; Thu, 20 Jul 2023 16:32:54 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C1F9740C206F; Thu, 20 Jul 2023 16:32:46 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 07/20] tracing/filters: Optimise CPU vs cpumask filtering when the user mask is a single CPU Date: Thu, 20 Jul 2023 17:30:43 +0100 Message-Id: <20230720163056.2564824-8-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: E6F70A0025 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: mpachqug9ekfua3cphjfiarctpqfnwo6 X-HE-Tag: 1689870783-436012 X-HE-Meta: U2FsdGVkX19cadC6isKIWs3o5YN3inuyOmqb2uF4Pv1MMssgrVTV+8BStRAjaAgqoEsPHul81d2IbM0+Fb5ktoViuYRcIlnwbF/Y8ma+qnW97Piqe9CcBUhsnGsba6KMyY5kJo5E83jnsVhOXacWu7d/NmPiEV6/xcFpOQgfFeVWBPghcM1hBU0MFSlocnVKJxfk5L1tGW2UqP+5BbM8FZMax/YHBuJ2QwcCmXZVTY6jIXLMxPYhXfOnNeDIoueM/rZukXMsafhunPe3bViHQWSLLBRBhQs2EBc/GXLqVFTeatB0wH7mSSSuxNxOs7KJN56cnbh77sR7WgCcK2W6Sg/Rg9JirE+D8sc3+RBgr4IS58yJIqb385O12czoUm76jBUxLmU2qZ6psQoy81AfK0IghjNePZTh62EFKMa9cN54086ocpe9E7fFubsC7gahO1xYRv5VpDifIxL8h7QjIo+RU033LW0zFg+OMxbgj06mc8GUaVAjgzbz64zssGglYAEzKP+ikp5ZAWVzb2nmwTBuXY2SXLAOdLkOlcVHutPxPKHP6KbWLl0KRuk3uUaNKscMV0daluE4/Er6/cF0rldzHEHxUysmSURpTHTU7XrYKI0uOSZVGi24xmPfXfDP9G6IwIH6fOfMBmqXaJPQWI52dIVETYI+zbOBndeBCm9tuxdpyL322fNdgkFLkBlaNcsZYzcvwJMWvcmxmhy4hGRSTLbiM+WbCQDp9KfdHKMD7EtyWz8GZv4yhhZ+ZLqp3VT57sPooP6fDR6G1QWjNTLyY9JYfdYUZINuecYN0J6ejneSC5rzYqbXMVbOy+cnIoJbw+qaP+NBYi7ukzW8Y3GeQWICRuW5KfFZnp6MUSE/1PLlQBdmo//zy+DGdjozbVHub+Zd37a/IQSAHajRtpmC9cpCSz2nH+CZbnFN64gKR1HwNleMENgL5fW5jX9xaq8j/TOX5QUH7uUkr7T ZABqgMJF Z7upb5zpPiPGPon8JkfOfEGskPDGIOtlR6Ffile/hksT4MI0unJc9dluK3oQR4DWJHc99aDTECmNCM2mMsvS5xtZnUo0tPlkDu00Z5Ts6+m84EBgfIpF08uEkfH8F7ENzJjqtaMDTJfyqED6B0u2M0PDobSrfOvis9P/6d9541Vu+SqqQHCenFkv3gayXOcXJrq9R/7DflRTb5DlCD9WV+vNmlJPeCrr5YVioDlCGSNNqljma4JEWWGLLhNZ9b+kydyYkUl5M+YmDNyiuLLW3kmARkk4YynBqmQ6OTOGMWn/LtBeGXvxOTLIWQ+okppGLxLb6Np9DYBrU41FdpeX4iFS562fcJ2LpI2pZ5EnQtA0vd77BYaYD8BwBOg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Steven noted that when the user-provided cpumask contains a single CPU, then the filtering function can use a scalar as input instead of a full-fledged cpumask. In this case we can directly re-use filter_pred_cpu(), we just need to transform '&' into '==' before executing it. Suggested-by: Steven Rostedt Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 54d642fabb7f1..fd72dacc5d1b8 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -1750,7 +1750,7 @@ static int parse_pred(const char *str, void *data, * then we can treat it as a scalar input. */ single = cpumask_weight(pred->mask) == 1; - if (single && field->filter_type != FILTER_CPU) { + if (single) { pred->val = cpumask_first(pred->mask); kfree(pred->mask); } @@ -1760,7 +1760,12 @@ static int parse_pred(const char *str, void *data, FILTER_PRED_FN_CPUMASK_CPU : FILTER_PRED_FN_CPUMASK; } else if (field->filter_type == FILTER_CPU) { - pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK; + if (single) { + pred->op = pred->op == OP_BAND ? OP_EQ : pred->op; + pred->fn_num = FILTER_PRED_FN_CPU; + } else { + pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK; + } } else if (single) { pred->op = pred->op == OP_BAND ? OP_EQ : pred->op; pred->fn_num = select_comparison_fn(pred->op, field->size, false); From patchwork Thu Jul 20 16:30:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FD07EB64DA for ; Thu, 20 Jul 2023 16:33:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0CA14280136; Thu, 20 Jul 2023 12:33:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 052E828004C; Thu, 20 Jul 2023 12:33:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE860280136; Thu, 20 Jul 2023 12:33:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CE9B528004C for ; Thu, 20 Jul 2023 12:33:14 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 78CEF1401E2 for ; Thu, 20 Jul 2023 16:33:14 +0000 (UTC) X-FDA: 81032535108.23.77DE031 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 8A766140028 for ; Thu, 20 Jul 2023 16:33:12 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VjOWU8+5; spf=pass (imf26.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870792; a=rsa-sha256; cv=none; b=FA8QQPE7L/wcmF2X+rADioVECHIe0g006LFCsgidk2v9kPXKxsyI6X45cXt/ikxNE5QTlT CyBSFw0ZECoaBPa1ZVTfEfmiuSzistcchHjTrZKdvviZvu3FuN5XaLoT8rwXsF6HntBmcw MPqcvLeICxumWWNmtZSA3Z2jeVRn6sE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VjOWU8+5; spf=pass (imf26.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870792; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v3SClOtTb5K9y8gZXjCHv42uNQ6hcmWjrBQMdqu658k=; b=omG2W0+EsUqxXNuJAOSXtL4SLytGcqcyAdOO7/+7xN17dosewmbxcFispodSS/zg3AGMHl 5sPcxs5paPZY1XQd+mQshk4K8a2njslBBBkO7ACch9tcngrrjZVRq6ozOlwRoK1C+oe6Zn wGoeOk7vSJV42FB9Nf5GoYlVFkMLDrc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3SClOtTb5K9y8gZXjCHv42uNQ6hcmWjrBQMdqu658k=; b=VjOWU8+5G/fcqAtYwAKheLjCO12nLiYDfWnEIvKWEVeMGzWBaNUdGl0lcbHNhPtrU12Wrn yuQU828CaPz3mg8inFGV1sL1eHekPTBX08fnaPFOAK5egiubw0qs182Z24Je/Qo+56e5uZ hemdks6HFz6C9m8Ul5Hb1fGZAK512fg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-6-SbuvRRjOPo6HLoecNE0CHQ-1; Thu, 20 Jul 2023 12:33:05 -0400 X-MC-Unique: SbuvRRjOPo6HLoecNE0CHQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6DD1A101156F; Thu, 20 Jul 2023 16:33:02 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1270F40C206F; Thu, 20 Jul 2023 16:32:54 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 08/20] tracing/filters: Further optimise scalar vs cpumask comparison Date: Thu, 20 Jul 2023 17:30:44 +0100 Message-Id: <20230720163056.2564824-9-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8A766140028 X-Stat-Signature: 6nhh9dkabfsedh9whor1m4q4j4sz9j8a X-Rspam-User: X-HE-Tag: 1689870792-736188 X-HE-Meta: U2FsdGVkX18h2pNLMez+cMtdqiv2af8p/f+X7hAquMEYjJezKy1gUtyZZDrBwbXzu76UBFOzrwHXJX/njPzDEiFldoj39knWK36eJxo5i0O6EkHLKGhpIPG/DiQyyDZZv0vC0eL5cbeLdD5df97nwF4Xxaeemu/6ZsZF4JkfJhVA4rTFYbD04ZGFHItVewmYS3bCq6+MHHTORQNagpfsfwaJ/4pZHEGlfBM6GEXnieioWa9thQT5wweP3t+rSZTkM7O560dtABNrHoFvoLUiJIqJBK7Bc/K6ZiEbU11jUwFUNVoRUpdfYgLtyGcEeCiguBXtQ/R4bf6oij4KReD2o3bHwPrWJ8a/nSct2f5xj9WINycI376clvxrpS2/G/luJNoH6i3uru9C0/Ny8L8X2Fjd7WH3Awf0NT4q0HWlHqQ3e1/ipck+27fRYA567WajCJiZdme2ivZnyPQdLlYX88pwZ3FAodikVnUrg91D4VFCdnR7QW37IxxEPdaq4wf+Iy+5sjl3xtjSREWXRjrcHP8132qBwwW1kcUz4tZTb8wnjYGtx1fUkA6tz0fgvtR4MiryKXUSD3Iy/+u6tHY0iP9cSaya3P+Oj+9RnJS382/F1tGJKDMnNvea+Ncwc/B7CILZNiHkNh2qgGOXBIQTzwQ1n3efcLfIk7KvscdwtPQlo2vY1tEvYCMd9Q4fMHaodPEuoJGesfszEFlPWDN93AbpSjYkgqZf1yw5kBaZ5lPPu4gIwuUhehV+QHH/LW+0U8FvUj7zQn6PURfG6v6nuevkXFHRVfEXdgEx0C944CnZ57Qn2C8vU65d42baRWAPjPE+XbEKjiYoNndvE6SSxOsVtYAhoxyY3ly7KlxRuYCCE2EtTipz1haYebomevhY/oo+PjgND/KpP+UxKDniEkP7b1vucjmWznM8pb7LaVWhnfrKkB0pXiNBlFpmyWYknImfyCT5FM+6mgCjjsn 2hEUDWeo gmiRJI6fsXGghPKv4GhiStnM3NteUzSo+AA3cIbiG3boS591wA0/pAhsg2mf0N7doxI+VAu9e1AmxDWXpv+++HMmHwsBcUh80zTEH29HJFiUiOPmHEqQNTsw/yXZuuoq7R6BWQA+ztsDJQ9vWERVZXkPHG97HjPnrOzLh6bhGLMBua518G2F8P9NSvpjXKCAKM5f0yA9k/IOitY4zzxCpdQqD4bDjIfRwN6HykZm2diEw2wF40/8EsjEETboIIR6Nik4jI96U6Zw4sLodbLcxhMitG7RHTW4ZWSl8iUiQHeykMmE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per the previous commits, we now only enter do_filter_scalar_cpumask() with a mask of weight greater than one. Optimise the equality checks. Suggested-by: Steven Rostedt Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index fd72dacc5d1b8..3a529214a21b7 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -667,6 +667,25 @@ do_filter_cpumask(int op, const struct cpumask *mask, const struct cpumask *cmp) /* Optimisation of do_filter_cpumask() for scalar fields */ static inline int do_filter_scalar_cpumask(int op, unsigned int cpu, const struct cpumask *mask) +{ + /* + * Per the weight-of-one cpumask optimisations, the mask passed in this + * function has a weight >= 2, so it is never equal to a single scalar. + */ + switch (op) { + case OP_EQ: + return false; + case OP_NE: + return true; + case OP_BAND: + return cpumask_test_cpu(cpu, mask); + default: + return 0; + } +} + +static inline int +do_filter_cpumask_scalar(int op, const struct cpumask *mask, unsigned int cpu) { switch (op) { case OP_EQ: @@ -966,12 +985,7 @@ static int filter_pred_cpumask_cpu(struct filter_pred *pred, void *event) const struct cpumask *mask = (event + loc); unsigned int cpu = pred->val; - /* - * This inverts the usual usage of the function (field is first element, - * user parameter is second), but that's fine because the (scalar, mask) - * operations used are symmetric. - */ - return do_filter_scalar_cpumask(pred->op, cpu, mask); + return do_filter_cpumask_scalar(pred->op, mask, cpu); } /* Filter predicate for COMM. */ From patchwork Thu Jul 20 16:30:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F9B6EB64DC for ; Thu, 20 Jul 2023 16:33:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C9E4280137; Thu, 20 Jul 2023 12:33:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 07A4D28004C; Thu, 20 Jul 2023 12:33:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E35B7280137; Thu, 20 Jul 2023 12:33:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D2EA328004C for ; Thu, 20 Jul 2023 12:33:18 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A5E0812020E for ; Thu, 20 Jul 2023 16:33:18 +0000 (UTC) X-FDA: 81032535276.30.09E4E64 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id CFE084001F for ; Thu, 20 Jul 2023 16:33:16 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="X/5UOLEm"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=thuwuWrD51j/PCw+xxhEtLF5o2/5RQb/+JUh6XwKh6w=; b=vMdyL2b+V0tLW4bNghb62k9dzCVd6f1CRXVAnfsQlJXa82oEJbSJgeiahiQf1qNcuYpHbU ZU2f4PZQTFSW+SYiYlqOCPQeuhXQ/wyukemoK5U2rdiIFdnAEu2Gc3Zbr0jOswuAhPfz1i DJ8ZPigDRjM3mh8MuHvFKko1LvRYj0E= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="X/5UOLEm"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870796; a=rsa-sha256; cv=none; b=FL9YB1MAQrmHLDRvvuED/iBZKKwb1AuuSrpcliYdpEYaVXA75vJeuftT52Fv39ndIKGqOV vu6Y5y65H93K7+0AtiIFutwHyCTjsZfGZEMCbLEzfb2qSOxFVsf5vu4TEaaLnPPVesKpR/ r7/43W4G7H+kvLPRxQXhYZXVmJ8Lqyg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=thuwuWrD51j/PCw+xxhEtLF5o2/5RQb/+JUh6XwKh6w=; b=X/5UOLEm5Rxsv2pFWbaBfNTyqIvyyXberChj9zp1iTWsNC6XWtzBw9h+3ptcZ3CYKt9+Yx 9RvQYUzMVgy7vzHpxXsRQLL4fmbqdQ0G81KqCHemqA65jV+9UZY45o40TSRsUYUMnuHpJm QjxPOUooVULKXiAjN3jVYeap+AXWhxQ= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-578-oEUvXGKYOk2j8KZachUjlQ-1; Thu, 20 Jul 2023 12:33:13 -0400 X-MC-Unique: oEUvXGKYOk2j8KZachUjlQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9CC113C1A6E4; Thu, 20 Jul 2023 16:33:10 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CC9AD40C206F; Thu, 20 Jul 2023 16:33:02 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 09/20] tracing/filters: Document cpumask filtering Date: Thu, 20 Jul 2023 17:30:45 +0100 Message-Id: <20230720163056.2564824-10-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Stat-Signature: 7j5yosibjrmxxkdf9i3aj8bgom9eg3yf X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: CFE084001F X-HE-Tag: 1689870796-809609 X-HE-Meta: U2FsdGVkX183K4pLMTIQ2XWjLZo/ES9x0WBzUIAzssKSHFUQJzQtNpkJ7N0mZKIrKdAe7GNoQXhbySJmJE9cpu/SjQo356rNusdF7WVLuSXbAU4hhuKwTuJJgJRbVomsfVES9dflR33C28R3bmhrQOG6zF74kOw5XS3W5lJeK9BA0mpIaKntfYSKK+kyL67vPDpkE0UUoe2gwvNLR3CrD4BX275NQGDhpK9sU170I8rls3SDjE5gGnrHUO8qcsdxC/xRS4Bz8UZ3oQTke2RLRu1vZiA265Nei3jw5Tb0zW2szukB3kH/C1TwInIxz/Cls4oNJYE78SpOHPLxeeFpOfwJtzYnTaeE96RxdgEyqhk5Q1KWZh+6zdphT3D9EZYXktdV8gTPrmedJwXE3LjkqBeGVaUs0vZlmiq+FJ64+etXHzzxg2th5WeqJr3KQg8wBTk7Ogo6YcnONCiT3MAnWyOXpgF06tIy4cuDmwy+21D7IncNlZQQtGxdKO13AATTf4m+W7CuXdiJlUodmXf4SxsDUUfOCmUQjtbV0SKxmL5nPVXyG4pAKFT7JWDaR3b72KjSdnovQm/fq5Fe/C16RKwQIvKAHkgnsprlzvUsIFv4orO++nJNJmYVrvZFR2YjaSSg15uiHz/8Lv1T6PqOazZ6WYG+Pz6gPo7C6J2zJti+qbJ88/KKy3DZ8nQsrNQYQCAuHd6Cs6mYD80Vm+eOw5s4B5JPSxaUoqOHGw/ISKP6zKq/PTYkSPhXOoL0aHlc8nbthbU6TDz6Lm+eaPbg2bYX/4gjs/kxdwrxiNLeHm1H/kTwdsuAPAMnTWNTyprbfkpsvP/55qe+cwWJwrkxzFKTZFFwxFa85YiGSaIIUxy3anh5d6siNIG8FE6PT0DC1toC/xdVB0oZNFBcI4OT4FWUXAVCTxz2eRv7kHCYVZ4XhA2YAWAP+rD5/Oap4LeLmcUaZkDMutZ4DryeMG8 9/a62ZwQ C8DCt0U5Y1ZqGyc5AyTCt413vuhfvtjZdOLRBk+yoUIZShM8yILSt5TfeQkE/EWY9FrFIAhisBAa0LihJ65x5kXdjIHn/39O3oHtFLw4Gf/Yd+GEURlKCNbnnrO9eNdwIcgGu4TzGGdVQPhsz9eUq/7E+aRkI3sgVAPNLaitBzLXaTwt9UmJ9Vz7YZWRov/lt3ai0hfg/LVsFepwXPv3p2hZ7S6z/OHuZxKKdRxiBUQPabhGcRcpMqeCWUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cpumask, scalar and CPU fields can now be filtered by a user-provided cpumask, document the syntax. Signed-off-by: Valentin Schneider --- Documentation/trace/events.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst index f5fcb8e1218f6..34108d5a55b41 100644 --- a/Documentation/trace/events.rst +++ b/Documentation/trace/events.rst @@ -219,6 +219,20 @@ the function "security_prepare_creds" and less than the end of that function. The ".function" postfix can only be attached to values of size long, and can only be compared with "==" or "!=". +Cpumask fields or scalar fields that encode a CPU number can be filtered using +a user-provided cpumask in cpulist format. The format is as follows:: + + CPUS{$cpulist} + +Operators available to cpumask filtering are: + +& (intersection), ==, != + +For example, this will filter events that have their .target_cpu field present +in the given cpumask:: + + target_cpu & CPUS{17-42} + 5.2 Setting filters ------------------- From patchwork Thu Jul 20 16:30:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C860C00528 for ; Thu, 20 Jul 2023 16:33:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADB4E280138; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A647128004C; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B759280138; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 794D128004C for ; Thu, 20 Jul 2023 12:33:26 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 52E96160210 for ; Thu, 20 Jul 2023 16:33:26 +0000 (UTC) X-FDA: 81032535612.05.D5F53A2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 5CD5440017 for ; Thu, 20 Jul 2023 16:33:24 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PkTLPdFg; spf=pass (imf17.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870804; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G5HBHaVT0pfIJAbbI8C24iwVVdenu59ApFnKMQXGGMQ=; b=djd0FhlwuY2U0cJ6PSsRvVI46UiuVG/GfgTLsXGVEaBjgQ8uq7Nzi/1HYrHtzStfcVHaTR Pw435yc/6kFI6pSyWIqBKuxgyKVe5OmJfRk0uKDNFFQg/IX6K1RFTf/nBZAe3cMSMHFVI+ iRtVOg7j/Iv5a31owHfGcJ5Hu6T7wEQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PkTLPdFg; spf=pass (imf17.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870804; a=rsa-sha256; cv=none; b=XB7ELUQKxGZWFyPCqVYs8gUB4Xyy8P2mPlTHtgYccisVXnnYw4LPLBxASR1V0xihlX8EuI 9QtekAIlFoE3aRlZULHA4xYtC3KjOcsdenNpcDxxbhiSVjfmy2eRaPBT0VMmt8HIA+qB7p IQotwz5ijy2G3L1lQqmGlW0WDzb88aE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G5HBHaVT0pfIJAbbI8C24iwVVdenu59ApFnKMQXGGMQ=; b=PkTLPdFgleBGcxnnJQ8Tb0+vei7LSM4eGj6jWRj6xyCBPTfmDFltcVMLYaBYxXYc41Gve5 09CPyfrMJap5knsTV5fiL2SYblhtRkydia5ATZicDi6d4zwVF38jZy/AC2yB/4a6rjGytN STzFqzVY5W0LUU51gy2xwUdQM8nFNeg= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-543-hhF8YKTPPJa4Xr8UF_kvXw-1; Thu, 20 Jul 2023 12:33:21 -0400 X-MC-Unique: hhF8YKTPPJa4Xr8UF_kvXw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 679323815EF2; Thu, 20 Jul 2023 16:33:18 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E3F8940C206F; Thu, 20 Jul 2023 16:33:10 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 10/20] jump_label,module: Don't alloc static_key_mod for __ro_after_init keys Date: Thu, 20 Jul 2023 17:30:46 +0100 Message-Id: <20230720163056.2564824-11-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 5CD5440017 X-Rspam-User: X-Stat-Signature: 9irw839t5bbuu1rgnqkw7zbi8nknwyap X-Rspamd-Server: rspam01 X-HE-Tag: 1689870804-625382 X-HE-Meta: U2FsdGVkX18JdbDx+RAjqmeXZOCgNjCDYGKX1i5pzIl/By0OBM/wGYh1eMCI3ycvmr4Wsn2EKN+cNk4WITJ6OvkCNljexIByEdZAEWNbgE8GMO2Opb9gHQATJt5tn21EWSSupa6A3fPH4LTdw+h9sdKaC2HW347TDd4btFFZ+I2UsqrPzXFynE1v+kKFI81wRUbyiu76MeC+iQYT1gYgFn9SlfgkDqrzvRBkwueLo5SCSoq0C+NKHD8hciccheeyl9VL/MQKE7+J1ZJDbAuzojbsb6uzPBKWYUs1XCzPTe8RjzJVXgt3gr6xspo7W1YIxA5GZsrE6JnQpI3lYUx+ZAb/IWlWiKHpzrtZMKhKR8TT1c+rGzZOwWIZuSML01siSiX4PD/hfhbVAJoPJyPzpg613JAjj5aVAeQB0nDMVP/HrQL/kjubdy3XSehK0ay9NnXMwGxOlVvELp4F5KpDvbIE11QOdHN41IK+gTEMcP5V2xrB+K3GsWhCtouTm8ZnTGjgReHTei9qhUJNlJjOACXje5BCGxlGtLVWnYNl3vWOIwDQ1CV925yUhs/P4GMRsvb0wmOYdKz5U0jrOGcaApxyLembez4T4yFUEtWVF9AVk0M+bM2OuOh1lVPY2pQQECglWM9Xxzwim93Vhk8LLRn/TcnDkuW1+MGa1gedrnNUSXbX0EjRwKHt66tRdIMdrR8Uk4xuz9uWS2BmqBh9icMp0Q6/l29NmvE99U9mDtxYMC3TEAi7EleB5QeLkOZ1lTn59YxPEsuz/2MnsPS+4zFHqv2nIrcxUoD/tKhwpK1QKs3RModmoxrMynQ4Zg6ZaVVfANrdSgjoEMf2R7coSBZEaKWYWO2NbpScBEOXNVyHfMSXCTdtPC3dSzToUBjPw8QNzLA49Tmowi6m9mTwUtDDEqOySENarVGWXKFHtg4RJEWqLUMeYEO5TY5Z8D7UmB8ZXKwJVCZWmjeu8kX EFvBcpQD diw0j3K8jYu6mU98pwFAAHHJnTjah4pQn4h1bqQdqKS2DTdxmVixmBE3gHe78tOj+ndpjNVhrgyQcWfyeYLQ6W06OfsnC7BiFSZdEjL6AhkH5grffT58fz1/3vxCFDNeAy50I77ChQmfNee6xVh6UnsR43xB7y6un50dByYO5Cv6Xh+3ACuUPm9xmc0Kv6nU5NYR8qBOMoJootSMOj96MPuVE3KKqOhcUtP5MryBbDCeBnWC6ypV9LNO8zjet2W6JS+6d2nT7Arx/amjR/k2JAnKrsAoNHB96mc3R6FOs4nsxaKMGQGwFM1YVX7psZtP4eWUmiV3QLISpcihxjmr1xHlkyI0fRMDlkzT8meTJ/VYPiIlZ1Rm+QT8n5X6vYtn96/iJjUKSawW73gnBFSVhhenOCO4FA3pSmYY9E6NckLTs8qF/VvvsXAyPfgIFdXMWhWy7PimpwfTAIsGT5zlp7zNjTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Zijlstra When a static_key is marked ro_after_init, its state will never change (after init), therefore jump_label_update() will never need to iterate the entries, and thus module load won't actually need to track this -- avoiding the static_key::next write. Therefore, mark these keys such that jump_label_add_module() might recognise them and avoid the modification. Use the special state: 'static_key_linked(key) && !static_key_mod(key)' to denote such keys. Link: http://lore.kernel.org/r/20230705204142.GB2813335@hirez.programming.kicks-ass.net NOT-Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Valentin Schneider --- @Peter: I've barely touched this patch, it's just been writing a comment and fixing benign compilation issues, so credit's all yours really! --- include/asm-generic/sections.h | 5 ++++ include/linux/jump_label.h | 1 + init/main.c | 1 + kernel/jump_label.c | 49 ++++++++++++++++++++++++++++++++++ 4 files changed, 56 insertions(+) diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h index db13bb620f527..c768de6f19a9a 100644 --- a/include/asm-generic/sections.h +++ b/include/asm-generic/sections.h @@ -180,6 +180,11 @@ static inline bool is_kernel_rodata(unsigned long addr) addr < (unsigned long)__end_rodata; } +static inline bool is_kernel_ro_after_init(unsigned long addr) +{ + return addr >= (unsigned long)__start_ro_after_init && + addr < (unsigned long)__end_ro_after_init; +} /** * is_kernel_inittext - checks if the pointer address is located in the * .init.text section diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index f0a949b7c9733..88ef9e776af8d 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -216,6 +216,7 @@ extern struct jump_entry __start___jump_table[]; extern struct jump_entry __stop___jump_table[]; extern void jump_label_init(void); +extern void jump_label_ro(void); extern void jump_label_lock(void); extern void jump_label_unlock(void); extern void arch_jump_label_transform(struct jump_entry *entry, diff --git a/init/main.c b/init/main.c index ad920fac325c3..cb5304ca18f4d 100644 --- a/init/main.c +++ b/init/main.c @@ -1403,6 +1403,7 @@ static void mark_readonly(void) * insecure pages which are W+X. */ rcu_barrier(); + jump_label_ro(); mark_rodata_ro(); rodata_test(); } else diff --git a/kernel/jump_label.c b/kernel/jump_label.c index d9c822bbffb8d..661ef74dee9b7 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -530,6 +530,45 @@ void __init jump_label_init(void) cpus_read_unlock(); } +static inline bool static_key_sealed(struct static_key *key) +{ + return (key->type & JUMP_TYPE_LINKED) && !(key->type & ~JUMP_TYPE_MASK); +} + +static inline void static_key_seal(struct static_key *key) +{ + unsigned long type = key->type & JUMP_TYPE_TRUE; + key->type = JUMP_TYPE_LINKED | type; +} + +void jump_label_ro(void) +{ + struct jump_entry *iter_start = __start___jump_table; + struct jump_entry *iter_stop = __stop___jump_table; + struct jump_entry *iter; + + if (WARN_ON_ONCE(!static_key_initialized)) + return; + + cpus_read_lock(); + jump_label_lock(); + + for (iter = iter_start; iter < iter_stop; iter++) { + struct static_key *iterk = jump_entry_key(iter); + + if (!is_kernel_ro_after_init((unsigned long)iterk)) + continue; + + if (static_key_sealed(iterk)) + continue; + + static_key_seal(iterk); + } + + jump_label_unlock(); + cpus_read_unlock(); +} + #ifdef CONFIG_MODULES enum jump_label_type jump_label_init_type(struct jump_entry *entry) @@ -650,6 +689,15 @@ static int jump_label_add_module(struct module *mod) static_key_set_entries(key, iter); continue; } + + /* + * If the key was sealed at init, then there's no need to keep a + * a reference to its module entries - just patch them now and + * be done with it. + */ + if (static_key_sealed(key)) + goto do_poke; + jlm = kzalloc(sizeof(struct static_key_mod), GFP_KERNEL); if (!jlm) return -ENOMEM; @@ -675,6 +723,7 @@ static int jump_label_add_module(struct module *mod) static_key_set_linked(key); /* Only update if we've changed from our initial state */ +do_poke: if (jump_label_type(iter) != jump_label_init_type(iter)) __jump_label_update(key, iter, iter_stop, true); } From patchwork Thu Jul 20 16:30:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B8BAEB64DA for ; Thu, 20 Jul 2023 16:33:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A65B8280139; Thu, 20 Jul 2023 12:33:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9EF5B28004C; Thu, 20 Jul 2023 12:33:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86931280139; Thu, 20 Jul 2023 12:33:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 763F528004C for ; Thu, 20 Jul 2023 12:33:41 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4D466B0AA5 for ; Thu, 20 Jul 2023 16:33:41 +0000 (UTC) X-FDA: 81032536242.14.8298541 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 718E540024 for ; Thu, 20 Jul 2023 16:33:39 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UdUytoeq; spf=pass (imf04.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870819; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gxt5b+dw2J3MlRZgV6DoONRBYEDjEJLqkxMRP/UHfXc=; b=wmWgkzNJeteDjFxDiW4HUh33scLNuTtSFz/Kcg4O55yCmP6odQASD+KPdFC7g8A57zdMnA ap70VvDsnyd8HqCO/mlbpTUARItho3gBFI5M8ow5rL/Mjnq1yQC67AavD+t9kQ8iRyWMit jzh9mxWP3bRj9+vjba3Yft/JTg35xWk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UdUytoeq; spf=pass (imf04.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870819; a=rsa-sha256; cv=none; b=2vOTdk5X46kWFLjqfKn8+q404rHVDOeAbvTN8u8Rf+88wiF0MfmIJGYshEa3KFcWa90UlA 6TiVTtU6bbm8ZOWwNBjb7I//+z8RKQgSSU54bdI++6Hj54Flkkappn6IkIxDv1MFqfn1k9 w9EfeY4aL0Fq4QH71vd7zvqRJH/JY9E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gxt5b+dw2J3MlRZgV6DoONRBYEDjEJLqkxMRP/UHfXc=; b=UdUytoeqCFg/Fw9s0WqWqSojhl3ADm0COqr37rDs+fLsToB14zv4y4EOdshlY2KDYI75CE ZRTw0mcJ9kZ19D8BPP1OHIE3IV//bQPQs7ozyyTE8T/krXFhyEa+q2CKFTYcUoomIC/3WJ Dv+MhkYMYVUYIOqyu41Tf776FCmS2/M= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-637-bk8eKh1PMsKjdr3SYosUIQ-1; Thu, 20 Jul 2023 12:33:32 -0400 X-MC-Unique: bk8eKh1PMsKjdr3SYosUIQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 902321C05EAA; Thu, 20 Jul 2023 16:33:25 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C6B9240C2070; Thu, 20 Jul 2023 16:33:18 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 11/20] objtool: Flesh out warning related to pv_ops[] calls Date: Thu, 20 Jul 2023 17:30:47 +0100 Message-Id: <20230720163056.2564824-12-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 718E540024 X-Rspam-User: X-Stat-Signature: quwrt8sxurqdmhgn9qsixs4fmfoa1iy5 X-Rspamd-Server: rspam01 X-HE-Tag: 1689870819-730657 X-HE-Meta: U2FsdGVkX1+f8aoQxEzoAkS9sTrUWBnN9tHKy5AyWRI26LERpLYnI8hJb4zPEVQQQjFV5OwnEqtbIttu7lwf43j3fBwUpyq2eeLBTu10K/TpNG07Zh5s+LGXBPUko8W28WSnCzRy2aIXuCiaA5SeX8f/aoxSeu6CRhZ/miFGckX2jirEWNQZVByJHpkAnnE6OmHfFPeyuUwEVhs5+nJaXlNJcxLQEa8dmF50QDETzWhUQ0SlnLETpywrhhZ9Xrd0pFvSdsEDINOUwf6pETUPHMDis5MlK4Am0RfNNhgWvgzp6v7FLqFDwJK+L5MsX/QqcG+ezuEjjZ+QuBX9z+/e/eneCTDD6pDlaRTdZT9QEzCbl/dgml9nBWQLHjL69MFalRLTKoMqbMNXY7jZ+NWA6YEfQEiAbngb5+gJ4YLzUq4XphOKMOWoOk3Iljk47EE7Iddmb6K+M83eG6e053FTl8kDVDOZxMPs51Qie/+xYhsM1hESSIYGodl/JLwTPUsh41ilO4beq0XZgp2qxy0rr50/K4TiN8kVesMBNUTRKvXUR9ORPufTr6Z1z1Vr9ovjMJMwfwvYkpumqbiUwZGGW6w+f+VWnFx4AF/Bahykzurn5BZ2SQPcalEL9n1MRjIThyjOmFG1C0JDsA+BLqckmMhemmM/6ssWEqv1JogXsBgTMWXtIdWlKeQ4K9fiKSoMWYB/RPaPJUqe6wuFToXkrdMwxx3sj+1BGmtaqWtIskzHyiVI55yiIz+FkQr5VqaXr+XlWBup1M0dKh1cCmfC0Y4sJ8JuBqjZtTM9uC8OfvlgtbMyx9FXp8FoR9mxjcsSs/btlG7CAMuBtYOqsw4W9J7N9oULZiJIAAPuzkEFiJX7CTr+TSqBrVMNsXmX267xEl24cu0Dlf7G9aemv5WdfU2OAjRilOKi3FsagcnqXso87+/mcYHBPaAfe4UXibxLo8sebUFuXS+H/YaoaH0 2Xfu9kW8 tW7kdw+QI3WGYX1ohwnwAVZgZEP9kKMlMNJ2MZC0osrg2klzNZbJSSyQTbIQLPcmAEJ3/TKnd64w9/T2xTM7FmmFjml/Rfp50knSRSHPLl75+2FUuVjGkX5wJS6HimMnxeqpeKpOqVavsV/hDCVDA3dVKe/Pw9onGVdc35F150FIpvqP9rc7ebLcOmP+1kKMW8gUdjLaoyWg0SECnSyMLdNCapn224ilQjw4aNeZFnb0NVSSSs1iOwcxyI9iMlrKKOHCRDgdkGligB2d2nRyIlS3zDiQWjoGnQwHN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I had to look into objtool itself to understand what this warning was about; make it more explicit. Signed-off-by: Valentin Schneider --- tools/objtool/check.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 8936a05f0e5ac..d308330f2910e 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -3360,7 +3360,7 @@ static bool pv_call_dest(struct objtool_file *file, struct instruction *insn) list_for_each_entry(target, &file->pv_ops[idx].targets, pv_target) { if (!target->sec->noinstr) { - WARN("pv_ops[%d]: %s", idx, target->name); + WARN("pv_ops[%d]: indirect call to %s() leaves .noinstr.text section", idx, target->name); file->pv_ops[idx].clean = false; } } From patchwork Thu Jul 20 16:30:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 588BAEB64DD for ; Thu, 20 Jul 2023 16:33:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E848A28013A; Thu, 20 Jul 2023 12:33:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E35A028004C; Thu, 20 Jul 2023 12:33:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAF2628013A; Thu, 20 Jul 2023 12:33:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BC34C28004C for ; Thu, 20 Jul 2023 12:33:51 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 96A7A120110 for ; Thu, 20 Jul 2023 16:33:51 +0000 (UTC) X-FDA: 81032536662.24.497B3B7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id C4AFF140015 for ; Thu, 20 Jul 2023 16:33:49 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A9cIk7hV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w2k5nztFy3WXKlhL7LY6m+LcDW7MzG+nauWvTii+Ef4=; b=BoR5aQ55o6sDQLASNn8DVPrdB12Yu60R+B7x0wNELh2kyM1T4InRDuGlnfiAVrODT9bPeM hE7fkp6NndCiFvdMc+AnJ4I0j1YRAR/RCFfRkTmCPI3A8hUtEwc31exemp8hCf3cc62AZr YaV7SvdW91eu648IrhvigAN3f89Bm+4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A9cIk7hV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870829; a=rsa-sha256; cv=none; b=x9T0Q1PkbJeAY2mlHEJaLfIgvvzExwmPaqAfifnjyYgjXiOAnRSFeztUPQ1xmdbjOFE8UL Wt0kIvOYD++cFRddQFaylCHeUag/WJmD/fbu5ex/lFaMav4w6VCf9bdQREGG4ygFG5hy6J nEEftR5uxbtEwQsPnsxpde5lJCrTLmk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w2k5nztFy3WXKlhL7LY6m+LcDW7MzG+nauWvTii+Ef4=; b=A9cIk7hV7QKZoF+6s1x00u4oDJ4UYPgSPYNU1guRhg3Xp205VkA1MdHLzD4qrNZ/Lp33my bnBPEEye3dVSxgc276iTmzvA0mtyK7WSt/EQEiRKmMaVVZMD2LCrRMHRz3xQE0c7IHzomg NV+q9a8Iq9AFi6zrdLR6c6bYgtBhmbk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-77-dQqQ7nvGOe-_KMx6MDuCag-1; Thu, 20 Jul 2023 12:33:43 -0400 X-MC-Unique: dQqQ7nvGOe-_KMx6MDuCag-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F1ADA1044592; Thu, 20 Jul 2023 16:33:33 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F11B740C206F; Thu, 20 Jul 2023 16:33:25 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Josh Poimboeuf , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 12/20] objtool: Warn about non __ro_after_init static key usage in .noinstr Date: Thu, 20 Jul 2023 17:30:48 +0100 Message-Id: <20230720163056.2564824-13-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Stat-Signature: 8jhyqn1z5hcnbj6dcuxfezp514i3s68z X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C4AFF140015 X-HE-Tag: 1689870829-306628 X-HE-Meta: U2FsdGVkX193TVBdcU2HVWIASUFsRXXFrFoOEKh+ET80felfr7NIEvp7V18rGK49lhe0HHhq8dYdnjkfab2AaTxy9mkIVb/dMwuyrouMx/yB4PvreQrARULuUzPReaoUlXACSJef3jOBcT20pVleeTBv6R8gJZ53a6QkK/PsVDodhRGvlzj1rpmg+FQWBQKXXmT/osgUkVxPiWaVHmzg5M3UpRA04VwPnhiyQx46cNEqrbCWrTMAI/00HWqAzwhs5mN63FVVmWZavsF4ahxRtax4U1xlU9Rv8gOBdfm5L8pnQb5npH6g+duRGK453IJ4pQll5XeqmOnQIGwaHV8OHxVrfNzmr92y+di/ReBWeBohwZEcU1IJRJCpqbhRHbLMGycz1IowrzaizzC2b84Odt5j0CaQVGQMzgn5nwXiKSLrDndpn5LvIBWJHpfW7fcxVzXBL1UbJO8hs3XcdK61/MCpwdggfi2ERN5XsQDtHlHB6p9I4J/ylPjnvWHdnUtO8X6agtl73yIuX/3fcHXJ3my/TUu9uXUpLoJbZC72lDkcu8RhwxKGkZCLLrCe1XqNm1Txm0G+PVRWFMOhSv7J58Jgq2thUOTbKeeHj/+Opb8QVIAzCKfUt9qv+5Xb+1s9lLur8/yF693GocSeuRJ0nTLcAsOTc7Ltlo4p2hMtyi/YKNY5ndSWpkihyG17XaOa0y+4DGz1iA82gv9xYwYqlQNBV+KgVOwhA7M/mcIeqt8o2hkIq3UuXiRRuXeVGF4EuhgG8XkzCtgOGhR7xgF1IALNwoeif+jJRXfCVIgRFZ3AHKce9VbXy/bzoltVSajP3DDnZSXJahBtBLTuxOTZmnYMH2fURjtAG5s6VUM6gR+2wWNm9yd57ngX1M5qwQHaOx5ibYbFrKIDghNHiXA/HjerYP46YE77V9+O6dWQ5247/D/kJSRmBWed6cw/us5EpSg+CXvvDYWLV6GDF3k snY6uQl3 XYuvgwpuSF+mSNG2kd65j1VKsgDZLlhv2YitNPqv3FBLygON1S8dAABFCrSyuNuS1bVVMj9UPuB9zqxx+2eENoDUwzzH7u4knQA6p+NTW3K8dndrzce4Pr4PVTCWsNFOBrIkMo4EqAr1rE4294byYSFLUtxkOc870NEjXh1XQXabfHd195EVXxEdZmyzcyrEKN5BmfKWa7ZDmStTGzcFO8CqDZdX3YTkQSiprAVncxWux8Szh8ihselFdzAUeEkrvcjGwiyXQ8FMpqsiOOuY+7fS9sTtH7OdUQKY/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Later commits will depend on having no runtime-mutable text in early entry code. (ab)use the .noinstr section as a marker of early entry code and warn about static keys used in it that can be flipped at runtime. Suggested-by: Josh Poimboeuf Signed-off-by: Valentin Schneider --- tools/objtool/check.c | 20 ++++++++++++++++++++ tools/objtool/include/objtool/check.h | 1 + tools/objtool/include/objtool/special.h | 2 ++ tools/objtool/special.c | 3 +++ 4 files changed, 26 insertions(+) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index d308330f2910e..d973bb4df4341 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1968,6 +1968,9 @@ static int add_special_section_alts(struct objtool_file *file) alt->next = orig_insn->alts; orig_insn->alts = alt; + if (special_alt->key_sym) + orig_insn->key_sym = special_alt->key_sym; + list_del(&special_alt->list); free(special_alt); } @@ -3476,6 +3479,20 @@ static int validate_return(struct symbol *func, struct instruction *insn, struct return 0; } +static int validate_static_key(struct instruction *insn, struct insn_state *state) +{ + if (state->noinstr && state->instr <= 0) { + if ((strcmp(insn->key_sym->sec->name, ".data..ro_after_init"))) { + WARN_INSN(insn, + "Non __ro_after_init static key \"%s\" in .noinstr section", + insn->key_sym->name); + return 1; + } + } + + return 0; +} + static struct instruction *next_insn_to_validate(struct objtool_file *file, struct instruction *insn) { @@ -3625,6 +3642,9 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, if (handle_insn_ops(insn, next_insn, &state)) return 1; + if (insn->key_sym) + validate_static_key(insn, &state); + switch (insn->type) { case INSN_RETURN: diff --git a/tools/objtool/include/objtool/check.h b/tools/objtool/include/objtool/check.h index daa46f1f0965a..35dd21f8f41e1 100644 --- a/tools/objtool/include/objtool/check.h +++ b/tools/objtool/include/objtool/check.h @@ -77,6 +77,7 @@ struct instruction { struct symbol *sym; struct stack_op *stack_ops; struct cfi_state *cfi; + struct symbol *key_sym; }; static inline struct symbol *insn_func(struct instruction *insn) diff --git a/tools/objtool/include/objtool/special.h b/tools/objtool/include/objtool/special.h index 86d4af9c5aa9d..0e61f34fe3a28 100644 --- a/tools/objtool/include/objtool/special.h +++ b/tools/objtool/include/objtool/special.h @@ -27,6 +27,8 @@ struct special_alt { struct section *new_sec; unsigned long new_off; + struct symbol *key_sym; + unsigned int orig_len, new_len; /* group only */ }; diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 91b1950f5bd8a..1f76cfd815bf3 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -127,6 +127,9 @@ static int get_alt_entry(struct elf *elf, const struct special_entry *entry, return -1; } alt->key_addend = reloc_addend(key_reloc); + + reloc_to_sec_off(key_reloc, &sec, &offset); + alt->key_sym = find_symbol_by_offset(sec, offset & ~2); } return 0; From patchwork Thu Jul 20 16:30:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E29CCEB64DC for ; Thu, 20 Jul 2023 16:33:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A91028013B; Thu, 20 Jul 2023 12:33:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7307A28004C; Thu, 20 Jul 2023 12:33:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5836A28013B; Thu, 20 Jul 2023 12:33:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 44FF128004C for ; Thu, 20 Jul 2023 12:33:57 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 07FFBA01FA for ; Thu, 20 Jul 2023 16:33:56 +0000 (UTC) X-FDA: 81032536914.27.D2C5546 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id D8E3A140025 for ; Thu, 20 Jul 2023 16:33:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KUDGCOax; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B/1vdVWCM4YknPYZ5ppPG1wK5UcqdOcCaaYb4tQPZDs=; b=BGaWJB+Au4MtRfaYIhAak+vlxJjUWu1MNta/r9t3IfJk16rFWHJ2BAAdwh0pVtF/ObPcra bx1iwc6HtNNMGwGDim/GV+DtuQznVZYELqbkWrNs8rBec4xCVyEnr6JT/8gxwoN65I14Ko 4y0fhH+6CEQMpprSltHkJ++F0OmjwqI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KUDGCOax; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf09.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870834; a=rsa-sha256; cv=none; b=efgkev0AYeT8YZJ3O+NFdADEj7ruk8hugjb5omBUVQRA461xWfgsZuWymLJfij7Lx8NDUa UUW9XZw8qxkLxfhm0SAMU/DEFcKbvbt8eb4ihxi+4++WkX3Is45BSj815680sNFE2GWpcs 6IqKx9NmjBK2xkB8uIz55eKFsIL0qDI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B/1vdVWCM4YknPYZ5ppPG1wK5UcqdOcCaaYb4tQPZDs=; b=KUDGCOaxX4xatYbGfRJrE6hXAQ7r4LUc8cupDGS1HPvWG3gtm6TG9bqXDeL6GXLbaxOUug TIBtOZRbsu1wyQouSQwvOEyQ8LeENtMHUjOEcjePrQthaFuGdhdu+KfVFFppf2WbrPJnsX DxyVtO49mK7RTJ3MoJ3401AxrjfO2GE= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-504-yTDAMrF4N--Bk_WMn9tLGw-1; Thu, 20 Jul 2023 12:33:50 -0400 X-MC-Unique: yTDAMrF4N--Bk_WMn9tLGw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5A2A62812951; Thu, 20 Jul 2023 16:33:42 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5C73540C206F; Thu, 20 Jul 2023 16:33:34 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 13/20] context_tracking: Make context_tracking_key __ro_after_init Date: Thu, 20 Jul 2023 17:30:49 +0100 Message-Id: <20230720163056.2564824-14-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Stat-Signature: gkkxpqq89tuztq597qrjxsi9rkg3xxf3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D8E3A140025 X-HE-Tag: 1689870834-450574 X-HE-Meta: U2FsdGVkX184QtiHe0cmtxOu4JK/QAfETeMdV6UL+K5o3YbeeORAqHu7IWx9YodOOAzqvlBkqrLlvvGHrIHe+xJA6phE8Dzg61k2P2ei0t0k0v89ty/wh+iYt8VESZLWIg0HLmZlM3K77t7EHIThH8F83DgP1eiFgNQezbLKV1mbRwcJNzseE55lw5GX9Wg5mOzuRHOgSC2h8ylNJjgcgg1a6NyzUVEJD+iop8HyzR5ZGxGHtj8JFAGes3oQGbYWqEKdBRrRbgsNJ9mGjDzUnLCHbot+W/d3Alo3/LCs3JSEwlsM0gaRptjHHV2Y+4hMJFgbQ5as+uiFNSQBpTXqttbClkUGCXTXSaKP0vIwMJuHXktBpDZN2Hi1lRLGPHqu606losn+yNuB1TydTw/h2Qfz8OuiWV3ETHolwDc6cwHC4agerx7J0gTWOWSkHtGZhL8j9c6x5QvTd7NCPb+0S16Ck/85tsBPFQBXIJWgPnrknz0sRPSlYeRAnXTZgA5YkI2JSuP3q4deVuJdLMGH+KMcrt1ZLvMt97L7cDRkIToMbIsjD8wL6h9yagGKR2nWzcnRufVhUV+x/w6vtzu/4qhy4POF7mvtQ8pDaPAXvYLdTI0rR1TLlqooxtftCMClIhDkHob4Z8il7QBl2dFUtwpe+8I+Ux31fiSps/dE7tJzboQmcYQOqWTgRs43msP8+16PbKh7PuJsEUNBjov/6PVs8skRlQTmtTu1Fjttnlcgo08wLP12MWeesDKT7WJx4xSFy5XxM51M2WCyZJVrrWdNGVZj9jiJLQjGK4iVzxRsa/XH6zIOr3xekh9xFQJVsVHWTs+W7eh5xxMSy7caG97/tAbldeePPYH2u52jwtNpXlafeEhsU/AmB/wprrZ5mC7wyiSjxhYwrDbf1v7X65w634Pfe0xAImrAJe39uw7MssvF4MEI9dCosCPIbylF/QUA3Pxq6gDme3dpx7w 1pYzH7Zy ligx+lJy/j9XoHtP3KuZO2HG9b9f3dlE6T1BqeV6zangTKNZk5pCYDIErKsoNe4hVljsEdOGyiKtrS44lG5pcWaWcenm0aY7LSWMGLJyGGrSasqQ1D29f35ZgFKZ7Aun4gAS1zV57xfUgW05iwSS8SaeQ5UIdkvzy14oJaGns4myPwFfnc/8Xu17iarX18HIZooQhpS+AjaeQViqZeqkt+MHARLHYuTtqOKD+NCTj3J5qIirFsZLUrsktl9J3xQ4hwkWVTZdO9+qmoGCb9Qjcikbj39fxNtNSlCms X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: objtool now warns about it: vmlinux.o: warning: objtool: enter_from_user_mode+0x4e: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: enter_from_user_mode+0x50: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: syscall_enter_from_user_mode+0x60: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: syscall_enter_from_user_mode+0x62: Non __ro_after_init static key "context_tracking_key" in .noinstr section [...] The key can only be enabled (and not disabled) in the __init function ct_cpu_tracker_user(), so mark it as __ro_after_init. Signed-off-by: Valentin Schneider --- kernel/context_tracking.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 6ef0b35fc28c5..cc4f3a57f848c 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -432,7 +432,7 @@ static __always_inline void ct_kernel_enter(bool user, int offset) { } #define CREATE_TRACE_POINTS #include -DEFINE_STATIC_KEY_FALSE(context_tracking_key); +DEFINE_STATIC_KEY_FALSE_RO(context_tracking_key); EXPORT_SYMBOL_GPL(context_tracking_key); static noinstr bool context_tracking_recursion_enter(void) From patchwork Thu Jul 20 16:30:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 843DDC0015E for ; Thu, 20 Jul 2023 16:34:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F03828013C; Thu, 20 Jul 2023 12:34:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 07A2B28004C; Thu, 20 Jul 2023 12:34:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E35C028013C; Thu, 20 Jul 2023 12:34:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D2C1728004C for ; Thu, 20 Jul 2023 12:34:01 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7EC37160215 for ; Thu, 20 Jul 2023 16:34:01 +0000 (UTC) X-FDA: 81032537082.08.996D2BB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 727A68001C for ; Thu, 20 Jul 2023 16:33:59 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7r+keQY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870839; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=izi9UHkrSeZ4AHa49u6aA8W9HneeYs8L2V/Y70LmMpM=; b=MCBIOAsO0zgEXaHIl4M1tKD754UYtxT1ofaJGsASrs15yoanL7V+NGR71gPbVpOn0hKeXn jRLigA1tUWLjc4Q1gVDCJuBWnsd/zZcFPT3iiQvD2Wagsc4bbFxRn8LyRfhaXVtvVUl1vu Ym1+vtmJxWFg3Z1Yxk6Q8WspqrxOJ48= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d7r+keQY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870839; a=rsa-sha256; cv=none; b=c93rndAl1fpKck6ThTEJR7LEgU8Ks00jO/sBDWMQBXh8YrM4m8Ylic0wYPKQpeE7eg2OwY Tqh8kRB3aJ1toiugHqq+pLRRr2Ffttv5ZAoySLmOeMKw+atdLi0L8nawTSVrCuPjsHe9Ex aGS7FIHNCu3/cHfw6TBbJHogIUeg6yw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=izi9UHkrSeZ4AHa49u6aA8W9HneeYs8L2V/Y70LmMpM=; b=d7r+keQYKAY+C5D+bh+Jw3luE7skptr1cANVf2PwkM/S7W3B0lTHmM1kKkSuGYbBp/IY7q hQoWY+UkkAmUad+K+7ewNHa+bJcW42Vostv1OnCOfORo2QcdVy3BgiK9CXZ72Oxbv8aUP0 299PjnKn9NLOd083rD/Rcsphay37w38= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-267-UUYzhPw5OJ2eJI-qriS1tw-1; Thu, 20 Jul 2023 12:33:54 -0400 X-MC-Unique: UUYzhPw5OJ2eJI-qriS1tw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CEF671C05EB9; Thu, 20 Jul 2023 16:33:50 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BCF3340C2070; Thu, 20 Jul 2023 16:33:42 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 14/20] x86/kvm: Make kvm_async_pf_enabled __ro_after_init Date: Thu, 20 Jul 2023 17:30:50 +0100 Message-Id: <20230720163056.2564824-15-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 727A68001C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: wz1zwb14tmarod84gkf8sw1tx6zo16a9 X-HE-Tag: 1689870839-618310 X-HE-Meta: U2FsdGVkX1+WkJ1HGZ27zG2pvHL0R+fBYDQSlKFvxz24HAk5zcWvXLAa8755ZFiyBXHJhRJNNa1dvpGmSrhoNhzqBLdQxJwo+cG2jFVXwfPdFcSUMTVE1KJ2mRXlS4HGImg+HC9JwmLiFiTisZ36NYMDkvazgDTQAOGXZr7JuoaNviFfFesO1WU9sjZM1t1PbCZjwRTJ39+1YFPGvfF0wFde/c8O5KwVTPQ667/g6kkm4DNi2U10Plrz6camYwHiS3I0wYBQBD6vVMrennm3wvHhVw+44gPnSmfWz122Nd9Ta0cTcIB/mEqjlexeyFeuche7JQghTt6x3McP6IBnXE2xldk0MuausVNNcY0HwS2AFWTxeD1BWoIT5SfsIdkG0Ki1J4aMjl99bwuuBOjAa56CrJizBOAsrrIgWvK5tP40J2wPRBIp3Ioh5PucrViQv2G43wsF0RfpPXdDrjiTocO1z83oa88BQxVP2DYfY6p4g2SVMzyr0o+eDWAgaww1USulvNBmgOgvdzRyjuIgRfoXVamOihGttxcxrJTHb+XORD1BXcaN8CrD3h7Vod30DbZ+D4oU/TrCMwOTSDsKjcCVM/zgYhLUXHLxsQX4aKiQhgZyTPsO9978uumzdAqArfyngnKCod8zVUxwc377Bxuf0RCenQeLVadND+XC6G00TA0vBnuQj+5R0VDJ2mZiRTzUk6H0r9e2IzNUA+eEME9scFnjCmpKNEs6WIY6aBGEwhSYbc2G+V+6GS/mY58Zd3rAW0f2zMTzJd3VvZVmVhir9KJAcCoa5jvUPNcKN+EptZR0Y2r/kvtwLNw6d5krxAaQzVhEoe5NEnZsZm5LZD3kv+46lkGkLZtQQ4uvvFiAKnZilU3GIA1m5vELJDhDHk+/yEbubUeHofskZxdAtWoyj8Wg/ZbKPSIWLJ3h+tlDtEuyAa9jfjo+YHgZ6DofS4ugm5c+9wsTgL01HCP vQHe0bfC R7iGchGvoAAaHaZGt+KKPNn5Hhzs3m1KLYsxGu0GciLGYPMeCLWMX+lUtF2CcBjr8Zt4EKGwQwhjhqjxwPHKwx6qi6mDvvnDTuVI/A9pfQC7d4Ek21peJTfSjpG1Pi7DJBpVIykEXmj/RQYj7Cj5KKX9nFqa6oHTqYzT6LESkE6fehryx7KkD7hEQe7U3W+9bkS5t2lpArLubhiwN2RAFRORdrqpK43dNbxiMKi9ZJraZRKjkUJzfATrmNZDqkgRmYUMIbZ+jV+7mPI8u6dnkPLocJmecKnJcKRKF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: objtool now warns about it: vmlinux.o: warning: objtool: exc_page_fault+0x2a: Non __ro_after_init static key "kvm_async_pf_enabled" in .noinstr section The key can only be enabled (and not disabled) in the __init function kvm_guest_init(), so mark it as __ro_after_init. Signed-off-by: Valentin Schneider Reviewed-by: Maxim Levitsky --- arch/x86/kernel/kvm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 1cceac5984daa..319460090a836 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -44,7 +44,7 @@ #include #include -DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled); +DEFINE_STATIC_KEY_FALSE_RO(kvm_async_pf_enabled); static int kvmapf = 1; From patchwork Thu Jul 20 16:30:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49B4BEB64DC for ; Thu, 20 Jul 2023 16:34:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE8A228013D; Thu, 20 Jul 2023 12:34:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D715028004C; Thu, 20 Jul 2023 12:34:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC57E28013D; Thu, 20 Jul 2023 12:34:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A97DD28004C for ; Thu, 20 Jul 2023 12:34:07 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7C76F401D8 for ; Thu, 20 Jul 2023 16:34:07 +0000 (UTC) X-FDA: 81032537334.12.356B6C9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 87CBA4001A for ; Thu, 20 Jul 2023 16:34:05 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e0aWU2El; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870845; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cIltxl7gcT/0RKwYlLQ4QEv0npfMouojSk9ktXBKfqI=; b=QHeKE4HWfli9uy/io7zPZGBzA9akeDl+k0LkSKYprLCsXI9DQ4xaSrvuJ7+MHZTasnKhzw iUZV1zRnwLmCVs7XbC+Z2cxsUZl+c9ox/Hb3cyAk9SRBWxeXZf6gZrt7eNMLLrXguf5FWH djVOOQH54CO3gZkIK8kpmbGn8Om7N5s= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e0aWU2El; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870845; a=rsa-sha256; cv=none; b=1R5VVUBmpd7XE07ro3Ol2Q4S+kEkiUkOjljAyfaPQObep6drKBkrpq+zYLp8I4z/Ht1Ym+ CUa3Hf/4QVs1ARjJQtHY4sE2zru9g8A7dGbPxAHjBOEBT8Q4ZAMIcI1zxTaEDDxtW33oys hx0I6fzLfW1Hg3QdKPrEpQNlA0J0Ols= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cIltxl7gcT/0RKwYlLQ4QEv0npfMouojSk9ktXBKfqI=; b=e0aWU2El2R+LrUbSPviYx3UQQnPC+CPqr5Q9K3HVi3OJTC5mMWnwZ6RHVm5DfelzepAFid j6GB+ezYxLAnqcrLQQj+MFJKnE9X2N0b25IZUDvO6Pn9BQ2b/455snApzZ2S+YnGFHkvDJ IbZwCBpgfSRpJym44TwrmCFWF1llIzg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-397-kJOWCWFhOl2lgITiWh1clA-1; Thu, 20 Jul 2023 12:34:03 -0400 X-MC-Unique: kJOWCWFhOl2lgITiWh1clA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E3F4B936D34; Thu, 20 Jul 2023 16:33:58 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1A11640C2070; Thu, 20 Jul 2023 16:33:51 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Nicolas Saenz Julienne , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 15/20] context-tracking: Introduce work deferral infrastructure Date: Thu, 20 Jul 2023 17:30:51 +0100 Message-Id: <20230720163056.2564824-16-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 87CBA4001A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: yp9bi4pefk971da9xqi9wg6mi6kqpi9o X-HE-Tag: 1689870845-944121 X-HE-Meta: U2FsdGVkX1/vPGICI6EphJTp2c0x8z7tj/VbcfS36DiHI3wanJw4D2u4a6fvvhafgPTVs0h0WpUhGl/0wany2nfINLXXr7rWSgZ7L7x4KRGZPqU2Oi9CqpJeCBtUV0B8hyfNi1r2TGaB4Zu4btGvLwNTJkomZYwRBoCocZUW6m/+2tAFXfA7MRuZu0EIr2LA3ryazGrJXutJ1DKBZVdJ41ic48ulfCTo2GVuuW7ApC/dOHIRgOyv3WoRqpJRxYyHbbU2GE9o1S68gNT8mjp6k+VnFJwTg1FNxK+0VYc0KlB6Hp/MghKkuyyIeSbhGi1ICXajaXUAPFGkJHthnvlxwqjoPjdIrlFT/9ybpPz7tVopDMTj5stoXBJIq7AqbAVG1FtlCz85fDE9/jl/jzqq21vlnOlTZvHbzDS8pwHP/CDRt+VQZBMCp9B7J2P3ljeiX7MyXdFtwtJKl+EhoI0qcNoYMc39ztYFs4atvhePRAd6lgBxczpujYeQJ+JalK/Ku3OibvpkyLKqXkprY0XeNGeq/49a3lfcTm+ehauvIJZarF+ZL9rntpjhbWDkmWySeRKgQmvKww6vMn4wZSP1qIo3NT9FLcgUsP8hTWM+c8Ldr2mq6lz8D/9JIqOr2PNlOMy9VnVgOV+U8K+ZRcBhrTYsf4WgLZfrhl+C1NAC5qc2EOkIDgZrfLvSFoAKATJAV/4eY0iHAcGUExPY2GLkcHIZqnFOvtAKI4zoQuMyCI4U4M7ZNoiPmA4S1jnVrXj+43IF4ASy44wk3ZdyaQ3Xs5/Sk80Ci0mVXzPChJYVABoIURSBbF2S9MDGFC8Ss3MPVwL3WCKB5sXEjkkaL7axtVlqchOst1ifgXlIJoF6HHhyVbSshKBped+6PZ1lRgColPAo6qM9bP9SmyJGBDnl5RVRKqZ2DNwuUtUVPz4YggjS+rLPHk5SyX/uNuLe2HSclxFBBs6q/bFZdlo7uSa S9hHWuDL kg0311yCabHhpNf9GeJTNo/65/OTRQKOhscTgoeiuMe+HZaVipVEwXi94v+XjbK7pzVyCO9qFEBbZYOMbVe0J5XBk9gP3ne0Pfme9JXfuWcH6ZrTfzsi3EeF28RDzLvPwWAYLI1rnUPFgYJ51vtiHqjuMlDrUIPIT0pR51x6NW06lvIf4nWNVn932881IaeE6k2HvM06svJ4u8K6GR1I84Sa40XKuvk2t4bbmF3WQ/8ywvN3z9GK4XEtzYDpWf2UJs9TCAHkIFdafzZGyyovZOd/acIA5NxHIRGHO/sG8J1Ed53V7lfiHGSGqF8y/pG2XO8t00IN1LzisjFri3svfuHKmBOGkpaF6SiEhecd7vsBOcUc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: smp_call_function() & friends have the unfortunate habit of sending IPIs to isolated, NOHZ_FULL, in-userspace CPUs, as they blindly target all online CPUs. Some callsites can be bent into doing the right, such as done by commit: cc9e303c91f5 ("x86/cpu: Disable frequency requests via aperfmperf IPI for nohz_full CPUs") Unfortunately, not all SMP callbacks can be omitted in this fashion. However, some of them only affect execution in kernelspace, which means they don't have to be executed *immediately* if the target CPU is in userspace: stashing the callback and executing it upon the next kernel entry would suffice. x86 kernel instruction patching or kernel TLB invalidation are prime examples of it. Reduce the RCU dynticks counter width to free up some bits to be used as a deferred callback bitmask. Add some build-time checks to validate that setup. Presence of CONTEXT_KERNEL in the ct_state prevents queuing deferred work. Later commits introduce the bit:callback mappings. Link: https://lore.kernel.org/all/20210929151723.162004989@infradead.org/ Signed-off-by: Nicolas Saenz Julienne Signed-off-by: Valentin Schneider --- arch/Kconfig | 9 +++ arch/x86/Kconfig | 1 + arch/x86/include/asm/context_tracking_work.h | 14 +++++ include/linux/context_tracking.h | 25 ++++++++ include/linux/context_tracking_state.h | 62 +++++++++++++++----- include/linux/context_tracking_work.h | 26 ++++++++ kernel/context_tracking.c | 51 +++++++++++++++- kernel/time/Kconfig | 5 ++ 8 files changed, 176 insertions(+), 17 deletions(-) create mode 100644 arch/x86/include/asm/context_tracking_work.h create mode 100644 include/linux/context_tracking_work.h diff --git a/arch/Kconfig b/arch/Kconfig index aff2746c8af28..1bcb3bbdddaad 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -871,6 +871,15 @@ config HAVE_CONTEXT_TRACKING_USER_OFFSTACK - No use of instrumentation, unless instrumentation_begin() got called. +config HAVE_CONTEXT_TRACKING_WORK + bool + help + Architecture supports deferring work while not in kernel context. + This is especially useful on setups with isolated CPUs that might + want to avoid being interrupted to perform housekeeping tasks (for + ex. TLB invalidation or icache invalidation). The housekeeping + operations are performed upon re-entering the kernel. + config HAVE_TIF_NOHZ bool help diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7422db4097701..71481a80774f6 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -198,6 +198,7 @@ config X86 select HAVE_CMPXCHG_LOCAL select HAVE_CONTEXT_TRACKING_USER if X86_64 select HAVE_CONTEXT_TRACKING_USER_OFFSTACK if HAVE_CONTEXT_TRACKING_USER + select HAVE_CONTEXT_TRACKING_WORK if X86_64 select HAVE_C_RECORDMCOUNT select HAVE_OBJTOOL_MCOUNT if HAVE_OBJTOOL select HAVE_OBJTOOL_NOP_MCOUNT if HAVE_OBJTOOL_MCOUNT diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h new file mode 100644 index 0000000000000..5bc29e6b2ed38 --- /dev/null +++ b/arch/x86/include/asm/context_tracking_work.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CONTEXT_TRACKING_WORK_H +#define _ASM_X86_CONTEXT_TRACKING_WORK_H + +static __always_inline void arch_context_tracking_work(int work) +{ + switch (work) { + case CONTEXT_WORK_n: + // Do work... + break; + } +} + +#endif diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 6e76b9dba00e7..8aee086d0a25f 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -5,10 +5,15 @@ #include #include #include +#include #include #include +#ifdef CONFIG_CONTEXT_TRACKING_WORK +static_assert(CONTEXT_WORK_MAX_OFFSET <= CONTEXT_WORK_END + 1 - CONTEXT_WORK_START, + "Not enough bits for CONTEXT_WORK"); +#endif #ifdef CONFIG_CONTEXT_TRACKING_USER extern void ct_cpu_track_user(int cpu); @@ -131,6 +136,26 @@ static __always_inline unsigned long ct_state_inc(int incby) return raw_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state)); } +#ifdef CONTEXT_TRACKING_WORK +static __always_inline unsigned long ct_state_inc_clear_work(int incby) +{ + struct context_tracking *ct = this_cpu_ptr(&context_tracking); + unsigned long new, old, state; + + state = arch_atomic_read(&ct->state); + do { + old = state; + new = old & ~CONTEXT_WORK_MASK; + new += incby; + state = arch_atomic_cmpxchg(&ct->state, old, new); + } while (old != state); + + return new; +} +#else +#define ct_state_inc_clear_work(x) ct_state_inc(x) +#endif + static __always_inline bool warn_rcu_enter(void) { bool ret = false; diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index bbff5f7f88030..828fcdb801f73 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -9,21 +9,6 @@ /* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ #define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) -enum ctx_state { - CONTEXT_DISABLED = -1, /* returned by ct_state() if unknown */ - CONTEXT_KERNEL = 0, - CONTEXT_IDLE = 1, - CONTEXT_USER = 2, - CONTEXT_GUEST = 3, - CONTEXT_MAX = 4, -}; - -/* Even value for idle, else odd. */ -#define RCU_DYNTICKS_IDX CONTEXT_MAX - -#define CT_STATE_MASK (CONTEXT_MAX - 1) -#define CT_DYNTICKS_MASK (~CT_STATE_MASK) - struct context_tracking { #ifdef CONFIG_CONTEXT_TRACKING_USER /* @@ -44,6 +29,53 @@ struct context_tracking { #endif }; +enum ctx_state { + /* Following are values */ + CONTEXT_DISABLED = -1, /* returned by ct_state() if unknown */ + CONTEXT_KERNEL = 0, + CONTEXT_IDLE = 1, + CONTEXT_USER = 2, + CONTEXT_GUEST = 3, + CONTEXT_MAX = 4, +}; + +/* + * We cram three different things within the same atomic variable: + * + * CONTEXT_STATE_END RCU_DYNTICKS_END + * | CONTEXT_WORK_END | + * | | | + * v v v + * [ context_state ][ context work ][ RCU dynticks counter ] + * ^ ^ ^ + * | | | + * | CONTEXT_WORK_START | + * CONTEXT_STATE_START RCU_DYNTICKS_START + */ + +#define CT_STATE_SIZE (sizeof(((struct context_tracking *)0)->state) * BITS_PER_BYTE) + +#define CONTEXT_STATE_START 0 +#define CONTEXT_STATE_END (bits_per(CONTEXT_MAX - 1) - 1) + +#define RCU_DYNTICKS_BITS (IS_ENABLED(CONFIG_CONTEXT_TRACKING_WORK) ? 16 : 31) +#define RCU_DYNTICKS_START (CT_STATE_SIZE - RCU_DYNTICKS_BITS) +#define RCU_DYNTICKS_END (CT_STATE_SIZE - 1) +#define RCU_DYNTICKS_IDX BIT(RCU_DYNTICKS_START) + +#define CONTEXT_WORK_START (CONTEXT_STATE_END + 1) +#define CONTEXT_WORK_END (RCU_DYNTICKS_START - 1) + +/* Make sure all our bits are accounted for */ +static_assert((CONTEXT_STATE_END + 1 - CONTEXT_STATE_START) + + (CONTEXT_WORK_END + 1 - CONTEXT_WORK_START) + + (RCU_DYNTICKS_END + 1 - RCU_DYNTICKS_START) == + CT_STATE_SIZE); + +#define CT_STATE_MASK GENMASK(CONTEXT_STATE_END, CONTEXT_STATE_START) +#define CT_WORK_MASK GENMASK(CONTEXT_WORK_END, CONTEXT_WORK_START) +#define CT_DYNTICKS_MASK GENMASK(RCU_DYNTICKS_END, RCU_DYNTICKS_START) + #ifdef CONFIG_CONTEXT_TRACKING DECLARE_PER_CPU(struct context_tracking, context_tracking); #endif diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h new file mode 100644 index 0000000000000..fb74db8876dd2 --- /dev/null +++ b/include/linux/context_tracking_work.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_CONTEXT_TRACKING_WORK_H +#define _LINUX_CONTEXT_TRACKING_WORK_H + +#include + +enum { + CONTEXT_WORK_n_OFFSET, + CONTEXT_WORK_MAX_OFFSET +}; + +enum ct_work { + CONTEXT_WORK_n = BIT(CONTEXT_WORK_n_OFFSET), + CONTEXT_WORK_MAX = BIT(CONTEXT_WORK_MAX_OFFSET) +}; + +#include + +#ifdef CONFIG_CONTEXT_TRACKING_WORK +extern bool ct_set_cpu_work(unsigned int cpu, unsigned int work); +#else +static inline bool +ct_set_cpu_work(unsigned int cpu, unsigned int work) { return false; } +#endif + +#endif diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index cc4f3a57f848c..1a3f6e355826d 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -72,6 +72,51 @@ static __always_inline void rcu_dynticks_task_trace_exit(void) #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } +#ifdef CONFIG_CONTEXT_TRACKING_WORK +static noinstr void ct_work_flush(unsigned long seq) +{ + int bit; + + seq = (seq & CT_WORK_MASK) >> CONTEXT_WORK_START; + + /* + * arch_context_tracking_work() must be noinstr, non-blocking, + * and NMI safe. + */ + for_each_set_bit(bit, &seq, CONTEXT_WORK_MAX) + arch_context_tracking_work(BIT(bit)); +} + +bool ct_set_cpu_work(unsigned int cpu, unsigned int work) +{ + struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); + unsigned int old; + bool ret = false; + + preempt_disable(); + + old = atomic_read(&ct->state); + /* + * Try setting the work until either + * - the target CPU no longer accepts any more deferred work + * - the work has been set + * + * NOTE: CONTEXT_GUEST intersects with CONTEXT_USER and CONTEXT_IDLE + * as they are regular integers rather than bits, but that doesn't + * matter here: if any of the context state bit is set, the CPU isn't + * in kernel context. + */ + while ((old & (CONTEXT_GUEST | CONTEXT_USER | CONTEXT_IDLE)) && !ret) + ret = atomic_try_cmpxchg(&ct->state, &old, old | (work << CONTEXT_WORK_START)); + + preempt_enable(); + return ret; +} +#else +static __always_inline void ct_work_flush(unsigned long work) { } +static __always_inline void ct_work_clear(struct context_tracking *ct) { } +#endif + /* * Record entry into an extended quiescent state. This is only to be * called when not already in an extended quiescent state, that is, @@ -88,7 +133,8 @@ static noinstr void ct_kernel_exit_state(int offset) * next idle sojourn. */ rcu_dynticks_task_trace_enter(); // Before ->dynticks update! - seq = ct_state_inc(offset); + seq = ct_state_inc_clear_work(offset); + // RCU is no longer watching. Better be in extended quiescent state! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & RCU_DYNTICKS_IDX)); } @@ -100,7 +146,7 @@ static noinstr void ct_kernel_exit_state(int offset) */ static noinstr void ct_kernel_enter_state(int offset) { - int seq; + unsigned long seq; /* * CPUs seeing atomic_add_return() must see prior idle sojourns, @@ -108,6 +154,7 @@ static noinstr void ct_kernel_enter_state(int offset) * critical section. */ seq = ct_state_inc(offset); + ct_work_flush(seq); // RCU is now watching. Better not be in an extended quiescent state! rcu_dynticks_task_trace_exit(); // After ->dynticks update! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & RCU_DYNTICKS_IDX)); diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index bae8f11070bef..fdb266f2d774b 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -181,6 +181,11 @@ config CONTEXT_TRACKING_USER_FORCE Say N otherwise, this option brings an overhead that you don't want in production. +config CONTEXT_TRACKING_WORK + bool + depends on HAVE_CONTEXT_TRACKING_WORK && CONTEXT_TRACKING_USER + default y + config NO_HZ bool "Old Idle dynticks config" help From patchwork Thu Jul 20 16:30:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EA7BC04A6A for ; Thu, 20 Jul 2023 16:34:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 387D428013E; Thu, 20 Jul 2023 12:34:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3383928004C; Thu, 20 Jul 2023 12:34:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B38128013E; Thu, 20 Jul 2023 12:34:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0B79028004C for ; Thu, 20 Jul 2023 12:34:15 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D18C0B0C7E for ; Thu, 20 Jul 2023 16:34:14 +0000 (UTC) X-FDA: 81032537628.15.E5040A7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id DFA1D40020 for ; Thu, 20 Jul 2023 16:34:12 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=djwRwyoR; spf=pass (imf27.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870852; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gPdSzcTTJlOeO2aEdgyECLhQy37txJLWdi6TDmez9kY=; b=SGmVuxrsmuDLcEuNjQIDKdQwX+YkGmOVqRX8HNr3gI2dasH5F32gv93atXi3RFD/LWZeU6 6KwCyjfbDcWc2fOxu+Cko2nJvDQF3VqhZ1b8t2XzH1o4tjEQb9PSifYt1G7WpkLf1hy9sQ xYRcC5wnEXytwZg3a2UmcanrkK0wzek= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870852; a=rsa-sha256; cv=none; b=AQlCeA0aX0MXTNd8NiqeNEXorNys5CaL8zoLJ9vhQc8KJdQXtb5FBLgRcfeWap4GXarXI1 nI7bls6wOMoNMzNDFvmhbuvps32oWTQIxfsbPQwkyFt6rFxWQAEdfuXDxWB6jIdqaNnaOL TVs23M++An3CTzruEBAO0ZIOxdjOClc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=djwRwyoR; spf=pass (imf27.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gPdSzcTTJlOeO2aEdgyECLhQy37txJLWdi6TDmez9kY=; b=djwRwyoRT4WBUlXX5F+QuYV/5Oo8VHV8+ZA2T1d4Bn3OxZW/wZbbtOEmBgsHyZr9dya+i9 6hCXz8OxlkYgsku+TWVuFaCg5Crl4eMjDPyswWwSD26caHOL/pLzL1SGCjubgxaF1TaJ5v O6qMRxE35rhDtVJiHPvrH1OSUxDkCoA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-230-kFK-c7oyNgC289gF1X8KZw-1; Thu, 20 Jul 2023 12:34:11 -0400 X-MC-Unique: kFK-c7oyNgC289gF1X8KZw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4FA07101A54E; Thu, 20 Jul 2023 16:34:08 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A9240C206F; Thu, 20 Jul 2023 16:33:59 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: "Paul E . McKenney" , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 16/20] rcu: Make RCU dynticks counter size configurable Date: Thu, 20 Jul 2023 17:30:52 +0100 Message-Id: <20230720163056.2564824-17-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: DFA1D40020 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ihq1a3opemecedihex1f39grxts63bhm X-HE-Tag: 1689870852-990100 X-HE-Meta: U2FsdGVkX18y5TtrkaiJGG+DI/3qMI5I5L+6zCxJT4MQw4cJgHbbVfSR/FIsKWg27CHczHq7bV7ZlSryhoG5BHjFWbhfrmQOZXfXWABByH7tF7vShECFN6ZDvrAuPugyEpRtXiSC+43IeXiknGVnySCuSyDog0zvqrVZp0qmwqYofLY2XjqFAh1J+AnCCdXyi5n2I9iCLLWCiWVILs+Zb4wucApKLMmy/fLOoI1EV4fjY54l3W/N0fUN6d0GkXwzOfVdUMxvFeib5DVgfzqLZV63NmFKBk/qLv8cD1C/EEU6sOJanpOBEjcs2AbjBF6AUeQf3i7/hIn4WkhwMkiN+46WVMpCAz79Ut+zAFEpZ5Gi964cOo3ZEhTynyykZ2o5BbYCFmsTC+Jb4+l6jf5PbfC7k7LZRMKeTK+80OSitFHnvqZwe60D2CieN7ZPFTuPoLq++m4+zaursMuP3wBFwvpYqtlfYCW1CWmDcrVi7NG9riEI83IVEqIFiB9U4M3f6zryfP1QOExiV2Xh/Yka1vPWfZ5WVFliOYT91yDi7Z6sEKW9ikPG4MEkvMOpF5jMGSRaFeNhZUbY5qNaq4BsaLiyLTM5Ku0mLjYoSl7spoFDWfpcbagEBQgEZbncdODnwpHil3wFl8n21TbTL+cBGh55U3yYejyQedELyglr7hEcSMaNNehmUJUMdUogeisCv15XiY9tVfkJQxFkoXrpsFZF03rzYMKPBFNeQnhKufVXD8B+Bq7n7bnq5f72MBgWWcrawTFZ1C3qeDliKWcLaEdjpRF/fsyrweZva2X471MA2gK2TgAe4D0dONx13yzMUhraDb/fsTfOon8ejSZEYvMCxLeUGo6RHPJo5XCLJGdFflWC/BfHaSGtB9TmdUiYbdbYJKjp3SFPNmIa83gs331/2EuV9lyqBFjWE6JKVPVqRAResdkKx7o9EcW2ChswnddgkeM6WZm1yl66GKj 7/sC83pt DqbcuUb5K/TXG6YI/ksrZqJBT/IQXeuwvR2ZpCfY4Z2GGC/aNat51zTiTHcakeKRag0ahHuc0RJHanC/HWghay23zjf3u1t9FrKPdFqqXeWIzxrrF530Da6/kpK67K2zKmaiZr8RR34AooBiw/+mfTpDX4bmoLl654qFS1JgbBRD/1UwmfYqQgDUi+rUVtdKjH4XS/HPjv+u4qFYJmFbxKHY7K9o6zDY0WCJNqC7w3CKYVAlq9T30IvlnC9u+X4Ch96Kz2eJU5of6xh8zQq+OGxc8yePIxjfk/BZQDcDCUotF7wNKUsdn/bNZeQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: CONTEXT_TRACKING_WORK reduces the size of the dynticks counter to free up some bits for work deferral. Paul suggested making the actual counter size configurable for rcutorture to poke at, so do that. Make it only configurable under RCU_EXPERT. Previous commits have added build-time checks that ensure a kernel with problematic dynticks counter width can't be built. Link: http://lore.kernel.org/r/4c2cb573-168f-4806-b1d9-164e8276e66a@paulmck-laptop Suggested-by: Paul E. McKenney Signed-off-by: Valentin Schneider --- include/linux/context_tracking.h | 3 ++- include/linux/context_tracking_state.h | 3 +-- kernel/rcu/Kconfig | 33 ++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 8aee086d0a25f..9c0c622bc27bb 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -12,7 +12,8 @@ #ifdef CONFIG_CONTEXT_TRACKING_WORK static_assert(CONTEXT_WORK_MAX_OFFSET <= CONTEXT_WORK_END + 1 - CONTEXT_WORK_START, - "Not enough bits for CONTEXT_WORK"); + "Not enough bits for CONTEXT_WORK, " + "CONFIG_RCU_DYNTICKS_BITS might be too high"); #endif #ifdef CONFIG_CONTEXT_TRACKING_USER diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 828fcdb801f73..292a0b7c06948 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -58,8 +58,7 @@ enum ctx_state { #define CONTEXT_STATE_START 0 #define CONTEXT_STATE_END (bits_per(CONTEXT_MAX - 1) - 1) -#define RCU_DYNTICKS_BITS (IS_ENABLED(CONFIG_CONTEXT_TRACKING_WORK) ? 16 : 31) -#define RCU_DYNTICKS_START (CT_STATE_SIZE - RCU_DYNTICKS_BITS) +#define RCU_DYNTICKS_START (CT_STATE_SIZE - CONFIG_RCU_DYNTICKS_BITS) #define RCU_DYNTICKS_END (CT_STATE_SIZE - 1) #define RCU_DYNTICKS_IDX BIT(RCU_DYNTICKS_START) diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index bdd7eadb33d8f..1ff2aab24e964 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -332,4 +332,37 @@ config RCU_DOUBLE_CHECK_CB_TIME Say Y here if you need tighter callback-limit enforcement. Say N here if you are unsure. +config RCU_DYNTICKS_RANGE_BEGIN + int + depends on !RCU_EXPERT + default 31 if !CONTEXT_TRACKING_WORK + default 16 if CONTEXT_TRACKING_WORK + +config RCU_DYNTICKS_RANGE_BEGIN + int + depends on RCU_EXPERT + default 2 + +config RCU_DYNTICKS_RANGE_END + int + default 31 if !CONTEXT_TRACKING_WORK + default 16 if CONTEXT_TRACKING_WORK + +config RCU_DYNTICKS_BITS_DEFAULT + int + default 31 if !CONTEXT_TRACKING_WORK + default 16 if CONTEXT_TRACKING_WORK + +config RCU_DYNTICKS_BITS + int "Dynticks counter width" if CONTEXT_TRACKING_WORK + range RCU_DYNTICKS_RANGE_BEGIN RCU_DYNTICKS_RANGE_END + default RCU_DYNTICKS_BITS_DEFAULT + help + This option controls the width of the dynticks counter. + + Lower values will make overflows more frequent, which will increase + the likelihood of extending grace-periods. + + Don't touch this unless you are running some tests. + endmenu # "RCU Subsystem" From patchwork Thu Jul 20 16:30:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6B01EB64DC for ; Thu, 20 Jul 2023 16:34:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66E4C28013F; Thu, 20 Jul 2023 12:34:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61F2128004C; Thu, 20 Jul 2023 12:34:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4724828013F; Thu, 20 Jul 2023 12:34:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3A33128004C for ; Thu, 20 Jul 2023 12:34:25 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 100518020D for ; Thu, 20 Jul 2023 16:34:25 +0000 (UTC) X-FDA: 81032538090.02.D8FBA0B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 53F97180011 for ; Thu, 20 Jul 2023 16:34:23 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MpjZgTy+; spf=pass (imf06.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870863; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D+WoXJvLSOgt+Kve5Q+fkVLRoN9HdK3x6VSzpxIN5vU=; b=z3PE6kDY7xmCE65jNul/aEpe8+tY/g9aV5TUkOpqXWuQm9GxsBJfb8rgB1Tqtkz6GzxCr5 V27A0ksV3HCHd0P9DAuT+HD1MfqPQW8m27x8S1EHnipyGfvAyynpoaXKsxKwL2Q+EsQ0tZ 0+o95IjsJqWWdhTSRqOSYA9CF3dqutg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870863; a=rsa-sha256; cv=none; b=NcFuVCJ+R1RW1imgk6sPjj65sGxkH9H0P33MwaPtzXguM29Ouv7X5ybu1bJ3IdkKTa+goU jXI112p9R+W0m/G2axuyrLfq2ekMa77CYxg9Dn+gkpWfhmRt5aqeZ44G6kNry5g1c3/ZFS JW5YbJLOhCJycC8uBiSGK7uLfbvcBA8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MpjZgTy+; spf=pass (imf06.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870862; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D+WoXJvLSOgt+Kve5Q+fkVLRoN9HdK3x6VSzpxIN5vU=; b=MpjZgTy+FyLJp4Tzz9180XUCw9rP7JhSugHK0mrM/krsclQajkpXnoDkEH8YjH04ol9t/K Y4t3U3MX8GbQqgqebokWfGt+xWDcsBLfNSupHacylk2AbRbFKPnBuNul0sYmXTOtFbraJT nx8E9+XPBhJ8hsMNVsFcveVZ6haxMww= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-225-n6Pvv7lTMfCgsmkY3ui-qw-1; Thu, 20 Jul 2023 12:34:19 -0400 X-MC-Unique: n6Pvv7lTMfCgsmkY3ui-qw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5937B185A78F; Thu, 20 Jul 2023 16:34:16 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8D32A40C2070; Thu, 20 Jul 2023 16:34:08 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: "Paul E . McKenney" , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 17/20] rcutorture: Add a test config to torture test low RCU_DYNTICKS width Date: Thu, 20 Jul 2023 17:30:53 +0100 Message-Id: <20230720163056.2564824-18-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 53F97180011 X-Rspam-User: X-Stat-Signature: ms1i71qegis487kstr63mks435fg7gtb X-Rspamd-Server: rspam03 X-HE-Tag: 1689870863-797680 X-HE-Meta: U2FsdGVkX1/SKRNZMr0OrDBr8ec1x0/4hCAmR5xxJkeGOcD9a+jRBufe4+yCXYJS0exutwy4o11wq3hwppcFtOjdAFAvtw+QLOjAvpUWhDVuGUlGVJtQsFizm/vLZjIGRD3d1hceHGa42YYzNiqrDibdFXRLTSMHLQ9mgqFrHLr1sfX4SjHJNTt1WhVdXXqv2Gym9ImUME6awUiD5cUuA2jWT92EEn0voo6PUasoBQwXnICSDcanxt5nPd88xGx5zBIIUO+Wy9j5Jv8kmsks103EV7iUGz/FqCmUjyOLdgJ7od0bREG6Fs61/Uz1tpOLVrxXW5vO5kyn4T5jTcXy11AQg+xUeyINyxKtwjMwAfBfw5slZuU1aBMZXO2knAq/6ZXdEmCVAxWd2pJE8VMhP7YGxZHS5jfMkgwa2Ixl79f58zWbm7zVJGosD0XLQvEcVwP+XwIwmHFFiSLtVQo7M9hY6oVBt4viZYqk08Kk5L8YXlDy07440m+fmW7WaBw/BB94+OVPOkb9idFryqDbk0k3fozwg+eeYNKu5icAaztBdvvYiH/LUzuA8O9ancK1u0AjNLD1fBn4P/ZkUJQFEexzCBZ5WJKMg2klMqJVOSCyEoNfhPqELvwXchIPV84slxJxvj6wo2wsHiK+rBGYpd+T8w4FIpcS/VKQxxRsVIA2gKroR+QdmDDs+2JYXGRVqSSf5NKzI5bxHUe3Ozu41IEu940KEOdGq1tnaDYiwZYF6XQ8WLkr0W8eXf6KWWq7Tdwec3gsWQgHOZVszPAQs4DIZ58Spz4/rC6ABeKBIAVMePs5wTMBF8fj/F3X22AE4zp0rXZ+S5LLHBEAwgrVyHFR1STYaDDsarSqblmL4mC8EpPnfEfw+6T0I1ygm3j6JafuIetKOlNlx61w9XP84RzdG2D57NCCmgI2PGBl1lFYxAkok4nJ3j3jAuesbYj+1D1vhwTi04Rv2TPhrcb uG1YU8s7 kQzXi9c02y0w7WGQLxXP+nbU6YaA8GmbKDsF966dfUCEXpcoMcwsTjk4x85Ypihzxcy9ppg1W1QClIW9Jyk4NKOalHKXAflU0/N4P7lroAL14RANCyvMzVRgsdDT+XZfSRn5ghLNeY155Xl2c0CNMEMABZ2apAdPqDWxlW68BN2hY9VCJQsyrfZ3WBdTbyvgurrhnCtA4VMowMao4/SZUmGrDNY9i7ScGhz3xu9OaYoRRdktOkHr0zyCq4/c5AG6fsKcX9K1MuF2eNZD32AkE/QOq2cccGc5cV7qrxwsqSdwseIxCbkW0znDAhG4OtWd5e/CN1b5aWq2pOkPkHoe5glD8ktLPYFpIRNFe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We now have an RCU_EXPORT knob for configuring the size of the dynticks counter: CONFIG_RCU_DYNTICKS_BITS. Add a torture config for a ridiculously small counter (2 bits). This is ac opy of TREE4 with the added counter size restriction. Link: http://lore.kernel.org/r/4c2cb573-168f-4806-b1d9-164e8276e66a@paulmck-laptop Suggested-by: Paul E. McKenney Signed-off-by: Valentin Schneider --- .../selftests/rcutorture/configs/rcu/TREE11 | 19 +++++++++++++++++++ .../rcutorture/configs/rcu/TREE11.boot | 1 + 2 files changed, 20 insertions(+) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TREE11 create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE11 b/tools/testing/selftests/rcutorture/configs/rcu/TREE11 new file mode 100644 index 0000000000000..aa7274efd9819 --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE11 @@ -0,0 +1,19 @@ +CONFIG_SMP=y +CONFIG_NR_CPUS=8 +CONFIG_PREEMPT_NONE=n +CONFIG_PREEMPT_VOLUNTARY=y +CONFIG_PREEMPT=n +CONFIG_PREEMPT_DYNAMIC=n +#CHECK#CONFIG_TREE_RCU=y +CONFIG_HZ_PERIODIC=n +CONFIG_NO_HZ_IDLE=n +CONFIG_NO_HZ_FULL=y +CONFIG_RCU_TRACE=y +CONFIG_RCU_FANOUT=4 +CONFIG_RCU_FANOUT_LEAF=3 +CONFIG_DEBUG_LOCK_ALLOC=n +CONFIG_DEBUG_OBJECTS_RCU_HEAD=n +CONFIG_RCU_EXPERT=y +CONFIG_RCU_EQS_DEBUG=y +CONFIG_RCU_LAZY=y +CONFIG_RCU_DYNTICKS_BITS=2 diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot b/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot new file mode 100644 index 0000000000000..a8d94caf7d2fd --- /dev/null +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE11.boot @@ -0,0 +1 @@ +rcutree.rcu_fanout_leaf=4 nohz_full=1-N From patchwork Thu Jul 20 16:30:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D66E3EB64DA for ; Thu, 20 Jul 2023 16:34:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77FBB280140; Thu, 20 Jul 2023 12:34:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E14428004C; Thu, 20 Jul 2023 12:34:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55BB1280140; Thu, 20 Jul 2023 12:34:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4648F28004C for ; Thu, 20 Jul 2023 12:34:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0EEAB1C83FE for ; Thu, 20 Jul 2023 16:34:35 +0000 (UTC) X-FDA: 81032538510.12.8A3C076 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 2894B40029 for ; Thu, 20 Jul 2023 16:34:32 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iYAu5M6z; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870873; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QESi3DN0fC8QnZsNV/N4VXfEg//bCfMhfN1lMbJjDRM=; b=cPNn0LYggzTCkxYRrImN591VHrIT7KrCOV67fsBjfMpUnxhaJ9aEijc59Qw7ybhybmbU82 5RCFAAnVdT+YT87VAoOb3w7Lo9RHvF8c5FD2V7v6UrBPkiJ+XywqLIVH02t9qTQn6ELPjX DGUFs3pxmBwcLRPwIfr3g44hU7g9FHo= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iYAu5M6z; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870873; a=rsa-sha256; cv=none; b=m/4K/Nq3acHKd+ZbWHokykSnCUdJ5UlBUScHG9dF26TVk9Dn4yweturq3goot5eMjuuyEx mVmXgZALTk0VJUa1GpJZvTosLB4hN73/k5T2Ij28AFhWsowUrqYLV785UG111CMLE+or1b pUYFfJCLtKBYQhym/1TMb6SrtPDlvr0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QESi3DN0fC8QnZsNV/N4VXfEg//bCfMhfN1lMbJjDRM=; b=iYAu5M6zCjLKYgJd6MiOOy3pI5pJVoRh09d3p7UQMYpLtVWvh8Bvx5FAbYsN9eYWps6V5B +SADsAHX0QlRfTJolto6JT/WTzG/qLL0D4RYvB3fxcBSzdatiiiMEC8tu7qXYvZ3aK/QQv euU7onrEbbSchANCayhSlaSnBpiYlqo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-183-lKQNgwDjNBmDn1pKucKVhA-1; Thu, 20 Jul 2023 12:34:27 -0400 X-MC-Unique: lKQNgwDjNBmDn1pKucKVhA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF85A936D34; Thu, 20 Jul 2023 16:34:24 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BFF9140C2070; Thu, 20 Jul 2023 16:34:16 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Peter Zijlstra , Nicolas Saenz Julienne , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 18/20] context_tracking,x86: Defer kernel text patching IPIs Date: Thu, 20 Jul 2023 17:30:54 +0100 Message-Id: <20230720163056.2564824-19-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 2894B40029 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 1bwkua64tr9fzz11h614j691z8j378no X-HE-Tag: 1689870872-684950 X-HE-Meta: U2FsdGVkX1/ZO9tVe+CO8YlLCzcfrAFK48+tpjFIDim5WjisCxEDcfu/BPG6zCRiBQCpsCuNU8rgye5e7JJneFuWyl9OP1g8vyykHxElTrDc48mxZWDf/UCGBdDvSUccAVqwYP/M50MvCcNQ+nb6qfOybhNJcqkOTQw6nHwRJX0BVaSeC/+/fZdZ6XH9BlgW9E3PZVsX/SGDjAtAggw7FH2D3tkOYGU7Cid1DBMo9fXDGa0PCYBJN5lcfLmCMvYFnumBE3+lUdgobnoqDRPeJ0meGh1QhxCLvbZfGEDWR4G8SPEmfXKugA8sTsLBZpaMNVgoOew/Ee0bT1+epGESYWF12TakVosaoqWlL6GWm7rN2HSzSKku+A8zhoWlmvXa+cjy/0DjVRE2FCOrgBZ6e8wyUzfbQRIX24DBLnJY/PGD7UeaXXoE2H12D16sB2NlOVWVUfTmbaqXjpuOwUowwtl7v35dwiLTnlnr8uCwQiBWfQ1UtN65um5FU7g+XstfVcEmhkslu0Ye4FnuxvmO14BngXOlmrVThDs1hSXbZYp/UHPjReaVv/htVWgWoQ17j1KHe3aT1g5EfcmC9JDXXpT4pZbrfCduoDgzJO6tmGmOtKjlfAS5GiwD8nOmeicDnDHkL35AZ9MvYhcdGPBS+neLv5rxZuSlWMbkATqDRNxCtoZkiy+bRqY4r/NQzgo8j3J+ab7hqp6LveAxbbun9kSUz9XCFAY5q2+J9dG3B9qAVA213zsexralFILquzcTxmYRXQJO8X2ihQ47EFN3CmK+Z/IsTSLjFCyiJqwmGkAAZKI5YLSqTb8FKMR25L+eogN+5KM4bv13YTsKeOUs+99HMii4zzmowLA/hjSETK4foaq6WY21/T9u2QyUrRizm43EP6e1Lih1zcAWd7CwGLKXq8vlZD/N/My+tfJ0YOdSaU/cOnCUULnk5qCIYAzY+fHKvfnAkwYYc617GqZ j1EwWqHJ 67mQ0BOBsvtx02SiBx4Xip2xxuspgyvOyvWFCsgxieNIhC5FitGnYMP85cACfTHc0lbRRg5/GLvJVZY5yCFg3DyWvpo1UqkYJwpsZsAJVScowSQHII1qH10g39oCFWelF88dTwtgG9xgUz6bD0kcLiPVNRx+hCRPfOq5K4M9TxrJ4jhVIW3xz4pQzIK+AR4ODKTmsDuAdUfqpFsP6UWJypHQYF+xLStE0veGWWoCOUocLC9fvAth4yp0c2PFBujFRyFYME0zohL3M7fS0kiuiwags5VxEF8GtC3ogrIDCoDTM4owBgfZE5IjjlDcXCIehfwm03z9tsdabswh/dNL4q9obK8D3xLUKBREm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: text_poke_bp_batch() sends IPIs to all online CPUs to synchronize them vs the newly patched instruction. CPUs that are executing in userspace do not need this synchronization to happen immediately, and this is actually harmful interference for NOHZ_FULL CPUs. As the synchronization IPIs are sent using a blocking call, returning from text_poke_bp_batch() implies all CPUs will observe the patched instruction(s), and this should be preserved even if the IPI is deferred. In other words, to safely defer this synchronization, any kernel instruction leading to the execution of the deferred instruction sync (ct_work_flush()) must *not* be mutable (patchable) at runtime. This means we must pay attention to mutable instructions in the early entry code: - alternatives - static keys - all sorts of probes (kprobes/ftrace/bpf/???) The early entry code leading to ct_work_flush() is noinstr, which gets rid of the probes. Alternatives are safe, because it's boot-time patching (before SMP is even brought up) which is before any IPI deferral can happen. This leaves us with static keys. Any static key used in early entry code should be only forever-enabled at boot time, IOW __ro_after_init (pretty much like alternatives). Objtool is now able to point at static keys that don't respect this, and all static keys used in early entry code have now been verified as behaving like so. Leverage the new context_tracking infrastructure to defer sync_core() IPIs to a target CPU's next kernel entry. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Nicolas Saenz Julienne Signed-off-by: Valentin Schneider --- arch/x86/include/asm/context_tracking_work.h | 6 +++-- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 24 ++++++++++++++++---- arch/x86/kernel/kprobes/core.c | 4 ++-- arch/x86/kernel/kprobes/opt.c | 4 ++-- arch/x86/kernel/module.c | 2 +- include/linux/context_tracking_work.h | 4 ++-- 7 files changed, 32 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h index 5bc29e6b2ed38..2c66687ce00e2 100644 --- a/arch/x86/include/asm/context_tracking_work.h +++ b/arch/x86/include/asm/context_tracking_work.h @@ -2,11 +2,13 @@ #ifndef _ASM_X86_CONTEXT_TRACKING_WORK_H #define _ASM_X86_CONTEXT_TRACKING_WORK_H +#include + static __always_inline void arch_context_tracking_work(int work) { switch (work) { - case CONTEXT_WORK_n: - // Do work... + case CONTEXT_WORK_SYNC: + sync_core(); break; } } diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index 29832c338cdc5..b6939e965e69d 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -43,6 +43,7 @@ extern void text_poke_early(void *addr, const void *opcode, size_t len); */ extern void *text_poke(void *addr, const void *opcode, size_t len); extern void text_poke_sync(void); +extern void text_poke_sync_deferrable(void); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern void *text_poke_copy(void *addr, const void *opcode, size_t len); extern void *text_poke_copy_locked(void *addr, const void *opcode, size_t len, bool core_ok); diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 72646d75b6ffe..fcce480e1919e 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -1933,9 +1934,24 @@ static void do_sync_core(void *info) sync_core(); } +static bool do_sync_core_defer_cond(int cpu, void *info) +{ + return !ct_set_cpu_work(cpu, CONTEXT_WORK_SYNC); +} + +static void __text_poke_sync(smp_cond_func_t cond_func) +{ + on_each_cpu_cond(cond_func, do_sync_core, NULL, 1); +} + void text_poke_sync(void) { - on_each_cpu(do_sync_core, NULL, 1); + __text_poke_sync(NULL); +} + +void text_poke_sync_deferrable(void) +{ + __text_poke_sync(do_sync_core_defer_cond); } /* @@ -2145,7 +2161,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries text_poke(text_poke_addr(&tp[i]), &int3, INT3_INSN_SIZE); } - text_poke_sync(); + text_poke_sync_deferrable(); /* * Second step: update all but the first byte of the patched range. @@ -2207,7 +2223,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries * not necessary and we'd be safe even without it. But * better safe than sorry (plus there's not only Intel). */ - text_poke_sync(); + text_poke_sync_deferrable(); } /* @@ -2228,7 +2244,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries } if (do_sync) - text_poke_sync(); + text_poke_sync_deferrable(); /* * Remove and wait for refs to be zero. diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index f7f6042eb7e6c..a38c914753397 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -735,7 +735,7 @@ void arch_arm_kprobe(struct kprobe *p) u8 int3 = INT3_INSN_OPCODE; text_poke(p->addr, &int3, 1); - text_poke_sync(); + text_poke_sync_deferrable(); perf_event_text_poke(p->addr, &p->opcode, 1, &int3, 1); } @@ -745,7 +745,7 @@ void arch_disarm_kprobe(struct kprobe *p) perf_event_text_poke(p->addr, &int3, 1, &p->opcode, 1); text_poke(p->addr, &p->opcode, 1); - text_poke_sync(); + text_poke_sync_deferrable(); } void arch_remove_kprobe(struct kprobe *p) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 57b0037d0a996..88451a744ceda 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -521,11 +521,11 @@ void arch_unoptimize_kprobe(struct optimized_kprobe *op) JMP32_INSN_SIZE - INT3_INSN_SIZE); text_poke(addr, new, INT3_INSN_SIZE); - text_poke_sync(); + text_poke_sync_deferrable(); text_poke(addr + INT3_INSN_SIZE, new + INT3_INSN_SIZE, JMP32_INSN_SIZE - INT3_INSN_SIZE); - text_poke_sync(); + text_poke_sync_deferrable(); perf_event_text_poke(op->kp.addr, old, JMP32_INSN_SIZE, new, JMP32_INSN_SIZE); } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b05f62ee2344b..8b4542dc51b6d 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -242,7 +242,7 @@ static int write_relocate_add(Elf64_Shdr *sechdrs, write, apply); if (!early) { - text_poke_sync(); + text_poke_sync_deferrable(); mutex_unlock(&text_mutex); } diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h index fb74db8876dd2..13fc97b395030 100644 --- a/include/linux/context_tracking_work.h +++ b/include/linux/context_tracking_work.h @@ -5,12 +5,12 @@ #include enum { - CONTEXT_WORK_n_OFFSET, + CONTEXT_WORK_SYNC_OFFSET, CONTEXT_WORK_MAX_OFFSET }; enum ct_work { - CONTEXT_WORK_n = BIT(CONTEXT_WORK_n_OFFSET), + CONTEXT_WORK_SYNC = BIT(CONTEXT_WORK_SYNC_OFFSET), CONTEXT_WORK_MAX = BIT(CONTEXT_WORK_MAX_OFFSET) }; From patchwork Thu Jul 20 16:30:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5DAFEB64DA for ; Thu, 20 Jul 2023 16:34:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 731F2280141; Thu, 20 Jul 2023 12:34:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E1F428004C; Thu, 20 Jul 2023 12:34:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53487280141; Thu, 20 Jul 2023 12:34:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4344628004C for ; Thu, 20 Jul 2023 12:34:40 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1B84D401D8 for ; Thu, 20 Jul 2023 16:34:40 +0000 (UTC) X-FDA: 81032538720.21.E1754C1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 4488220023 for ; Thu, 20 Jul 2023 16:34:37 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OcU9zw0h; spf=pass (imf13.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870878; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qlKugqrUieeoCK15s+XaSmSpmpWi+6cYdLHEecxMVFQ=; b=7fUZyN6PFF/QGuaeqOq2bgAf+ZrvF7BvaUr4vYDPy93gChxcIaO2Knw579h86tUDwveqq4 nuo5sHkR7PZKQVZvKVfUpjvf+fbkf32kemXxiKS9p058f+ARGbPMmm29XUbRXASDZuheam sRoB7v3mIsq1+6GPUbdTLt9pk3/8lMw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OcU9zw0h; spf=pass (imf13.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870878; a=rsa-sha256; cv=none; b=N60fUKVej2HDF1CAkybyJ9tlTgsp3nY3oAEc+y+Pn15m5k1PgHQwiG5ivoOwgooebohALL FD/wQPige0WXcsYC2xSR1fVnJryXctbtYbTwni1fjvgMAjYXnL5dOu9t5nWKsIRix5b/Wu Jrm3oHm6UJgb+6DUq6spvdkG0SzW/b8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qlKugqrUieeoCK15s+XaSmSpmpWi+6cYdLHEecxMVFQ=; b=OcU9zw0h7zBApjuoQctxqPIx+OV5Yk94dSB3XH1Cc4Mb2vtV1zGd2B6CHa9w0kLD4d5YxZ 09Lw5NcGCCla97nSdJLkcF/SDLGqlXwjev+LmeIdzpo3kddNrFyyAyJzQ4uJfoj6wdSTep xH60arkvKDWjLkvVfuUMjJga9KH9Sbc= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-240-myilGmoaNUKUY_DOur6_gg-1; Thu, 20 Jul 2023 12:34:35 -0400 X-MC-Unique: myilGmoaNUKUY_DOur6_gg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 95D7E3815EEF; Thu, 20 Jul 2023 16:34:32 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 28EAC40C2070; Thu, 20 Jul 2023 16:34:24 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 19/20] context_tracking,x86: Add infrastructure to defer kernel TLBI Date: Thu, 20 Jul 2023 17:30:55 +0100 Message-Id: <20230720163056.2564824-20-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 4488220023 X-Rspam-User: X-Stat-Signature: ry8698oo9wmkutqccaphuz18wd19oett X-Rspamd-Server: rspam01 X-HE-Tag: 1689870877-841647 X-HE-Meta: U2FsdGVkX1+JDD+f6Z6/oEf7AOf4jODjoCHLisIluzFCX915mPKnk3tbwMvgk8oL4Jd343QQ/vWh0rO+qjAdOBdV2MvWRqZYBBiqPyuxkJak78jsAUBuxsLIOFZyjwmS215FCzedjeP2MICACRJJaBjS4dIGicSqVX6dUvWgvKxA72eHvn15DQjl+B4fTHoVHoSXsnpdIUaSxh5n8rTPTeEkjcWnbJQ5Di1CvAjcR2iYYY7ziK3loz+nMtjy4fKW+8z8V5NetcDiDKhbhRpxxuQppVXydG/eHJCMOMZJG4R+FHY35s/wIGPGXZqgBu+qUAIqA+heXzIMPhvPpiN8/ChIhHntS1CSBFc3O5uNRkDH0DvZqQz+vj9dgtoXzxbWUTsLkMXaeO3MIzNwhwq9zxr0aClVmKx/FweWUu40zZs+UoO/mf/HVM3Brz+uceJpyML/P7tKMfn1PVuWTb+TGgI6KiSlhEUvIwHuWuX91Sqy4XWCVZK1H9x8Oj5StEjp8Kd7iZObgyNV6FrCmWc66kIwqGL6FPsu5UyBCsa+VxJkpY7qVwcA/dBWv7Z5T+aAHUss8eoYOA8MZ31us/4lHjkQijq+03DAG/J087s6EMDMJ9gSbm/ZvWleKIdLjeLC8vs6/VAY7zkFWWznuI9ipwJmW2I1a3bpByDiNGPXOgy/84jXu8b5YArv9qLmlt9F2LA2Ul78Js4nPzbIkPnjm5hRk9YTBqlSuRtEAxEwh2/VQ5OqaVaD9XcHzPcBXQ6UoWXhTic+WNm92xaG6NSpGaPfRkgIExHjMW++CfPsdyLqv7KKyXZEf/cnPDlL/XoA9cbk45PVL7P0IthCADorLGCtJKSdAGw1jNRj9slMr4D5d+QanX9gRs0DB5kqtgLyHhqdQ9cKJ019FPsRqAxFFmYGGpDdMW8pBSytyqdfvIRBHglrLFrIdYIRZPvm+plUnR/rPQUKP4zTwiOm4W2 E6U08hjT OHFHRWvccZUN6KZAU2V95Ff+0635nr4DqIkdqIHJrco1IXtMRbInFJUG0FBtcstaglQArc+ZX/GAQI39afyxCY1xT+Wf81qbc7VHGQeGSkrE+99W4orBtUZkgOegDQUXipWr129/NWwyH5GgMQCW6uJxCbkeDAFm6O5BGFdWkFa8IMT1Yvm8ioFxNWs0wehH2+LIqrpLlFWPnPuJiOXoLVP30Ym71+eNxa12tRW3RIBmJG3kYTV7xJp8GWPDPxtGZOiC9IuRtiayYMY7oeVegHL6h9xVfdegGMVyq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kernel TLB invalidation IPIs are a common source of interference on NOHZ_FULL CPUs. Given NOHZ_FULL CPUs executing in userspace are not accessing any kernel addresses, these invalidations do not need to happen immediately, and can be deferred until the next user->kernel transition. Rather than make __flush_tlb_all() noinstr, add a minimal noinstr variant that doesn't try to leverage INVPCID. FIXME: not fully noinstr compliant XXX: same issue as with ins patching, when do we access data that should be invalidated? Signed-off-by: Valentin Schneider --- arch/x86/include/asm/context_tracking_work.h | 4 ++++ arch/x86/include/asm/tlbflush.h | 1 + arch/x86/mm/tlb.c | 17 +++++++++++++++++ include/linux/context_tracking_state.h | 4 ++++ include/linux/context_tracking_work.h | 2 ++ 5 files changed, 28 insertions(+) diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h index 2c66687ce00e2..9d4f021b5a45b 100644 --- a/arch/x86/include/asm/context_tracking_work.h +++ b/arch/x86/include/asm/context_tracking_work.h @@ -3,6 +3,7 @@ #define _ASM_X86_CONTEXT_TRACKING_WORK_H #include +#include static __always_inline void arch_context_tracking_work(int work) { @@ -10,6 +11,9 @@ static __always_inline void arch_context_tracking_work(int work) case CONTEXT_WORK_SYNC: sync_core(); break; + case CONTEXT_WORK_TLBI: + __flush_tlb_all_noinstr(); + break; } } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 80450e1d5385a..323b971987af7 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -17,6 +17,7 @@ DECLARE_PER_CPU(u64, tlbstate_untag_mask); void __flush_tlb_all(void); +void noinstr __flush_tlb_all_noinstr(void); #define TLB_FLUSH_ALL -1UL #define TLB_GENERATION_INVALID 0 diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 267acf27480af..631df9189ded4 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1237,6 +1237,23 @@ void __flush_tlb_all(void) } EXPORT_SYMBOL_GPL(__flush_tlb_all); +void noinstr __flush_tlb_all_noinstr(void) +{ + /* + * This is for invocation in early entry code that cannot be + * instrumented. A RMW to CR4 works for most cases, but relies on + * being able to flip either of the PGE or PCIDE bits. Flipping CR4.PCID + * would require also resetting CR3.PCID, so just try with CR4.PGE, else + * do the CR3 write. + * + * TODO: paravirt + */ + if (cpu_feature_enabled(X86_FEATURE_PGE)) + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4)); + else + flush_tlb_local(); +} + void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info *info; diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index 292a0b7c06948..3571c62cbb9cd 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -62,6 +62,10 @@ enum ctx_state { #define RCU_DYNTICKS_END (CT_STATE_SIZE - 1) #define RCU_DYNTICKS_IDX BIT(RCU_DYNTICKS_START) +/* + * When CONFIG_CONTEXT_TRACKING_WORK=n, _END is 1 behind _START, which makes + * the CONTEXT_WORK size computation below 0, which is what we want! + */ #define CONTEXT_WORK_START (CONTEXT_STATE_END + 1) #define CONTEXT_WORK_END (RCU_DYNTICKS_START - 1) diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h index 13fc97b395030..47d5ced39a43a 100644 --- a/include/linux/context_tracking_work.h +++ b/include/linux/context_tracking_work.h @@ -6,11 +6,13 @@ enum { CONTEXT_WORK_SYNC_OFFSET, + CONTEXT_WORK_TLBI_OFFSET, CONTEXT_WORK_MAX_OFFSET }; enum ct_work { CONTEXT_WORK_SYNC = BIT(CONTEXT_WORK_SYNC_OFFSET), + CONTEXT_WORK_TLBI = BIT(CONTEXT_WORK_TLBI_OFFSET), CONTEXT_WORK_MAX = BIT(CONTEXT_WORK_MAX_OFFSET) }; From patchwork Thu Jul 20 16:30:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13320804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D1C1EB64DA for ; Thu, 20 Jul 2023 16:34:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF9B1280142; Thu, 20 Jul 2023 12:34:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAA2028004C; Thu, 20 Jul 2023 12:34:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D23F8280142; Thu, 20 Jul 2023 12:34:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BF2A828004C for ; Thu, 20 Jul 2023 12:34:48 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 80C3E1401BC for ; Thu, 20 Jul 2023 16:34:48 +0000 (UTC) X-FDA: 81032539056.18.EEBE333 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 7B04B140009 for ; Thu, 20 Jul 2023 16:34:46 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UjbraSCb; spf=pass (imf09.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689870886; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=amrP88Jqmb7M02bSDE9mFh1MFKi+V3yBx6dizqUgPII=; b=nApdbU1CwyKd8amFxN50Gri6pZE8tNoCTISBCO+PVEj0H54gk4KDUQgbav61gBpCHEWQns xCMhEWvWr15aazjuDEInU+gRuWKFtj2zF5CFK+waQAy/Un3Dvr1kB0WZDZ09k1yUpGxXCL vzXuHO+/et+biK2vcQD4paKbXXg9Wck= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689870886; a=rsa-sha256; cv=none; b=ZFEUSaEVsOPQOEFh1jPmLkPXLm/O+QADgRpeL0lpcU2BLCRbZ1B859IJTBdXPc5WfX2Jkx Ro/SvheVKw9AndDYVp30v8YvGQkSkKspUMRWcfHYwIoQCHh/HROFzV6hS8fVjUImSfwCku bjYvYLxHV3tiapH0Z5/gNYfiiX4i64c= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UjbraSCb; spf=pass (imf09.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689870885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=amrP88Jqmb7M02bSDE9mFh1MFKi+V3yBx6dizqUgPII=; b=UjbraSCbDOty3zOeVEO4DScvsOpidest5WvzhUmstwfJexz/Z0FqW3HAyJ4Akgjd/4pwzH JnK8/MDfl+oi17pWZfwT2xLAVx7FpeS7GHra6ExpUe8PIs5r8hy+QV2MOQcdsrDZS98x6U uriK15V4SZISg6QK2S61Fj5C51zSqx0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-613-tiaKu1LnPgWKO7yNg3Rgug-1; Thu, 20 Jul 2023 12:34:43 -0400 X-MC-Unique: tiaKu1LnPgWKO7yNg3Rgug-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF946800B35; Thu, 20 Jul 2023 16:34:40 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.48]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C55AF40C206F; Thu, 20 Jul 2023 16:34:32 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, rcu@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Jason Baron , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH v2 20/20] x86/mm, mm/vmalloc: Defer flush_tlb_kernel_range() targeting NOHZ_FULL CPUs Date: Thu, 20 Jul 2023 17:30:56 +0100 Message-Id: <20230720163056.2564824-21-vschneid@redhat.com> In-Reply-To: <20230720163056.2564824-1-vschneid@redhat.com> References: <20230720163056.2564824-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 7B04B140009 X-Rspam-User: X-Stat-Signature: gh1jato1p35tb737pzxi85k87uua1p1g X-Rspamd-Server: rspam03 X-HE-Tag: 1689870886-333517 X-HE-Meta: U2FsdGVkX1/1jz1GM2wGlUVW9mQ3YbyNxfMbNEb4L6zo7GTOkCN0gbViNBXKRb6tPaey5XSfQJiIYnLWTNUSXifi8HTWlIDfflRLB9nvdLhud/zmiwAu55CRt324hNxPQyqZlozii9A3ZSvoWV5AGnwpSsWKsIt9Att3IANQdPQXHXAzXPG/Lk5a+R4o+QregfZoLpzg10blthk7NQ/6VZJByJflTHO5+n5Sw4rF5OOsHIv1cnA1ae3doKjGnus8HcaWF6OJavxR/01z7vVK7PGfZ3BK1Icz61Dk0IlI0L0tLqa0v1Zv3WlrCivs7ZOpxcbNq+RiiTaH52jgCDXnrQK4LdPLdRbEHl9T9AzJuogYxK2M48VBeCXTzqs8o+QDBU2Zr8zFvW4ogiBg/mwTm3Fo9ePfTzGJInr3whP+b89XEo6egEy0LHuKnVXPsT724fwCsr6jvUdVjfABddOnoRZjns0zD5honq1TnbAZq5zw2SrO9tu5SP4QzaBNQdNIywcN+p+DqmnYxUyXxNjfO8oTNM+B5w5XSgwN9lOg0di9k2kShhcZkGGRk3nsbhl8pOn17yxaVn2tNfKzkSpFtQTJKseVCmmmUh6vXN/87e9lgZAiDsPAOmHa+aPBWAugOlrks7qTWbnbullPlVhLz9/vMT/aKc7I3w9tX4hyCM9bHplGpURyoet/v1WPIT8vbad3Wx9hQyLOcvklJ4Xl5dBPxAOHKs5fWcy6LFVIenkbdbSfLo5YiqAd20k7X8NxEf1C/qKzcCb9oUmnQpOmCzDOfiUbTZYG8VSsbQ2bRa5EeEYlN4i+ToJJUQqdqMFVvuAFTI4h4HeCn6wTaDtNMRo8W5f3dd+NjeeE5f1urAPGREXNbwegO7ogcxXhfpE2fG3zjerzZ2dbCr9c4cyk/aHmnXbiSpaekKjfMbbHM88U1tnjECiLMvj3llzvoj1RyciRWg9TrFf8RBWi1H0 grqhS7FU MijtFplihXcI0t6+6h9n7oDFYZbbA+renlEaS6vRsstRpl3qZEl5e+JmJnYhUyo2vwDszWbVYGHEJKXcF8JIswAbiQFvTxRAh8SGv+YwwZ1LqjI34xalAvIMLuUrQeFoJOZWO6LSmqRqXspDldtc9qg0O9YoLzn/SccDmLHiv71qdBSbVEEtwqRZ28I/djH3QmJI31/HZSARca2pZIR7N/ScI8cUkMkli2Sn+XivDHpaboHNfFxiIWT19j7iCvLlWEwskTrUWQGc/i4v3G6yDOG4PzKcNQ9EkewM4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vunmap()'s issued from housekeeping CPUs are a relatively common source of interference for isolated NOHZ_FULL CPUs, as they are hit by the flush_tlb_kernel_range() IPIs. Given that CPUs executing in userspace do not access data in the vmalloc range, these IPIs could be deferred until their next kernel entry. This does require a guarantee that nothing in the vmalloc range can be accessed in early entry code. vmalloc'd kernel stacks (VMAP_STACK) are AFAICT a safe exception, as a task running in userspace needs to enter kernelspace to execute do_exit() before its stack can be vfree'd. XXX: Validation that nothing in the vmalloc range is accessed in .noinstr or somesuch? Blindly deferring any and all flush of the kernel mappings is a risky move, so introduce a variant of flush_tlb_kernel_range() that explicitly allows deferral. Use it for vunmap flushes. Note that while flush_tlb_kernel_range() may end up issuing a full flush (including user mappings), this only happens when reaching a invalidation range threshold where it is cheaper to do a full flush than to individually invalidate each page in the range via INVLPG. IOW, it doesn't *require* invalidating user mappings, and thus remains safe to defer until a later kernel entry. Signed-off-by: Valentin Schneider --- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/mm/tlb.c | 23 ++++++++++++++++++++--- mm/vmalloc.c | 19 ++++++++++++++----- 3 files changed, 35 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 323b971987af7..0b9b1f040c476 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -248,6 +248,7 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); +extern void flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 631df9189ded4..bb18b35e61b4a 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -1045,6 +1046,11 @@ static void do_flush_tlb_all(void *info) __flush_tlb_all(); } +static bool do_kernel_flush_defer_cond(int cpu, void *info) +{ + return !ct_set_cpu_work(cpu, CONTEXT_WORK_TLBI); +} + void flush_tlb_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); @@ -1061,12 +1067,13 @@ static void do_kernel_range_flush(void *info) flush_tlb_one_kernel(addr); } -void flush_tlb_kernel_range(unsigned long start, unsigned long end) +static inline void +__flush_tlb_kernel_range(smp_cond_func_t cond_func, unsigned long start, unsigned long end) { /* Balance as user space task's flush, a bit conservative */ if (end == TLB_FLUSH_ALL || (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) { - on_each_cpu(do_flush_tlb_all, NULL, 1); + on_each_cpu_cond(cond_func, do_flush_tlb_all, NULL, 1); } else { struct flush_tlb_info *info; @@ -1074,13 +1081,23 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) info = get_flush_tlb_info(NULL, start, end, 0, false, TLB_GENERATION_INVALID); - on_each_cpu(do_kernel_range_flush, info, 1); + on_each_cpu_cond(cond_func, do_kernel_range_flush, info, 1); put_flush_tlb_info(); preempt_enable(); } } +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + __flush_tlb_kernel_range(NULL, start, end); +} + +void flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end) +{ + __flush_tlb_kernel_range(do_kernel_flush_defer_cond, start, end); +} + /* * This can be used from process context to figure out what the value of * CR3 is without needing to do a (slow) __read_cr3(). diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 93cf99aba335b..e08b6c7d22fb6 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -439,6 +439,15 @@ void vunmap_range_noflush(unsigned long start, unsigned long end) __vunmap_range_noflush(start, end); } +#ifdef CONFIG_CONTEXT_TRACKING_WORK +void __weak flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end) +{ + flush_tlb_kernel_range(start, end); +} +#else +#define flush_tlb_kernel_range_deferrable(start, end) flush_tlb_kernel_range(start, end) +#endif + /** * vunmap_range - unmap kernel virtual addresses * @addr: start of the VM area to unmap @@ -452,7 +461,7 @@ void vunmap_range(unsigned long addr, unsigned long end) { flush_cache_vunmap(addr, end); vunmap_range_noflush(addr, end); - flush_tlb_kernel_range(addr, end); + flush_tlb_kernel_range_deferrable(addr, end); } static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, @@ -1746,7 +1755,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) list_last_entry(&local_purge_list, struct vmap_area, list)->va_end); - flush_tlb_kernel_range(start, end); + flush_tlb_kernel_range_deferrable(start, end); resched_threshold = lazy_max_pages() << 1; spin_lock(&free_vmap_area_lock); @@ -1849,7 +1858,7 @@ static void free_unmap_vmap_area(struct vmap_area *va) flush_cache_vunmap(va->va_start, va->va_end); vunmap_range_noflush(va->va_start, va->va_end); if (debug_pagealloc_enabled_static()) - flush_tlb_kernel_range(va->va_start, va->va_end); + flush_tlb_kernel_range_deferrable(va->va_start, va->va_end); free_vmap_area_noflush(va); } @@ -2239,7 +2248,7 @@ static void vb_free(unsigned long addr, unsigned long size) vunmap_range_noflush(addr, addr + size); if (debug_pagealloc_enabled_static()) - flush_tlb_kernel_range(addr, addr + size); + flush_tlb_kernel_range_deferrable(addr, addr + size); spin_lock(&vb->lock); @@ -2304,7 +2313,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) free_purged_blocks(&purge_list); if (!__purge_vmap_area_lazy(start, end) && flush) - flush_tlb_kernel_range(start, end); + flush_tlb_kernel_range_deferrable(start, end); mutex_unlock(&vmap_purge_lock); }