From patchwork Thu May 28 23:52:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577271 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E97C90 for ; Thu, 28 May 2020 23:52:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E75712075F for ; Thu, 28 May 2020 23:52:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PsD5NVCh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E75712075F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0D38B800B6; Thu, 28 May 2020 19:52:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0AB6780010; Thu, 28 May 2020 19:52:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F029B800B6; Thu, 28 May 2020 19:52:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0209.hostedemail.com [216.40.44.209]) by kanga.kvack.org (Postfix) with ESMTP id D647D80010 for ; Thu, 28 May 2020 19:52:54 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 95F952C84 for ; Thu, 28 May 2020 23:52:54 +0000 (UTC) X-FDA: 76867780668.26.ants77_8db7aca2ba853 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 71FC81804B660 for ; Thu, 28 May 2020 23:52:54 +0000 (UTC) X-Spam-Summary: 2,0,0,11eff60e22d152d4,d41d8cd98f00b204,31e7qxg0kck8pmtagphbjhhtcvddvat.rdbaxcjm-bbzkprz.dgv@flex--axelrasmussen.bounces.google.com,,RULES_HIT:41:152:327:355:379:421:541:800:960:966:968:973:988:989:1260:1277:1313:1314:1345:1431:1437:1516:1518:1593:1594:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2380:2393:2553:2559:2562:2693:2731:2899:2903:2907:3138:3139:3140:3141:3142:3152:3165:3608:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:4605:5007:6261:6653:6737:6742:6743:7875:7903:8603:8660:9969:10004:11026:11233:11473:11657:11658:11914:12043:12048:12291:12294:12296:12297:12438:12555:12683:12895:12986:13141:13148:13230:14096:14097:14394:14659:21080:21433:21444:21450:21451:21627:21795:21939:21972:21990:30012:30034:30054:30070:30079:30090,0,RBL:209.85.219.201:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Cust om_rules X-HE-Tag: ants77_8db7aca2ba853 X-Filterd-Recvd-Size: 21249 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:52:53 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id v194so945960ybv.5 for ; Thu, 28 May 2020 16:52:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=5ipO+I0wmX1R5+u3O2fbii3lYImNO/YXeMipTGcgmLw=; b=PsD5NVChLrErIMDxT2LexeTmFYPvJhY2+NLr+Mp71MljLnCvnQnyMr8J7ozdoMsuus ldvOXwRyJXcq5nLR1DGF4jh5Kqm0wzSDYbi+QMkROis/lN9+WSI9h3p3u7Iv78hK+8A5 dDTVzXZKG6EnY6ccYOkP9SGWgIhJQ4ZJ60awlLLtOvDFb2Kdp+zyxcMZXWl58W3fzpvQ 1Eoi4FMLLYGe3SBrmG0dOYgwOneDBZxm4CdKmcL4/7lcweCbkZoAHfawSFvfBYuPdb+b AiPL26fA+nhpyH2iQaeL73XJ/x9Ok12JmqxhZ/nke6QsH4FbhohJS1Jfd2mPsatGkUKq U+pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=5ipO+I0wmX1R5+u3O2fbii3lYImNO/YXeMipTGcgmLw=; b=Rtu91ESv6yyu+++Pbju0QQ0eMEThsbXBYiNpya49GhYoCb9ywxjlT2FhPKlYN4oJl9 1m5GuJYTcdaWpoIXcqEL4/M/x8bH0iuqDNTsv7tYsw73gmZpZDBroNIt3w88EAbePH4U mqdgFwCABxbz9DD7AvD5YDWEkq7Zban6ds74p/rpX+foiwoodlGpAPzeE0DrbCzHf9AC b+LZcxxHevPw0Q/PHMahMxEiBi/acgtqNLCpdJLskjdlSaSMNVzWjOm5CV4OJo/qwAcb gIVzCEd9Ol5rniX1GvLn6gfug73cD3oBQl61x3riPFexrSuBWxsh+rQoLQliS9KifiRU k/JQ== X-Gm-Message-State: AOAM531ow0ISuFAi6ZP1B1tTPyf42ecxp6erdL3HyFpIZnjEU958QpJH 12SFuKq+LALhMyFSeDDUA+e48UoeGXPBr6C4N2yT X-Google-Smtp-Source: ABdhPJxedEQ+Tl+cL2CfSt/L6YsIfZL5QG9qge1QUHoR4ZDwAUeF8788myEPgWNuqb26EYrbag7L9HlfziwDiH2yPDGQ X-Received: by 2002:a25:7386:: with SMTP id o128mr9164352ybc.266.1590709972962; Thu, 28 May 2020 16:52:52 -0700 (PDT) Date: Thu, 28 May 2020 16:52:44 -0700 Message-Id: <20200528235244.74960-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 1/7] histogram: add struct histogram From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: 71FC81804B660 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: struct histogram provides a histogram that counts u64 samples in arbitrary user-configurable buckets, optimized for efficient concurrent recording. This is a squashed, refactored, and modified version of a previously-internal implementation. Thanks to the following individuals for portions of the implementation: Jamie Liu - Original implementation Yu Zhao - Code cleanups + simplification Signed-off-by: Axel Rasmussen --- include/linux/histogram.h | 270 ++++++++++++++++++++++++++++++++++++++ lib/Kconfig | 3 + lib/Makefile | 2 + lib/histogram.c | 157 ++++++++++++++++++++++ 4 files changed, 432 insertions(+) create mode 100644 include/linux/histogram.h create mode 100644 lib/histogram.c diff --git a/include/linux/histogram.h b/include/linux/histogram.h new file mode 100644 index 000000000000..137930ca933f --- /dev/null +++ b/include/linux/histogram.h @@ -0,0 +1,270 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_HISTOGRAM_H +#define _LINUX_HISTOGRAM_H + +#include +#include +#include +#include + +/* + * Histograms for counting events represented by u64s. + * + * Each histogram may have any number of thresholds. A histogram with + * nr_thresholds thresholds has nr_thresholds buckets, where bucket[i] counts + * samples in the half-open interval ( thresholds[i-1], thresholds[i] ]. + * (thresholds[-1] is implicitly defined as 0, and thresholds[nr_thresholds-1] + * must be 2**64-1.) Each histogram's thresholds are immutable after creation. + * Each bucket counts up to a u64 worth of events. + * + * Example usage: + * + * u64 thresholds[] = {1, 2, 5, 10, 20, 50, 100, ~0}; + * u64 buckets[ARRAY_SIZE(thresholds)]; + * struct histogram *hist = histogram_alloc(thresholds, ARRAY_SIZE(thresholds)); + * if (IS_ERR(hist)) {...} + * histogram_record(hist, 0, 1); + * histogram_record(hist, 8, 2); + * histogram_record(hist, 20, 3); + * histogram_record(hist, 110, 4); + * histogram_read_buckets(hist, buckets); + * // buckets == {1, 0, 0, 2, 3, 0, 0, 4} + * histogram_free(hist); + * + * For convenience, a struct histogram_rcu type is also provided which wraps an + * RCU-sched protected pointer to a struct histogram, allowing thresholds to be + * modified even while other threads may be recording to the histogram. + * Functions that operate on a histogram_rcu are distinguished by an _rcu + * suffix. + * + * Example usage: + * + * u64 thresholds_1[] = {1, 2, 5, 10, 20, 50, 100, ~0}; + * u64 thresholds_2[] = {100, 200, 500, 1000, ~0}; + * u64 buckets[ARRAY_SIZE(thresholds_1)]; + * struct histogram_rcu hrcu; + * if (histogram_init_rcu(&hrcu, thresholds_1, ARRAY_SIZE(thresholds_1))) {...} + * histogram_record_rcu(&hrcu, 0, 1); + * histogram_record_rcu(&hrcu, 8, 2); + * histogram_record_rcu(&hrcu, 20, 3); + * histogram_record_rcu(&hrcu, 110, 4); + * if (histogram_read_rcu(&hrcu, buckets, NULL, + * ARRAY_SIZE(thresholds_1)) < 0) {...} + * // buckets == {1, 0, 0, 2, 3, 0, 0, 4} + * if (histogram_set_thresholds_rcu(&hrcu, thresholds_2, + * ARRAY_SIZE(thresholds_2)) {...} + * if (histogram_read_rcu(&hrcu, buckets, NULL, + * ARRAY_SIZE(thresholds_2)) < 0) {...} + * // buckets == {0, 0, 0, 0, 0} + * histogram_record_rcu(&hrcu, 50, 1); + * histogram_record_rcu(&hrcu, 150, 2); + * histogram_record_rcu(&hrcu, 5000, 3); + * if (histogram_read_rcu(&hrcu, buckets, NULL, + * ARRAY_SIZE(thresholds_2)) < 0) {...} + * // buckets == {1, 2, 0, 0, 3} + * histogram_destroy_rcu(&hrcu); + */ + +struct histogram { + struct rcu_head rcu; + u64 __percpu *buckets; + size_t nr_thresholds; + u64 thresholds[0]; /* flexible array member */ +}; + +struct histogram_rcu { + struct histogram __rcu *hist; +}; + +/** + * histogram_record() - record samples in histogram + * @hist: histogram + * @val: sample value + * @count: number of samples + * + * histogram_record() does not require synchronization with concurrent + * histogram_record() or histogram_read_buckets(). + * + * histogram_record() may be called from tracing code. + */ +void histogram_record(struct histogram *hist, u64 val, u64 count); + +/** + * histogram_read_buckets() - read histogram buckets + * @hist: histogram + * @buckets: array with space for at least hist->nr_thresholds elements + * + * Sets each element of @buckets to the value of the corresponding histogram + * bucket. + * + * histogram_read_buckets() does not require synchronization with concurrent + * histogram_record() or histogram_read_buckets(). + * + * histogram_read_buckets() does not block recording, so values are not read + * from all CPUs atomically. If this is a problem, the caller should stop + * recording first. + */ +void histogram_read_buckets(const struct histogram *hist, u64 *buckets); + +/** + * histogram_alloc() - create a histogram + * @thresholds: thresholds array + * @nr_thresholds: number of elements in @thresholds + * + * Histogram buckets are initialized to zero. @thresholds must be sorted in + * ascending order and must not contain any duplicates. @nr_thresholds must be + * >= 1. thresholds[nr_thresholds-1] must be ~(u64)0. + * + * histogram_alloc() makes a copy of @thresholds if successful, so ownership of + * @thresholds is unaffected by histogram_alloc(). + * + * Context: Performs allocation with GFP_KERNEL. + * + * Returns: Pointer to new histogram, or a PTR_ERR on failure. + */ +struct histogram *histogram_alloc(const u64 *thresholds, size_t nr_thresholds); + +/** + * histogram_free() - delete a histogram + * @hist: histogram + */ +void histogram_free(struct histogram *hist); + +/** + * histogram_record_rcu() - record samples in RCU-protected histogram + * @hrcu: histogram + * @val: sample value + * @count: number of samples + * + * histogram_record_rcu() does not require external synchronization, even with + * histogram_destroy_rcu(). Calling histogram_record_rcu() on a @hrcu that + * histogram_destroy_rcu() has been called on is a no-op. + * + * histogram_record_rcu() may be called from tracing code. + */ +static inline void histogram_record_rcu(struct histogram_rcu *hrcu, u64 val, + u64 count) +{ + struct histogram *hist; + + rcu_read_lock_sched_notrace(); + hist = rcu_dereference_sched(hrcu->hist); + if (likely(hist)) + histogram_record(hist, val, count); + rcu_read_unlock_sched_notrace(); +} + +/** + * histogram_read_rcu() - read RCU-protected histogram + * @hrcu: histogram + * @buckets: array with space for at least nr_thresholds elements + * @thresholds: array with space for at least nr_thresholds elements + * @nr_thresholds: array size (see above) + * + * If @buckets is not NULL, sets each element of @buckets to the value of the + * corresponding histogram bucket. + * + * If @thresholds is not NULL, sets each element of @thresholds to the + * corresponding histogram bucket threshold. + * + * On failure (for example, if nr_thresholds < hrcu->hist->nr_thresholds), + * neither @buckets nor @thresholds will be modified. + * + * histogram_read_rcu() does not require external synchronization, even with + * histogram_destroy_rcu(). Calling histogram_read_rcu() on a @hrcu that + * histogram_destroy_rcu() has been called on returns 0. + * + * histogram_read_rcu() does not block recording, so bucket values are not read + * from all CPUs atomically. If this is a problem, the caller should stop + * recording first. + * + * Returns: If successful, returns the actual number of thresholds stored in + * @thresholds; if nr_thresholds is too small, returns the negative of the + * minimum required nr_thresholds to succeed; if the histogram has been + * destroyed by histogram_destroy_rcu(), returns 0. (Note that if nr_thresholds + * is too small, it is not guaranteed that calling histogram_read_rcu() again + * with the returned value of nr_thresholds will succeed, because another + * thread could raise the number of thresholds again in the interim.) + */ +ssize_t histogram_read_rcu(const struct histogram_rcu *hrcu, u64 *buckets, + u64 *thresholds, size_t nr_thresholds); + +/** + * histogram_set_thresholds_rcu() - set RCU-protected histogram thresholds + * @hrcu: histogram + * @thresholds: thresholds array + * @nr_thresholds: number of elements in @thresholds + * + * The semantics that apply to @thresholds are the same as for histogram_alloc. + * + * If successful, all buckets are atomically reset to zero. + * + * The caller must synchronize between concurrent calls to + * histogram_set_thresholds_rcu(), histogram_init_rcu(), and + * histogram_destroy_rcu(). + * + * Context: Performs allocation with GFP_KERNEL. + * + * Returns: Zero on success, or a negative error code on failure. + */ +int histogram_set_thresholds_rcu(struct histogram_rcu *hrcu, + const u64 *thresholds, size_t nr_thresholds); + +/** + * histogram_init_rcu() - initialize RCU-protected histogram + * @hrcu: histogram + * @thresholds: thresholds array + * @nr_thresholds: number of elements in @thresholds. + * + * Each struct histogram_rcu must be initialized by histogram_init_rcu() at + * least once before it is valid to call any other functions on it. A struct + * histogram_rcu that has been previously initialized cannot be initialized + * again, unless it has been subsequently destroyed by histogram_destroy_rcu(). + * + * The semantics that apply to @thresholds are the same as for + * histogram_alloc(), with one exception: @thresholds may be NULL iff + * @nr_thresholds is 0. In this case, @hrcu will behave as if it has already + * been destroyed (histogram_record_rcu() will no-op and histogram_read_rcu() + * will return 0). + * + * The caller must synchronize between concurrent calls to + * histogram_set_thresholds_rcu(), histogram_init_rcu(), and + * histogram_destroy_rcu(). + * + * Context: Performs allocation with GFP_KERNEL. + * + * Returns: Zero on success, or a negative error code on failure. + */ +static inline int histogram_init_rcu(struct histogram_rcu *hrcu, + const u64 *thresholds, + size_t nr_thresholds) +{ + RCU_INIT_POINTER(hrcu->hist, NULL); + if (!thresholds && !nr_thresholds) + return 0; + return histogram_set_thresholds_rcu(hrcu, thresholds, nr_thresholds); +} + +void histogram_destroy_rcu_cb(struct rcu_head *rcu); + +/** + * histogram_destroy_rcu() - destroy RCU-protected histogram + * @hrcu: histogram + * + * After histogram_destroy_rcu() has been called on a @hrcu, it is valid to call + * histogram_init_rcu() on it again. + * + * The caller must synchronize between concurrent calls to + * histogram_set_thresholds_rcu(), histogram_init_rcu(), and + * histogram_destroy_rcu(). + */ +static inline void histogram_destroy_rcu(struct histogram_rcu *hrcu) +{ + struct histogram *hist = rcu_dereference_raw(hrcu->hist); + + RCU_INIT_POINTER(hrcu->hist, NULL); + if (likely(hist)) + call_rcu(&hist->rcu, histogram_destroy_rcu_cb); +} + +#endif /* _LINUX_HISTOGRAM_H */ diff --git a/lib/Kconfig b/lib/Kconfig index 5d53f9609c25..4714bdfa343b 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -648,6 +648,9 @@ config OBJAGG config STRING_SELFTEST tristate "Test string functions" +config HISTOGRAM + bool + endmenu config GENERIC_IOREMAP diff --git a/lib/Makefile b/lib/Makefile index 685aee60de1d..f61b1c15d656 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -248,6 +248,8 @@ obj-$(CONFIG_ASN1) += asn1_decoder.o obj-$(CONFIG_FONT_SUPPORT) += fonts/ +obj-$(CONFIG_HISTOGRAM) += histogram.o + hostprogs := gen_crc32table hostprogs += gen_crc64table clean-files := crc32table.h diff --git a/lib/histogram.c b/lib/histogram.c new file mode 100644 index 000000000000..b68334275a46 --- /dev/null +++ b/lib/histogram.c @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +void histogram_record(struct histogram *hist, u64 val, u64 count) +{ + size_t i, lower, upper; + + /* Binary search invariant: correct bucket in [lower, upper] */ + lower = 0; + upper = hist->nr_thresholds - 1; + while (lower < upper) { + /* can't realistically overflow, nr_thresholds < 2**63 */ + i = (lower + upper) / 2; + if (val <= hist->thresholds[i]) + upper = i; + else + lower = i + 1; + } + i = upper; + /* _notrace because histogram_record may be called from tracing */ + preempt_disable_notrace(); + __this_cpu_add(hist->buckets[i], count); + preempt_enable_notrace(); +} +EXPORT_SYMBOL_GPL(histogram_record); + +void histogram_read_buckets(const struct histogram *hist, u64 *buckets) +{ + int cpu; + size_t nr_buckets; + size_t i; + + nr_buckets = hist->nr_thresholds; + memset(buckets, 0, nr_buckets * sizeof(u64)); + /* + * This must use for_each_possible_cpu rather than for_each_online_cpu + * to ensure we count records from any CPUs that have since been + * removed. + */ + for_each_possible_cpu(cpu) { + for (i = 0; i < nr_buckets; i++) + buckets[i] += per_cpu(hist->buckets[i], cpu); + } +} +EXPORT_SYMBOL_GPL(histogram_read_buckets); + +static int histogram_check_thresholds(const u64 *thresholds, + size_t nr_thresholds) +{ + unsigned int i; + + if (!nr_thresholds) + return -EINVAL; + if (!thresholds) + return -EFAULT; + for (i = 1; i < nr_thresholds; i++) + if (thresholds[i - 1] >= thresholds[i]) + return -EINVAL; + if (thresholds[nr_thresholds - 1] != ~0ULL) + return -EINVAL; + return 0; +} + +struct histogram *histogram_alloc(const u64 *thresholds, size_t nr_thresholds) +{ + struct histogram *hist; + size_t hist_size; + int ret; + + ret = histogram_check_thresholds(thresholds, nr_thresholds); + if (ret) + return ERR_PTR(ret); + hist_size = sizeof(struct histogram) + nr_thresholds * sizeof(u64); + hist = kmalloc(ALIGN(hist_size, cache_line_size()), GFP_KERNEL); + if (!hist) + return ERR_PTR(-ENOMEM); + hist->buckets = __alloc_percpu(nr_thresholds * sizeof(*hist->buckets), + __alignof__(*hist->buckets)); + if (!hist->buckets) { + kfree(hist); + return ERR_PTR(-ENOMEM); + } + hist->nr_thresholds = nr_thresholds; + memcpy(hist->thresholds, thresholds, nr_thresholds * sizeof(u64)); + return hist; +} +EXPORT_SYMBOL_GPL(histogram_alloc); + +void histogram_free(struct histogram *hist) +{ + if (!hist) + return; + free_percpu(hist->buckets); + kfree(hist); +} +EXPORT_SYMBOL_GPL(histogram_free); + +ssize_t histogram_read_rcu(const struct histogram_rcu *hrcu, u64 *buckets, + u64 *thresholds, size_t nr_thresholds) +{ + const struct histogram *hist; + ssize_t ret = 0; + + rcu_read_lock_sched(); + hist = rcu_dereference_sched(hrcu->hist); + if (!hist) { + ret = 0; + goto out; + } + if (nr_thresholds < hist->nr_thresholds) { + ret = -hist->nr_thresholds; + goto out; + } + if (buckets) + histogram_read_buckets(hist, buckets); + if (thresholds) + memcpy(thresholds, hist->thresholds, + hist->nr_thresholds * sizeof(u64)); + ret = hist->nr_thresholds; +out: + rcu_read_unlock_sched(); + return ret; +} +EXPORT_SYMBOL_GPL(histogram_read_rcu); + +int histogram_set_thresholds_rcu(struct histogram_rcu *hrcu, + const u64 *thresholds, size_t nr_thresholds) +{ + struct histogram *old_hist = rcu_dereference_raw(hrcu->hist); + struct histogram *new_hist = histogram_alloc(thresholds, nr_thresholds); + + if (IS_ERR(new_hist)) + return PTR_ERR(new_hist); + rcu_assign_pointer(hrcu->hist, new_hist); + if (old_hist) { + synchronize_rcu(); + histogram_free(old_hist); + } + return 0; +} +EXPORT_SYMBOL_GPL(histogram_set_thresholds_rcu); + +void histogram_destroy_rcu_cb(struct rcu_head *rcu) +{ + histogram_free(container_of(rcu, struct histogram, rcu)); +} +EXPORT_SYMBOL_GPL(histogram_destroy_rcu_cb); From patchwork Thu May 28 23:52:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577273 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 592311391 for ; Thu, 28 May 2020 23:52:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 17E0E2078C for ; Thu, 28 May 2020 23:52:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="IRNgeI7k" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 17E0E2078C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 23363800B7; Thu, 28 May 2020 19:52:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 20B6080010; Thu, 28 May 2020 19:52:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12087800B7; Thu, 28 May 2020 19:52:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id EDC1380010 for ; Thu, 28 May 2020 19:52:57 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B0AF92C84 for ; Thu, 28 May 2020 23:52:57 +0000 (UTC) X-FDA: 76867780794.20.copy78_8e350d6e68520 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 96055180C07AB for ; Thu, 28 May 2020 23:52:57 +0000 (UTC) X-Spam-Summary: 2,0,0,301d4a5cc5795433,d41d8cd98f00b204,32e7qxg0kclmtqxektlfnllxgzhhzex.vhfebgnq-ffdotvd.hkz@flex--axelrasmussen.bounces.google.com,,RULES_HIT:41:152:355:379:541:800:960:966:973:988:989:1260:1277:1313:1314:1345:1431:1437:1516:1518:1535:1543:1593:1594:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2899:3138:3139:3140:3141:3142:3152:3165:3355:3608:3865:3866:3867:3868:3870:3871:4119:4250:4321:4385:4605:5007:6261:6653:6737:6742:6743:7875:7903:9969:10004:10400:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:12986:14181:14394:14659:14721:14877:21080:21433:21444:21451:21627:21990:30012:30034:30054:30070,0,RBL:209.85.219.201:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: copy78_8e350d6e68520 X-Filterd-Recvd-Size: 8875 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:52:57 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id f130so927140yba.9 for ; Thu, 28 May 2020 16:52:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=uXM6Fov/4A7+5lOy4XDu2Yode4WUxdldKubf/JJk+Ys=; b=IRNgeI7k5mpd9pI0t6QJQPWME/3vAvlzp62U1uKbtD54d8xsgeYrdDPC5Ao4i/FZ1b ItsxLbIf0twoil5ukifuA0+3Wj9dVqePe3ZJ6fN0HMhlIIgnvhXRU4ditFe1xW22kExt ud6Dka7HnVgfy/jaXPCtVjQAK+QyUJ3yAGJmaKkJm2W5UiDptVkATKVJoSF/PJV+dNHs B2nffPSoYsra5frvS6ZZBHWgQTAtNuyzg4sYYGRT67FHeOyaLt1JKXaCdEaUmEWafhsI 47uFQn5Rx3xApoG0HR3XrZ7z/pDViWWwYa3Y3DlMlI5vvOssqt2ZljAByC8H5XKNyOcV SynA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=uXM6Fov/4A7+5lOy4XDu2Yode4WUxdldKubf/JJk+Ys=; b=D078f52tT8vomiijcTmmDvbbx7p1MiDrTG85JJUDLGKoDyTIWDkbsD9gIy1hGTRVic EGAK12lCfH9PbbJfKWcycnSfN9YU0rNI6pSpZSk2wP2npwIYFpe/MyjZwgKY8ZyNMVVe 7c51SAXmP6WT1mh2M3RPI9B6/jxUiUyiP5pgD9UcvqvEe1tXX/yna1+w5JcOPuY2Grxc wIL6zamsrs53f3ybiXVqFbMw4aoDQsifbHrrfFWkVseoy5m+h89J8Xy+OvGVZ4nhQJuM 5hxtEbV+/toNl6IiZ3GrmihBQ+aQ1LuyRdp6YnjeOKPryGwBkZqe05aUsjt0IgQEwYP7 SXPQ== X-Gm-Message-State: AOAM532NZwThWv26fFeaLWQr0VlKepavkyNTO048lW0xoAObgwV1H7v0 0QIXgBozcM5q1fiwBM6lKOKV12Fv7ctvUqbbszW6 X-Google-Smtp-Source: ABdhPJy/rfhM6KHn/G1HvFjIKBvVeSeuXQmfxsW2OduPIrlSFrsDtZtuxGzg0YGt33xYLF0SlmHehM1GOL3M3O2rtawO X-Received: by 2002:a25:8b02:: with SMTP id i2mr9363896ybl.283.1590709976338; Thu, 28 May 2020 16:52:56 -0700 (PDT) Date: Thu, 28 May 2020 16:52:53 -0700 Message-Id: <20200528235253.75570-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 2/7] histogram: add helper function to expose histograms to userspace From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: 96055180C07AB X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This change introduces the histogram_print_buckets_rcu(). This API is used for implementing e.g. procfs or sysfs files exposing histograms. print_buckets can be combined with e.g. seq_file to provide a read interface for such a file. This is a squashed, refactored, and modified version of a previously-internal implementation. Thanks to the following individuals for portions of the implementation: Junaid Shahid - Original implementation Yu Zhao - Simplification Signed-off-by: Axel Rasmussen --- include/linux/histogram.h | 23 ++++++++++++++++ lib/histogram.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/include/linux/histogram.h b/include/linux/histogram.h index 137930ca933f..7a1dc33fff5e 100644 --- a/include/linux/histogram.h +++ b/include/linux/histogram.h @@ -65,6 +65,12 @@ * histogram_destroy_rcu(&hrcu); */ +/* + * Max number of digits in decimal needed to represent U64_MAX threshold, + * plus one space. + */ +#define HISTO_MAX_CHARS_PER_THRESHOLD (20 + 1) + struct histogram { struct rcu_head rcu; u64 __percpu *buckets; @@ -189,6 +195,23 @@ static inline void histogram_record_rcu(struct histogram_rcu *hrcu, u64 val, ssize_t histogram_read_rcu(const struct histogram_rcu *hrcu, u64 *buckets, u64 *thresholds, size_t nr_thresholds); +/** + * histogram_print_buckets_rcu() - helper function to print histogram buckets + * @hrcu: histogram + * @buffer: output buffer to fill + * @len: length of the output buffer + * + * Reads buckets by calling histogram_read_rcu(), then fills the output buffer. + * + * Context: Performs allocation with GFP_ATOMIC. + * + * Returns: The number of characters written to @buffer, or a negative error + * code on failure. If the buffer isn't large enough to contain the output, + * -EINVAL is returned. + */ +int histogram_print_buckets_rcu(struct histogram_rcu *hrcu, char *buffer, + int len); + /** * histogram_set_thresholds_rcu() - set RCU-protected histogram thresholds * @hrcu: histogram diff --git a/lib/histogram.c b/lib/histogram.c index b68334275a46..1a23590fc6e9 100644 --- a/lib/histogram.c +++ b/lib/histogram.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include #include @@ -133,6 +134,60 @@ ssize_t histogram_read_rcu(const struct histogram_rcu *hrcu, u64 *buckets, } EXPORT_SYMBOL_GPL(histogram_read_rcu); +int histogram_print_buckets_rcu(struct histogram_rcu *hrcu, char *buffer, + int len) +{ + const struct histogram *hist; + u64 *buckets = NULL; + u64 lower = 0; + size_t nr_buckets; + size_t i; + int ret; + int remaining = len; + + rcu_read_lock_sched(); + hist = rcu_dereference_sched(hrcu->hist); + + nr_buckets = hist->nr_thresholds; + if (!nr_buckets) { + ret = -EINVAL; + goto out; + } + + buckets = kmalloc_array(nr_buckets, sizeof(*hist->buckets), GFP_ATOMIC); + if (!buckets) { + ret = -ENOMEM; + goto out; + } + + histogram_read_buckets(hist, buckets); + + for (i = 0; i < nr_buckets; i++) { + if (i == nr_buckets - 1) + ret = snprintf(buffer, remaining, "%llu-inf %llu\n", + lower, buckets[i]); + else + ret = snprintf(buffer, remaining, "%llu-%llu %llu\n", + lower, hist->thresholds[i], buckets[i]); + if (ret >= remaining) { + ret = -EINVAL; + goto out; + } + + buffer += ret; + remaining -= ret; + + lower = hist->thresholds[i] + 1; + } + + ret = len - remaining; +out: + rcu_read_unlock_sched(); + kfree(buckets); + return ret; +} +EXPORT_SYMBOL_GPL(histogram_print_buckets_rcu); + int histogram_set_thresholds_rcu(struct histogram_rcu *hrcu, const u64 *thresholds, size_t nr_thresholds) { From patchwork Thu May 28 23:52:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577275 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 987DC90 for ; Thu, 28 May 2020 23:53:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 649FF206E2 for ; Thu, 28 May 2020 23:53:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mBZa7X8r" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 649FF206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 82919800B8; Thu, 28 May 2020 19:53:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8019B80010; Thu, 28 May 2020 19:53:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71311800B8; Thu, 28 May 2020 19:53:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0188.hostedemail.com [216.40.44.188]) by kanga.kvack.org (Postfix) with ESMTP id 592D380010 for ; Thu, 28 May 2020 19:53:01 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1D85B181AC217 for ; Thu, 28 May 2020 23:53:01 +0000 (UTC) X-FDA: 76867780962.28.top62_8eb930efaae1c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 0409C6D73 for ; Thu, 28 May 2020 23:53:00 +0000 (UTC) X-Spam-Summary: 2,0,0,0f8a9feefd26fe24,d41d8cd98f00b204,3207qxg0kclywtahnwoiqooajckkcha.ykihejqt-iigrwyg.knc@flex--axelrasmussen.bounces.google.com,,RULES_HIT:41:152:355:379:541:800:960:973:988:989:1260:1277:1313:1314:1345:1431:1437:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2393:2559:2562:2693:2740:2912:3138:3139:3140:3141:3142:3152:3165:3353:3865:3866:3867:3868:3871:4117:4605:5007:6261:6653:6737:6742:6743:9010:9969:10004:10400:11026:11473:11658:11914:12043:12048:12296:12297:12438:12555:12895:13161:13229:14096:14097:14181:14394:14659:14721:21080:21444:21627:30054:30091,0,RBL:209.85.219.202:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: top62_8eb930efaae1c X-Filterd-Recvd-Size: 6986 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:53:00 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id i190so947056ybg.6 for ; Thu, 28 May 2020 16:53:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=XGMqQwnndi2D0kIOd8fRs3e9yQLxoc/+AqptDy+c8dk=; b=mBZa7X8ryPIaCW9k2DuaI60mMcnTbinIX3ABmtpuRFmwvWVtcp0dGCNjx8sGHXRWyO 3g9qYhidRLUvNzSaK4hlHqwNLVqh3X6JGWATs3+ziTkBG7ASddX0k9vhYSzADPrfd+X6 PejSGyWQ/jLxcM/9HTwsMOXoSUUZnB7pPeR9FKO84BMWwAO4CxBAVgV1LwMd64g+Sm6c trSUK4DZnHNtP6yOWtZU0Su8Xjv0Mrogv+wPHCJqxQntRkq19cV/oWkoRu0RUEVzzv34 wiF0PKkwFVZ2nxm7TWSQlHEdZ0wESIUI89uplbsCYVvQ9BmLdQ1nG4ukoIijp6oOOdWC 0iwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=XGMqQwnndi2D0kIOd8fRs3e9yQLxoc/+AqptDy+c8dk=; b=pHWAirw3x8tQMAm26PEMmxSMaEAy+8ZyNk+jq1tvzQYq83ZFoJt05sKZe9VNv+VCum Xa7AvdLJm6ECVqtS/5Vnl2S61erwDseysfJ8hMF3ImKgYmZ22eH7IYWAku+1X9ijx1Wm 35NsxV8dIElzEjlS+DDORcpdTHE8tKfqc1QM1c6lscnxK7ZqvdCrAt+seYE06/5y6akc tMWqkLa1eNrbCv36eA9uVBulVzs5lkwo/9LY48+qUfOfeILvQsrS/UUIPHcO+Jr+D/y4 livKV80nTFQ9cK2hB3rFC64IrQnAMIl8Tgh2isu7VCl9HsqSt5lL+n8oy6x+XPC2td6v azFA== X-Gm-Message-State: AOAM531U4+X3VfzxHYCz3FQxNFUbzzmbyDuw7A2JxjLS8DOHeT/hBNEl F4mRSU4uzvocV5f+RFuiSbCi5GJtLkA+ArV/auQm X-Google-Smtp-Source: ABdhPJwIS7bK1T6mMrfjsRUY0LC6TcdIrHTI+NXlXI0/bAT2o0dyiWoh778AfYHV+Kog73RIZyRpi4/AnxvHZZnwrlML X-Received: by 2002:a25:338b:: with SMTP id z133mr9235332ybz.329.1590709979886; Thu, 28 May 2020 16:52:59 -0700 (PDT) Date: Thu, 28 May 2020 16:52:56 -0700 Message-Id: <20200528235256.76129-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 3/7] mmap_lock: add a histogram structure to struct mm_struct From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: 0409C6D73 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This histogram will be used to measure lock acquisition latency, when the acquisition is contended. Cases where we acquire the lock immediately with no contention are recorded as "0" (to avoid incurring the cost of really computing the time, while retaining the ability to get an accurate total lock acquisition count). Signed-off-by: Axel Rasmussen --- include/linux/mm_types.h | 11 +++++++++++ mm/Kconfig | 13 +++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d13b90399c16..efeb5db1959d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -376,6 +376,10 @@ struct core_state { struct completion startup; }; +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS +struct histogram_rcu; +#endif + struct kioctx_table; struct mm_struct { struct { @@ -437,6 +441,13 @@ struct mm_struct { * counters */ struct rw_semaphore mmap_lock; +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + /* Histogram of mmap_lock contended acquire delays. kmalloc'd + * so we can use a forward declaration, to avoid circular + * dependencies between headers. + */ + struct histogram_rcu *mmap_lock_contention; +#endif struct list_head mmlist; /* List of maybe swapped mm's. These * are globally strung together off diff --git a/mm/Kconfig b/mm/Kconfig index c1acc34c1c35..231862afc2ce 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -817,6 +817,19 @@ config DEVICE_PRIVATE config FRAME_VECTOR bool +config MMAP_LOCK_HISTOGRAMS + bool "mmap_lock histograms" + select HISTOGRAM + default n + + help + Record and report latency histograms describing mmap_lock + contention. Stats can be read from /proc//mmap_lock_contention. + True latency is only computed for *contended* acquires, to minimize + overhead. Uncontended acquires are recorded as having "0" latency. + + If unsure, say "n". + config ARCH_USES_HIGH_VMA_FLAGS bool config ARCH_HAS_PKEYS From patchwork Thu May 28 23:53:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577277 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3822690 for ; Thu, 28 May 2020 23:53:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EDE33206E2 for ; Thu, 28 May 2020 23:53:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LaoKiP+c" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDE33206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1BF76800B9; Thu, 28 May 2020 19:53:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1957080010; Thu, 28 May 2020 19:53:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05C00800B9; Thu, 28 May 2020 19:53:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0207.hostedemail.com [216.40.44.207]) by kanga.kvack.org (Postfix) with ESMTP id DFBDD80010 for ; Thu, 28 May 2020 19:53:04 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id ABFE62C8B for ; Thu, 28 May 2020 23:53:04 +0000 (UTC) X-FDA: 76867781088.11.cream27_8f37fd8cd7c2e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 87AF6180F8B80 for ; Thu, 28 May 2020 23:53:04 +0000 (UTC) X-Spam-Summary: 2,0,0,81ba1603fe3114f7,d41d8cd98f00b204,3307qxg0kclof2jqwfxrzxxjslttlqj.htrqnsz2-rrp0fhp.twl@flex--axelrasmussen.bounces.google.com,,RULES_HIT:41:152:355:379:541:800:960:966:973:988:989:1260:1277:1313:1314:1345:1431:1437:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3152:3354:3622:3865:3870:4118:4321:4385:5007:6261:6653:6737:6742:6743:7875:7903:7904:9969:10004:10400:11026:11473:11657:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:12986:14096:14097:14110:14181:14394:14659:14721:21080:21444:21451:21627:21990:30054,0,RBL:209.85.219.201:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: cream27_8f37fd8cd7c2e X-Filterd-Recvd-Size: 7599 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:53:04 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id d69so884495ybc.22 for ; Thu, 28 May 2020 16:53:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=2yUKX5hqzt1oHPQ4zNvGMcIYmUYCvNSNOVGMDvCOr3Q=; b=LaoKiP+cg9anMNFNg9Kd4pZYQAtw0FOInUmBUGQu2wKaKQMBIhjiAIspR0+jNfFIUR Wy/vpw0/UipeJjuws5qV8Ox9SMPtVfFnRuwoOi0V8A+Wocp5oYNImICPhZelh6PWwS4/ So2FoMAKYaJb/PO12JSLeHAPj6EbyM/HTreSXbWscfXxhtBb9vvaPGTHXg1Hk7AqxaVp //w3S/CyuQKBwFdks95jIwGGVJLHBtOmyW1cF22iv8nXTF4fUV8kosw00ToCNEDBxtQ7 HQ/W+gonCfXZjojOvbE9fmMDE18lM57VoyhIBV2QG//7DAKMV0qfrn4KKYJ15WbdM4u4 PiiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=2yUKX5hqzt1oHPQ4zNvGMcIYmUYCvNSNOVGMDvCOr3Q=; b=DxRTKZ+0QvTHFRfvDOYh4eUuPh/WDAwRZJo2xIYTCCOd3xNGaUgrTvJXM5/+pvX9H+ EK8FpRFWJhxjnepnHAteG0/YLRqV5M0+55EcRBUOo1yoNat2iRKTb10wMt9TxnpZSTNe w3eRpvMTq2F2WwLhgg8qg0kMtsDfVKkbhitXy2QqI2nSDbGHqufihJeWz8gUNDXQDBPS jbjlX0M+G4F24e4748jgH++4GDY76Kw1cDJSkY/QxcORRh3rJLLVccPau1zOmC48U6Ms 5Y5ft6xQG8r0/xcVkgc2NTfOp50zmtEwbL6gzTvOmiguamVcitfrWH9VR1eo1Z29kQ68 uysA== X-Gm-Message-State: AOAM533LWBRn2KWw0vUIVs+uYYhENI5L75Qi7bfAfuPuWwWvsxBjC46I reXKHISdHEyc/Xm+Ii6TlfLZeCNcbs9nZIvq0wO+ X-Google-Smtp-Source: ABdhPJxTDy0JwP7FWRktZyHchnosXxFw+yM7WVZbQCJl7DvaCtN9Tv/xctTb9glc5ZOrQcq9e7nC5UJyNd/5xZElQ7UW X-Received: by 2002:a25:e7cf:: with SMTP id e198mr820935ybh.188.1590709983105; Thu, 28 May 2020 16:53:03 -0700 (PDT) Date: Thu, 28 May 2020 16:53:00 -0700 Message-Id: <20200528235300.77135-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 4/7] mmap_lock: allocate histogram (if enabled) in mm_init From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: 87AF6180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In effect, allocate a histogram for each process. On fork, initialize a new empty histogram (don't keep recorded samples from the parent). Signed-off-by: Axel Rasmussen --- kernel/fork.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index e702e84897fa..afb036d81a73 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -94,6 +94,7 @@ #include #include #include +#include #include #include @@ -975,6 +976,58 @@ __setup("coredump_filter=", coredump_filter_setup); #include +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS +static const u64 mmap_lock_contention_buckets[] = { + 1, /* 0ns - uncontended acquisitions */ + 250, /* 250 ns */ + 375, /* 375 ns */ + 500, /* 500 ns */ + 1000, /* 1 us */ + 10000, /* 10 us */ + 100000, /* 100 us */ + 500000, /* 500 us */ + 1000000, /* 1 ms */ + 5000000, /* 5 ms */ + 10000000, /* 10 ms */ + 50000000, /* 50 ms */ + 100000000, /* 100 ms */ + 500000000, /* 500 ms */ + 1000000000, /* 1 s */ + 5000000000UL, /* 5 s */ + 10000000000UL, /* 10 s */ + ~0 /* > 10s */ +}; +#endif + +static void mm_init_histograms(struct mm_struct *mm) +{ +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + mm->mmap_lock_contention = + kmalloc(sizeof(struct histogram_rcu), GFP_KERNEL); + if (unlikely(!mm->mmap_lock_contention)) + return; + + if (unlikely(histogram_init_rcu( + mm->mmap_lock_contention, mmap_lock_contention_buckets, + ARRAY_SIZE(mmap_lock_contention_buckets)))) { + kfree(mm->mmap_lock_contention); + mm->mmap_lock_contention = NULL; + } +#endif +} + +static void mm_free_histograms(struct mm_struct *mm) +{ +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + if (unlikely(!mm->mmap_lock_contention)) + return; + + histogram_destroy_rcu(mm->mmap_lock_contention); + kfree(mm->mmap_lock_contention); + mm->mmap_lock_contention = NULL; +#endif +} + static void mm_init_aio(struct mm_struct *mm) { #ifdef CONFIG_AIO @@ -1015,6 +1068,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, atomic_set(&mm->mm_users, 1); atomic_set(&mm->mm_count, 1); mmap_init_lock(mm); + mm_init_histograms(mm); INIT_LIST_HEAD(&mm->mmlist); mm->core_state = NULL; mm_pgtables_bytes_init(mm); @@ -1078,6 +1132,7 @@ static inline void __mmput(struct mm_struct *mm) { VM_BUG_ON(atomic_read(&mm->mm_users)); + mm_free_histograms(mm); uprobe_clear_state(mm); exit_aio(mm); ksm_exit(mm); From patchwork Thu May 28 23:53:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577279 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83C5D90 for ; Thu, 28 May 2020 23:53:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 46A182075F for ; Thu, 28 May 2020 23:53:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NjbrFoLg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 46A182075F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5B5D5800BA; Thu, 28 May 2020 19:53:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 53E5E80010; Thu, 28 May 2020 19:53:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E049800BA; Thu, 28 May 2020 19:53:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 21A3880010 for ; Thu, 28 May 2020 19:53:08 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DA10B180AD804 for ; Thu, 28 May 2020 23:53:07 +0000 (UTC) X-FDA: 76867781214.04.field12_8fb0c3849355b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id BD44C800794E for ; Thu, 28 May 2020 23:53:07 +0000 (UTC) X-Spam-Summary: 2,0,0,302f83297b331b18,d41d8cd98f00b204,34k7qxg0kcl0d0houdvpxvvhqjrrjoh.frpolqx0-ppnydfn.ruj@flex--axelrasmussen.bounces.google.com,,RULES_HIT:41:152:355:379:541:800:960:973:988:989:1260:1277:1313:1314:1345:1431:1434:1437:1516:1518:1535:1541:1593:1594:1711:1730:1747:1777:1792:1963:2393:2559:2562:3138:3139:3140:3141:3142:3152:3352:3865:3866:3870:3871:4117:4321:4605:5007:6261:6653:6737:6742:6743:8603:8784:9010:9969:10004:10400:11026:11473:11658:11914:12043:12048:12114:12291:12296:12297:12438:12555:12895:12986:13069:13311:13357:14093:14097:14181:14394:14659:14721:21080:21433:21444:21451:21627:21990:30034:30054,0,RBL:209.85.219.202:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: field12_8fb0c3849355b X-Filterd-Recvd-Size: 6669 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:53:07 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id k186so892921ybc.19 for ; Thu, 28 May 2020 16:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=DrykbNQsUW+LkI5EFeIZc2m/lCKE6nVRHmkg0Fikq0Q=; b=NjbrFoLgfToEtzxh2vM6jkIcH/+dCfjW2ZnHmN0H36EWQPdDB8fCib1ut5wYterxKp fxtTCZty8OwZmBIA6DcQZpRf0PfpSnLsUkn7sYfvUmZNne7cvov3ZU0ynP7/zOOnfK0m MpyG/PaVlnGmgxeERdeQRt22qxMDHZh5qzlR2vIK6/9ntuyV2U6X+ub4WxbxRueY0Nri GNB1fFXfFtpIN037tJVg5xtObmyuwPC3dlX+V0CHruF1qjbUNgT7LU3RZeW6VHMyVxFG 8n1cisFhPweNuo14bMc3+7WLFKEnhKmDLJaGTTUBhz7Oi0WWiqvhKIAlVqAYxwBwYsUi vttg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=DrykbNQsUW+LkI5EFeIZc2m/lCKE6nVRHmkg0Fikq0Q=; b=SuoinnWi5hx17jLst711AvZosZG0CGoad855G55gHq9EBWLLjIkvbxZOU2hyyM4kJP L3pgfh5nI17unWqkTYL19olqRkYMWYOiNdKlA/WaGzQkjXDTZn2mF0Wa1CEL5UQj2gGL 9dojmasYjT4ksKRxFS8II0Wgub421qLWaQ2Sy173HRccdIIUp7/WE9Sw0TfF4IxCQU47 jbjvc1OoUSN9tbhcFR5i9XIfn4iaienX1DIBvuxoIDAxSf4a+52oJ+UOHtZ8VXMeHgzn AKr8yrKuAyWu42fcRwyjsMimnhul9pqYiMoYrWAIb+D8+bWwAB9ZwrCvWk69wkyVls0h 2SPA== X-Gm-Message-State: AOAM532c+BWtmElt7Awo+kUrDmXDddeKRV5/6u/sShBIfgYm0Wg3bWEe pOJ7mw6VgClcWqb0RmFVT4mNdNMcGClbmZhxAOSe X-Google-Smtp-Source: ABdhPJxsppeOKhttkL5/R/h/LoBSti7muDvl+5mL5lbZdZSHW2TVrq3/YQgoNaaXBe3dS9qdOf7kPpjUasxZZYm/V/ER X-Received: by 2002:a05:6902:6c3:: with SMTP id m3mr9677431ybt.70.1590709986498; Thu, 28 May 2020 16:53:06 -0700 (PDT) Date: Thu, 28 May 2020 16:53:03 -0700 Message-Id: <20200528235303.77873-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 5/7] mmap_lock: add /proc//mmap_lock_contention interface From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: BD44C800794E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When this file is read, we print a human-readable listing of histogram buckets, as well as the number of samples recorded in each bucket. Signed-off-by: Axel Rasmussen --- fs/proc/base.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/fs/proc/base.c b/fs/proc/base.c index a96377557db7..42d706474971 100644 --- a/fs/proc/base.c +++ b/fs/proc/base.c @@ -96,6 +96,8 @@ #include #include #include +#include +#include #include #include "internal.h" #include "fd.h" @@ -3115,6 +3117,26 @@ static int proc_stack_depth(struct seq_file *m, struct pid_namespace *ns, } #endif /* CONFIG_STACKLEAK_METRICS */ +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS +static int proc_pid_mmap_lock_contention_show(struct seq_file *m, + struct pid_namespace *ns, + struct pid *pid, + struct task_struct *task) +{ + int rc; + + if (unlikely(!task->mm->mmap_lock_contention)) + return 0; + + rc = histogram_print_buckets_rcu( + task->mm->mmap_lock_contention, + m->buf + m->count, m->size - m->count); + if (rc > 0) + m->count += rc; + return 0; +} +#endif /* CONFIG_MMAP_LOCK_HISTOGRAMS */ + /* * Thread groups */ @@ -3228,6 +3250,9 @@ static const struct pid_entry tgid_base_stuff[] = { #ifdef CONFIG_PROC_PID_ARCH_STATUS ONE("arch_status", S_IRUGO, proc_pid_arch_status), #endif +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + ONE("mmap_lock_contention", S_IRUGO, proc_pid_mmap_lock_contention_show), +#endif }; static int proc_tgid_base_readdir(struct file *file, struct dir_context *ctx) From patchwork Thu May 28 23:53:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577281 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 310621391 for ; Thu, 28 May 2020 23:53:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E3E3B206E2 for ; Thu, 28 May 2020 23:53:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PbwvPJLA" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E3E3B206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D7E07800BB; Thu, 28 May 2020 19:53:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CDFA580010; Thu, 28 May 2020 19:53:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA7B4800BB; Thu, 28 May 2020 19:53:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0076.hostedemail.com [216.40.44.76]) by kanga.kvack.org (Postfix) with ESMTP id 9EF2380010 for ; Thu, 28 May 2020 19:53:11 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6137B181AC217 for ; Thu, 28 May 2020 23:53:11 +0000 (UTC) X-FDA: 76867781382.01.land86_90311b88df818 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 44E2C1801BB8F for ; Thu, 28 May 2020 23:53:11 +0000 (UTC) X-Spam-Summary: 2,0,0,55eedff901b3eca6,d41d8cd98f00b204,35u7qxg0kcmag3krxgys0yyktmuumrk.iusrot03-ssq1giq.uxm@flex--axelrasmussen.bounces.google.com,,RULES_HIT:1:2:41:152:355:379:541:800:960:973:988:989:1260:1277:1313:1314:1345:1431:1437:1516:1518:1593:1594:1605:1730:1747:1777:1792:1801:1981:2194:2199:2393:2559:2562:2899:3138:3139:3140:3141:3142:3152:3865:3866:3867:3868:3870:3871:3874:4051:4250:4321:4605:5007:6119:6261:6653:6737:6742:6743:7875:7903:7904:8660:9149:9969:10004:11026:11232:11233:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:12986:13148:13161:13180:13229:13230:14394:14659:21080:21324:21444:21451:21627:21939:21987:21990:30012:30029:30054,0,RBL:209.85.222.201:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:22,LUA_SUMMARY:none X-HE-Tag: land86_90311b88df818 X-Filterd-Recvd-Size: 13496 Received: from mail-qk1-f201.google.com (mail-qk1-f201.google.com [209.85.222.201]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:53:10 +0000 (UTC) Received: by mail-qk1-f201.google.com with SMTP id 16so454135qka.15 for ; Thu, 28 May 2020 16:53:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=y5z9RRmzPLSTV+z4iFsp+NbItXdDsIJJob5oit7HC+Q=; b=PbwvPJLA84j2FxNJqtanbEKCTBQAaEmTZzaMekn0CUs1sSSWHLmPBMQjXmtj9R8mzM 68IWTZCPPoJgquVrjCS87XRoGyyJNqThN8CcsFAkbAhVhZEmAIXax/GHc5j3DtCZ4xXp OOnfLrUVjiq8cfBALXXNd622/FZaLqRTIvx8s5+evsSZL3/3M59GiDuv+ix5NA9GAzzd KU6NXvn4eANDQR7e9zxn/xeyyWP3ylY/WxolD875Un0gxbBZqumgY4Em44zNbFCpG/eb 64e7hM+hBOIQIFVbYOADnl3KL47Uvg6irhJO5q//9L32af5/HdadEzRgRrYUExyj5AJg m9QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=y5z9RRmzPLSTV+z4iFsp+NbItXdDsIJJob5oit7HC+Q=; b=cMQjN2K0JQMX4nCDxkMalO9Yc36L1+0n3eyvs44S4kDuykPTeeBX8AdyD+JSgK+XCJ hIt1HLW1oN069e5CqLmnjPiW9fAkypnqlMUVEyWKU8qq712kIMZbtEm2EQnM71UfvPWM 8rRS+XO7MhxZOCBRpGLPLiiUIUW0nEFmNDv+pk3cHAJXkZKBJWujxSYwSjTdiqlj/GEu lt6LCmsDFQphAxxXSCRvesIWjqefWHFt8xgmutajDFUb+PmTDqHDh3c/uRg86HBN3lPY WkvQFAMsVimhczjiB1nkp/2seqFkzoY/6nDQpqrvzfZQQDe4E66XQizYZM4Wngs8Klir f4Qg== X-Gm-Message-State: AOAM530K4p6YRefLb9xpJWnO08RBunlwoTxBjRTlbkdXTAis1ZwV9iN9 MujeG/lUsF3dYE9G/ak18aniFQQWhcKHV6SME+XZ X-Google-Smtp-Source: ABdhPJwVwaf25xKjcGBcucJxv4w7szHWlqqDKdw8Yeo6/nQtd2SjVQBiMM+4DU6uiiszCVeYkO7I6OQfFXPDkO/Mt51F X-Received: by 2002:a0c:8168:: with SMTP id 95mr5860568qvc.68.1590709989878; Thu, 28 May 2020 16:53:09 -0700 (PDT) Date: Thu, 28 May 2020 16:53:07 -0700 Message-Id: <20200528235307.78535-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 6/7] mmap_lock: increment histogram whenever mmap_lock is acquired From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: 44E2C1801BB8F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To minimize overhead, we call down_{read,write}_trylock first. On success, record the acquisition with "0" latency. On failure, we call e.g. down_{read,write} and record the actual measured latency. The reason for recording uncontended acquisitions is so we have an accurate count of total acquisition attempts. This allows us to compute meaningful percentiles over the histogram data - e.g., "x% of lock acquisitions succeeded in <= y ms". Note that in e.g. the killable lock case, we record the latency even if the acquisition doesn't actually succeed. This is so we can see cases where we waited a long time, even if acquisition eventually ended up failing. Nested locks are a weird case, in that we can't do the "only compute latency in the contended case" optimization without reaching into rwsem.c's internal API. The approach this commit uses (removing "static inline" from the couple of functions we need, and forward declaring them in mmap_lock.c) was chosen instead of: - Adding mmap_lock specific code to rwsem.c directly, as this would pollute a generic file with a specific use case's implementation. - Moving the API internals we need up into rwsem.h, as this would expose them more widely than arguably we'd like. - Just computing latency in all cases, as this would add nontrivial overhead to the uncontended lock case. Signed-off-by: Axel Rasmussen --- include/linux/mmap_lock.h | 87 +++++++++++++++++++++++++++++++++++---- kernel/locking/rwsem.c | 4 +- mm/Makefile | 1 + mm/mmap_lock.c | 34 +++++++++++++++ 4 files changed, 117 insertions(+), 9 deletions(-) create mode 100644 mm/mmap_lock.c diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 9dc632add390..1c212a403911 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -1,11 +1,80 @@ #ifndef _LINUX_MMAP_LOCK_H #define _LINUX_MMAP_LOCK_H +#include +#include #include +#include +#include #define MMAP_LOCK_INITIALIZER(name) \ .mmap_lock = __RWSEM_INITIALIZER(name.mmap_lock), +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS +static inline void __mmap_lock_histogram_record_duration(struct mm_struct *mm, + u64 duration) +{ + if (likely(mm->mmap_lock_contention)) + histogram_record_rcu(mm->mmap_lock_contention, duration, 1); +} + +static inline void mmap_lock_histogram_record(struct mm_struct *mm, + u64 start_time_ns) +{ + __mmap_lock_histogram_record_duration(mm, + sched_clock() - start_time_ns); +} +#endif + +static inline bool __mmap_trylock(struct mm_struct *mm, + int (*trylock)(struct rw_semaphore *)) +{ + bool ret = trylock(&mm->mmap_lock) != 0; + +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + if (ret) + __mmap_lock_histogram_record_duration(mm, 0); +#endif + return ret; +} + +static inline void __mmap_lock(struct mm_struct *mm, + int (*trylock)(struct rw_semaphore *), + void (*lock)(struct rw_semaphore *)) +{ +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + u64 start_time_ns; + + if (!__mmap_trylock(mm, trylock)) { + start_time_ns = sched_clock(); + lock(&mm->mmap_lock); + mmap_lock_histogram_record(mm, start_time_ns); + } +#else + lock(&mm->mmap_lock); +#endif +} + +static inline int __mmap_lock_return(struct mm_struct *mm, + int (*trylock)(struct rw_semaphore *), + int (*lock)(struct rw_semaphore *)) +{ +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS + u64 start_time_ns; + int ret; + + if (!__mmap_trylock(mm, trylock)) { + start_time_ns = sched_clock(); + ret = lock(&mm->mmap_lock); + mmap_lock_histogram_record(mm, start_time_ns); + return ret; + } + return 0; +#else + return lock(&mm->mmap_lock); +#endif +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -13,22 +82,26 @@ static inline void mmap_init_lock(struct mm_struct *mm) static inline void mmap_write_lock(struct mm_struct *mm) { - down_write(&mm->mmap_lock); + __mmap_lock(mm, down_write_trylock, down_write); } +#ifdef CONFIG_MMAP_LOCK_HISTOGRAMS +void mmap_write_lock_nested(struct mm_struct *mm, int subclass); +#else static inline void mmap_write_lock_nested(struct mm_struct *mm, int subclass) { down_write_nested(&mm->mmap_lock, subclass); } +#endif static inline int mmap_write_lock_killable(struct mm_struct *mm) { - return down_write_killable(&mm->mmap_lock); + return __mmap_lock_return(mm, down_write_trylock, down_write_killable); } static inline bool mmap_write_trylock(struct mm_struct *mm) { - return down_write_trylock(&mm->mmap_lock) != 0; + return __mmap_trylock(mm, down_write_trylock); } static inline void mmap_write_unlock(struct mm_struct *mm) @@ -43,17 +116,17 @@ static inline void mmap_write_downgrade(struct mm_struct *mm) static inline void mmap_read_lock(struct mm_struct *mm) { - down_read(&mm->mmap_lock); + __mmap_lock(mm, down_read_trylock, down_read); } static inline int mmap_read_lock_killable(struct mm_struct *mm) { - return down_read_killable(&mm->mmap_lock); + return __mmap_lock_return(mm, down_read_trylock, down_read_killable); } static inline bool mmap_read_trylock(struct mm_struct *mm) { - return down_read_trylock(&mm->mmap_lock) != 0; + return __mmap_trylock(mm, down_read_trylock); } static inline void mmap_read_unlock(struct mm_struct *mm) @@ -63,7 +136,7 @@ static inline void mmap_read_unlock(struct mm_struct *mm) static inline bool mmap_read_trylock_non_owner(struct mm_struct *mm) { - if (down_read_trylock(&mm->mmap_lock)) { + if (__mmap_trylock(mm, down_read_trylock)) { rwsem_release(&mm->mmap_lock.dep_map, _RET_IP_); return true; } diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index f11b9bd3431d..d041c5ae8b4d 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -1380,7 +1380,7 @@ static inline int __down_read_trylock(struct rw_semaphore *sem) /* * lock for writing */ -static inline void __down_write(struct rw_semaphore *sem) +void __down_write(struct rw_semaphore *sem) { long tmp = RWSEM_UNLOCKED_VALUE; @@ -1405,7 +1405,7 @@ static inline int __down_write_killable(struct rw_semaphore *sem) return 0; } -static inline int __down_write_trylock(struct rw_semaphore *sem) +int __down_write_trylock(struct rw_semaphore *sem) { long tmp; diff --git a/mm/Makefile b/mm/Makefile index fccd3756b25f..2a54af3fc715 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -112,3 +112,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o +obj-$(CONFIG_MMAP_LOCK_HISTOGRAMS) += mmap_lock.o diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c new file mode 100644 index 000000000000..f166cf40c60a --- /dev/null +++ b/mm/mmap_lock.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +#define MMAP_LOCK_CONTENDED(_mm, try, lock) \ + do { \ + if (!try(&(_mm)->mmap_lock)) { \ + u64 start_time_ns; \ + lock_contended(&(_mm)->mmap_lock.dep_map, _RET_IP_); \ + start_time_ns = sched_clock(); \ + lock(&(_mm)->mmap_lock); \ + mmap_lock_histogram_record(_mm, start_time_ns); \ + } else { \ + __mmap_lock_histogram_record_duration(_mm, 0); \ + } \ + lock_acquired(&(_mm)->mmap_lock.dep_map, _RET_IP_); \ + } while (0) + +/* Defined in kernel/locking/rwsem.c. */ +extern int __down_write_trylock(struct rw_semaphore *sem); +extern void __down_write(struct rw_semaphore *sem); + +void mmap_write_lock_nested(struct mm_struct *mm, int subclass) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + might_sleep(); + rwsem_acquire(&mm->mmap_lock.dep_map, subclass, 0, _RET_IP_); + MMAP_LOCK_CONTENDED(mm, __down_write_trylock, __down_write); +#else + mmap_write_lock(mm); +#endif +} +EXPORT_SYMBOL(mmap_write_lock_nested); From patchwork Thu May 28 23:53:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 11577283 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 28ECE90 for ; Thu, 28 May 2020 23:53:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF3D5206E2 for ; Thu, 28 May 2020 23:53:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="N0GQM43B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DF3D5206E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D581C800BC; Thu, 28 May 2020 19:53:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CB97680010; Thu, 28 May 2020 19:53:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B80A1800BC; Thu, 28 May 2020 19:53:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0054.hostedemail.com [216.40.44.54]) by kanga.kvack.org (Postfix) with ESMTP id 9DC0280010 for ; Thu, 28 May 2020 19:53:14 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5A6308245571 for ; Thu, 28 May 2020 23:53:14 +0000 (UTC) X-FDA: 76867781508.13.jump70_909f6316d6a1c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 3A50F18140B60 for ; Thu, 28 May 2020 23:53:14 +0000 (UTC) X-Spam-Summary: 2,0,0,16a0a95475ef59b8,d41d8cd98f00b204,36e7qxg0kcmmj6nu0j1v311nwpxxpun.lxvurw36-vvt4jlt.x0p@flex--axelrasmussen.bounces.google.com,,RULES_HIT:41:152:355:379:541:800:960:973:988:989:1260:1277:1313:1314:1345:1431:1434:1437:1516:1518:1535:1543:1593:1594:1711:1730:1747:1777:1792:1801:2393:2559:2562:2892:2897:3138:3139:3140:3141:3142:3152:3354:3865:3866:3867:3868:3870:3871:3872:3874:4119:4321:4605:5007:6261:6653:6737:6742:6743:7875:9149:9969:10004:10400:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:12986:13161:13229:14181:14394:14659:14721:14819:21080:21444:21451:21627:21789:21966:22047:30054:30075,0,RBL:209.85.219.202:@flex--axelrasmussen.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: jump70_909f6316d6a1c X-Filterd-Recvd-Size: 8603 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Thu, 28 May 2020 23:53:13 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id g9so966954ybc.1 for ; Thu, 28 May 2020 16:53:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=b4hv8mBjeZzSMuC0+qYdun91rFKPD4QruDrMsYOAH0w=; b=N0GQM43BUCysqImySOile1VYR9C2wgq+hVF1K8OH0g/X8oP59BWXBbmG41oX3kb/bz c8fooc5AGVBDCPxu9dPZTJa2fmqN9ySgvnIzNUayrpg4WYK1lh1bWjIzYF1A3NTmFEM4 nIgMRn7bIQnKsCNWaeA6QPAIX2dKS/Q0WiWycoNgiBzi12Ygg6RpI8L+ST7qQD4Gghft N59LUKKo+VK+EVzTFqPuBIm9MF6pGZ3a4WuQhlZUqr/gflUrtQaEwqnuO2f+GMoYud1e rILCD+SK9XA7ZHvBanqnTtsmnM/WJJlbK4gcW6aXDn/JtXjJgepiPE0lCRwmYDOU4oTI BSpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=b4hv8mBjeZzSMuC0+qYdun91rFKPD4QruDrMsYOAH0w=; b=UQm4sn7meiA1urJYm9//02qvcFB6dKPVZ6TR0MynD5ZvqodBy9LT1pCJxSegcGOj8d qf8St4eTrAIJjraDKBKB24tZ/nppSYZwES1XrRmpR1HF4ikraLpj3RV5Snnn6V0nwcOX Vtxxk+t02PVN76Q7NWO8YmFvINI+lpiedWUmwQYKalT+n66n6hT6wwV33d46/rZGT55L ZBtDw89I8PXrE7oinqEzIrptpW1oQlPI+8scgReg/zS3Y37F0CYodXRuREU+C6pb2NcR DJyZpp96kiDIF3/KvZj6mAFmTD0Rcqfu/AEDnWPT62Sh9xne3UkrTUenFTodNLmqlmTv 425A== X-Gm-Message-State: AOAM533MX3uWR4Wyf9Zr8vBqnayDMcgPAbRCEttc4A6CQ1qjKJGj7xp3 xHmf+rE5oLJaOE/UjYuQMFPJ1Xu8gjrFSpuCDGVE X-Google-Smtp-Source: ABdhPJzZ8MROg+Gbmi6hypGFS6Dswd6mXkwle3uFwpRhbuwvJ8PaqPg2HSiw67IOoPn1MjktHFOSM2RBy7YuDJcBmstW X-Received: by 2002:a5b:5d0:: with SMTP id w16mr9499053ybp.110.1590709992949; Thu, 28 May 2020 16:53:12 -0700 (PDT) Date: Thu, 28 May 2020 16:53:10 -0700 Message-Id: <20200528235310.79194-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.rc0.183.gde8f92d652-goog Subject: [PATCH v2 7/7] mmap_lock: add a tracepoint to contended acquisitions From: Axel Rasmussen To: Andrew Morton , David Rientjes , Davidlohr Bueso , Ingo Molnar , Ingo Molnar , Jerome Glisse , Laurent Dufour , "Liam R . Howlett" , Matthew Wilcox , Michel Lespinasse , Peter Zijlstra , Vlastimil Babka , Will Deacon Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, AKASHI Takahiro , Aleksa Sarai , Alexander Potapenko , Alexey Dobriyan , Al Viro , Andrei Vagin , Ard Biesheuvel , Brendan Higgins , chenqiwu , Christian Brauner , Christian Kellner , Corentin Labbe , Daniel Jordan , Dan Williams , David Gow , "David S. Miller" , "Dmitry V. Levin" , "Eric W. Biederman" , Eugene Syromiatnikov , Jamie Liu , Jason Gunthorpe , John Garry , John Hubbard , Jonathan Adams , Junaid Shahid , Kees Cook , "Kirill A. Shutemov" , Konstantin Khlebnikov , Krzysztof Kozlowski , Mark Rutland , Masahiro Yamada , Masami Hiramatsu , Mathieu Desnoyers , Michal Hocko , Mikhail Zaslonko , Petr Mladek , Ralph Campbell , Randy Dunlap , Roman Gushchin , Shakeel Butt , Steven Rostedt , Tal Gilboa , Thomas Gleixner , " =?utf-8?q?Uwe_Kleine-K=C3=B6nig?= " , Vincenzo Frascino , Yang Shi , Yu Zhao , Axel Rasmussen X-Rspamd-Queue-Id: 3A50F18140B60 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows for the possibiilty of attaching to the tracepoint, to collect more detailed information than the contention histogram would otherwise provide. The tracepoint is called in the same codepath where we increment the contention histogram (so, it's not really available if the histogram is disabled). The intent is that this is a more detailed but more expensive *extension* to the basic data the contention histogram provides. Signed-off-by: Axel Rasmussen --- include/linux/mmap_lock.h | 9 +++++++-- include/trace/events/mmap_lock.h | 34 ++++++++++++++++++++++++++++++++ mm/mmap_lock.c | 12 +++++++++++ 3 files changed, 53 insertions(+), 2 deletions(-) create mode 100644 include/trace/events/mmap_lock.h diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 1c212a403911..4faed7a5bb49 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -11,6 +11,8 @@ .mmap_lock = __RWSEM_INITIALIZER(name.mmap_lock), #ifdef CONFIG_MMAP_LOCK_HISTOGRAMS +void __trace_mmap_lock_contended(struct mm_struct *mm, u64 duration); + static inline void __mmap_lock_histogram_record_duration(struct mm_struct *mm, u64 duration) { @@ -21,8 +23,11 @@ static inline void __mmap_lock_histogram_record_duration(struct mm_struct *mm, static inline void mmap_lock_histogram_record(struct mm_struct *mm, u64 start_time_ns) { - __mmap_lock_histogram_record_duration(mm, - sched_clock() - start_time_ns); + u64 duration = sched_clock() - start_time_ns; + + /* This function is only used in the contended case, so trace here. */ + __trace_mmap_lock_contended(mm, duration); + __mmap_lock_histogram_record_duration(mm, duration); } #endif diff --git a/include/trace/events/mmap_lock.h b/include/trace/events/mmap_lock.h new file mode 100644 index 000000000000..afd581f5fbdb --- /dev/null +++ b/include/trace/events/mmap_lock.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM mmap_lock + +#if !defined(_TRACE_MMAP_LOCK_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_MMAP_LOCK_H + +#include +#include + +struct mm_struct; + +TRACE_EVENT(mmap_lock_contended, + + TP_PROTO(struct mm_struct *mm, u64 duration), + + TP_ARGS(mm, duration), + + TP_STRUCT__entry( + __field(struct mm_struct *, mm) + __field(u64, duration) + ), + + TP_fast_assign( + __entry->mm = mm; + __entry->duration = duration; + ), + + TP_printk("mm=%p duration=%llu\n", __entry->mm, __entry->duration)); + +#endif /* _TRACE_MMAP_LOCK_H */ + +/* This part must be outside protection */ +#include diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c index f166cf40c60a..406c43039b0b 100644 --- a/mm/mmap_lock.c +++ b/mm/mmap_lock.c @@ -1,8 +1,20 @@ // SPDX-License-Identifier: GPL-2.0 +#define CREATE_TRACE_POINTS +#include + #include #include #include +void __trace_mmap_lock_contended(struct mm_struct *mm, u64 duration) +{ + /* This must be in a separate file, as otherwise there's a circular + * dependency between linux/mmap_lock.h and trace/events/mmap_lock.h. + */ + trace_mmap_lock_contended(mm, duration); +} +EXPORT_SYMBOL_GPL(__trace_mmap_lock_contended); + #define MMAP_LOCK_CONTENDED(_mm, try, lock) \ do { \ if (!try(&(_mm)->mmap_lock)) { \