From patchwork Wed Nov 3 18:47:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jimmy Shiu X-Patchwork-Id: 12601465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2600C433EF for ; Wed, 3 Nov 2021 18:47:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3A26F600EF for ; Wed, 3 Nov 2021 18:47:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3A26F600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id AEE226B006C; Wed, 3 Nov 2021 14:47:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9C1A6B0072; Wed, 3 Nov 2021 14:47:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B2986B0073; Wed, 3 Nov 2021 14:47:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id 8AC9E6B006C for ; Wed, 3 Nov 2021 14:47:22 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3D7DB5D5D4 for ; Wed, 3 Nov 2021 18:47:22 +0000 (UTC) X-FDA: 78768501924.13.B14C1A1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 49EECD0000B5 for ; Wed, 3 Nov 2021 18:47:13 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id v20-20020a25fc14000000b005c2109e5ad1so5188276ybd.9 for ; Wed, 03 Nov 2021 11:47:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=jmjrIilQHP+skOfmaa+xe8HJHINvgaf5bkHoaKFVkNg=; b=h7FZte4wdxvX/GoE7wG+OGNM9Zqxq/HwQ3TPnFC+qbpYCRLo7Qjzkv/G5jwFEbNivt Pa2bdTQezVjZ3tH+fvzam8NRDVPHuIE58v2YiTeQmLsF1arGgOxkFVoAn3PtOO0KXP70 scH4Qqf2J3pfmsLzHTAYLPBFHS/W4zN4GiLMfZ48Ww1zgKG5wmnXKz2Ylu57uGnhB3Zx Hj0nwNV/O8ctzUq9yMY+Gm/H0lk7SZHLJbgb0kK+dRHP/3EhSPsGgAFcxBW4CsneAZi8 YU+FG5Y/8mbzLNhsmujJwM4uq5umblGerG9nYxHhNqIsL3c3wTfEnj02efYMIhXqkb+/ wJNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=jmjrIilQHP+skOfmaa+xe8HJHINvgaf5bkHoaKFVkNg=; b=H64rbXXz499bAE1CqFX2FRu4j8kmh8hw5In7brQod0UlyvtqCfEmySR86bs61zrvac mOzaEG63KXwlxRT7YkiChK/ORVLPaksj6/C0/eHJmWfOusM63g1CavL0JdAOTO57gtDy RSIdGgWwLA/JLmwuy3QZGGq5wyP9X0UHAKYUUece0jy5l51o1YOwXuYs0FbBlz0isxQH TD2GeeC8QhDTYFYlEpITvJn9q2DKAXtwhtmLxWmALIBGpdQrSsfVCot2Pd4r7l96kGi4 ifAHcWWlbMEbdyZa+Th56W+xmT+Keblc80zF0NxlduwBPb967aap4OwKzCL4EPllRe+v tk2w== X-Gm-Message-State: AOAM531S6JbJfuMJ2z4Z7V5C0XmNHmgh1EEzPvMLjg6Qtrt+y9uJeIbx 9rfUAEyTR90RauCKWUt+biMHMSruBhYIKCc= X-Google-Smtp-Source: ABdhPJwNeslSipsEvLSi3Fhb2ny7o66d0r2EFJoRjTL5psoKrdCYQq7OkcMHfZtRNguGpXypg9taTa0BWE4xIqQ= X-Received: from jimmyshiu.ntc.corp.google.com ([2401:fa00:fc:202:61eb:5079:5daf:1c89]) (user=jimmyshiu job=sendgmr) by 2002:a25:5057:: with SMTP id e84mr42245627ybb.299.1635965240974; Wed, 03 Nov 2021 11:47:20 -0700 (PDT) Date: Thu, 4 Nov 2021 02:47:03 +0800 Message-Id: <20211103184708.1778294-1-jimmyshiu@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.34.0.rc0.344.g81b53c2807-goog Subject: [PATCH] SCHED: attribute page lock and waitqueue functions as sched From: Jimmy Shiu To: mingo@redhat.com Cc: jimmyshiu@google.com, joaodias@google.com, wvw@google.com, Minchan Kim , Will McVicker , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Andrew Morton , "Matthew Wilcox (Oracle)" , David Howells , Vlastimil Babka , William Kucharski , "Kirill A. Shutemov" , Andreas Gruenbacher , linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 49EECD0000B5 X-Stat-Signature: wk5fbaq7jbii91mg87y5o5jsaanjw4hr Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=h7FZte4w; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3ONmCYQkKCAksrvv71qr3pxxpun.lxvurw36-vvt4jlt.x0p@flex--jimmyshiu.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ONmCYQkKCAksrvv71qr3pxxpun.lxvurw36-vvt4jlt.x0p@flex--jimmyshiu.bounces.google.com X-HE-Tag: 1635965233-186460 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: trace_sched_blocked_trace in CFS is really useful for debugging via trace because it tell where the process was stuck on callstack. For example, <...>-6143 ( 6136) [005] d..2 50.278987: sched_blocked_reason: pid=6136 iowait=0 caller=SyS_mprotect+0x88/0x208 <...>-6136 ( 6136) [005] d..2 50.278990: sched_blocked_reason: pid=6142 iowait=0 caller=do_page_fault+0x1f4/0x3b0 <...>-6142 ( 6136) [006] d..2 50.278996: sched_blocked_reason: pid=6144 iowait=0 caller=SyS_prctl+0x52c/0xb58 <...>-6144 ( 6136) [006] d..2 50.279007: sched_blocked_reason: pid=6136 iowait=0 caller=vm_mmap_pgoff+0x74/0x104 However, sometime it gives pointless information like this. RenderThread-2322 ( 1805) [006] d.s3 50.319046: sched_blocked_reason: pid=6136 iowait=1 caller=__lock_page_killable+0x17c/0x220 logd.writer-594 ( 587) [002] d.s3 50.334011: sched_blocked_reason: pid=6126 iowait=1 caller=wait_on_page_bit+0x194/0x208 kworker/u16:13-333 ( 333) [007] d.s4 50.343161: sched_blocked_reason: pid=6136 iowait=1 caller=__lock_page_killable+0x17c/0x220 Such wait_on_page_bit, __lock_page_killable are pointless because it doesn't carry on higher information to identify the callstack. The reason is page_lock and waitqueue are special synchronization method unlike other normal locks(mutex, spinlock). Let's mark them as "__sched" so get_wchan which used in trace_sched_blocked_trace could detect it and skip them. It will produce more meaningful callstack function like this. <...>-2867 ( 1068) [002] d.h4 124.209701: sched_blocked_reason: pid=329 iowait=0 caller=worker_thread+0x378/0x470 <...>-2867 ( 1068) [002] d.s3 124.209763: sched_blocked_reason: pid=8454 iowait=1 caller=__filemap_fdatawait_range+0xa0/0x104 <...>-2867 ( 1068) [002] d.s4 124.209803: sched_blocked_reason: pid=869 iowait=0 caller=worker_thread+0x378/0x470 ScreenDecoratio-2364 ( 1867) [002] d.s3 124.209973: sched_blocked_reason: pid=8454 iowait=1 caller=f2fs_wait_on_page_writeback+0x84/0xcc ScreenDecoratio-2364 ( 1867) [002] d.s4 124.209986: sched_blocked_reason: pid=869 iowait=0 caller=worker_thread+0x378/0x470 <...>-329 ( 329) [000] d..3 124.210435: sched_blocked_reason: pid=538 iowait=0 caller=worker_thread+0x378/0x470 kworker/u16:13-538 ( 538) [007] d..3 124.210450: sched_blocked_reason: pid=6 iowait=0 caller=worker_thread+0x378/0x470 Test: build pass and boot to home. Bug: 144961676 Bug: 144713689 Bug: 172212772 Signed-off-by: Minchan Kim Signed-off-by: Jimmy Shiu (cherry picked from commit 1e4de875d9e0cfaccf5131bcc709ae8646cdc168) Signed-off-by: Will McVicker --- include/linux/pagemap.h | 17 +++++++++-------- kernel/sched/wait.c | 8 +++++--- mm/filemap.c | 14 +++++++------- 3 files changed, 21 insertions(+), 18 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index db2c3e3eb1cf..12e82ff7686a 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -15,6 +15,7 @@ #include #include /* for in_interrupt() */ #include +#include struct pagevec; @@ -687,7 +688,7 @@ static inline void folio_lock(struct folio *folio) /* * lock_page may only be called if we have the page's inode pinned. */ -static inline void lock_page(struct page *page) +static inline __sched void lock_page(struct page *page) { struct folio *folio; might_sleep(); @@ -697,7 +698,7 @@ static inline void lock_page(struct page *page) __folio_lock(folio); } -static inline int folio_lock_killable(struct folio *folio) +static inline __sched int folio_lock_killable(struct folio *folio) { might_sleep(); if (!folio_trylock(folio)) @@ -710,7 +711,7 @@ static inline int folio_lock_killable(struct folio *folio) * signals. It returns 0 if it locked the page and -EINTR if it was * killed while waiting. */ -static inline int lock_page_killable(struct page *page) +static inline __sched int lock_page_killable(struct page *page) { return folio_lock_killable(page_folio(page)); } @@ -722,7 +723,7 @@ static inline int lock_page_killable(struct page *page) * Return value and mmap_lock implications depend on flags; see * __folio_lock_or_retry(). */ -static inline bool lock_page_or_retry(struct page *page, struct mm_struct *mm, +static inline __sched bool lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags) { struct folio *folio; @@ -746,25 +747,25 @@ int folio_wait_bit_killable(struct folio *folio, int bit_nr); * ie with increased "page->count" so that the folio won't * go away during the wait.. */ -static inline void folio_wait_locked(struct folio *folio) +static inline __sched void folio_wait_locked(struct folio *folio) { if (folio_test_locked(folio)) folio_wait_bit(folio, PG_locked); } -static inline int folio_wait_locked_killable(struct folio *folio) +static inline __sched int folio_wait_locked_killable(struct folio *folio) { if (!folio_test_locked(folio)) return 0; return folio_wait_bit_killable(folio, PG_locked); } -static inline void wait_on_page_locked(struct page *page) +static inline __sched void wait_on_page_locked(struct page *page) { folio_wait_locked(page_folio(page)); } -static inline int wait_on_page_locked_killable(struct page *page) +static inline __sched int wait_on_page_locked_killable(struct page *page) { return folio_wait_locked_killable(page_folio(page)); } diff --git a/kernel/sched/wait.c b/kernel/sched/wait.c index 76577d1642a5..a5975579a741 100644 --- a/kernel/sched/wait.c +++ b/kernel/sched/wait.c @@ -404,7 +404,8 @@ void finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_en } EXPORT_SYMBOL(finish_wait); -int autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key) +__sched int autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned int mode, + int sync, void *key) { int ret = default_wake_function(wq_entry, mode, sync, key); @@ -440,7 +441,7 @@ static inline bool is_kthread_should_stop(void) * } smp_mb(); // C * remove_wait_queue(&wq_head, &wait); wq_entry->flags |= WQ_FLAG_WOKEN; */ -long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout) +__sched long wait_woken(struct wait_queue_entry *wq_entry, unsigned int mode, long timeout) { /* * The below executes an smp_mb(), which matches with the full barrier @@ -465,7 +466,8 @@ long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout) } EXPORT_SYMBOL(wait_woken); -int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key) +__sched int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned int mode, + int sync, void *key) { /* Pairs with the smp_store_mb() in wait_woken(). */ smp_mb(); /* C */ diff --git a/mm/filemap.c b/mm/filemap.c index bfcef6ff7a27..ad4268ee1bf1 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1271,7 +1271,7 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; -static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, +static inline __sched int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { wait_queue_head_t *q = folio_waitqueue(folio); @@ -1411,13 +1411,13 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR; } -void folio_wait_bit(struct folio *folio, int bit_nr) +__sched void folio_wait_bit(struct folio *folio, int bit_nr) { folio_wait_bit_common(folio, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); } EXPORT_SYMBOL(folio_wait_bit); -int folio_wait_bit_killable(struct folio *folio, int bit_nr) +__sched int folio_wait_bit_killable(struct folio *folio, int bit_nr) { return folio_wait_bit_common(folio, bit_nr, TASK_KILLABLE, SHARED); } @@ -1628,21 +1628,21 @@ EXPORT_SYMBOL_GPL(page_endio); * __folio_lock - Get a lock on the folio, assuming we need to sleep to get it. * @folio: The folio to lock */ -void __folio_lock(struct folio *folio) +__sched void __folio_lock(struct folio *folio) { folio_wait_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE, EXCLUSIVE); } EXPORT_SYMBOL(__folio_lock); -int __folio_lock_killable(struct folio *folio) +__sched int __folio_lock_killable(struct folio *folio) { return folio_wait_bit_common(folio, PG_locked, TASK_KILLABLE, EXCLUSIVE); } EXPORT_SYMBOL_GPL(__folio_lock_killable); -static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) +static __sched int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) { struct wait_queue_head *q = folio_waitqueue(folio); int ret = 0; @@ -1679,7 +1679,7 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) * If neither ALLOW_RETRY nor KILLABLE are set, will always return true * with the folio locked and the mmap_lock unperturbed. */ -bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm, +__sched bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags) { if (fault_flag_allow_retry_first(flags)) {