From patchwork Sat Sep 10 06:50:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12972428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1DB8C6FA8D for ; Sat, 10 Sep 2022 06:51:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04BAF6B0071; Sat, 10 Sep 2022 02:51:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDBF08D0001; Sat, 10 Sep 2022 02:51:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCE5A6B0071; Sat, 10 Sep 2022 02:51:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9F3F86B0072 for ; Sat, 10 Sep 2022 02:51:22 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7E3A2A0DA4 for ; Sat, 10 Sep 2022 06:51:22 +0000 (UTC) X-FDA: 79895254404.31.BC03778 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf29.hostedemail.com (Postfix) with ESMTP id 668391200B9 for ; Sat, 10 Sep 2022 06:51:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=TWMB057d75ixVxyjdIFV1XXGOo0lh1V+gLJp5z7ed9M=; b=NBIurit5r6+q8mICiFS5WC3bos uEapd4J0qYeXSXm7FM/nQTunbL1M8v4YBFR9liWo5ysvmgUYjxYsTtwK2M8sQCLKESTfdhMW3H/3l Jz7Q6BUAlaFmzwe8swKOZJT70jf5yW9XUqZoi1Yecm/TY3dhKnFbxNoLAsYNyqPvFRsmJiM/MEZiN npLdahDgF7NnGQTEutWPlQcdWL2ZeMoJ8yHrx6o8043Pvp0q9HwICGWwVeK1zI8LF6iZu6Xv0k04b vTmK+8o2q/D1f7TAGVbdim1bIj0B4DNljspC3aewMIQ4+UoC3TVM/JOldv1tphvsPX88P38NUZCGG TJI26X9A==; Received: from [2001:4bb8:198:38af:e8dc:dbbd:a9d:5c54] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWuKa-006pZL-QP; Sat, 10 Sep 2022 06:51:09 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 1/5] mm: add PSI accounting around ->read_folio and ->readahead calls Date: Sat, 10 Sep 2022 08:50:54 +0200 Message-Id: <20220910065058.3303831-2-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220910065058.3303831-1-hch@lst.de> References: <20220910065058.3303831-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=NBIurit5; spf=none (imf29.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662792682; a=rsa-sha256; cv=none; b=WIFifeQgzgVq6zM7LGu9yoYEUhdO6nknsnoC1Dn7hte9H13P8VZdUpXmQyU69ydA8NTnMj oSFIO5A6m8GHbmJoa6+ubDR5GLHsEp6jBhNoFyovYbhwF5A/bsnRcN/n7FHclVAg9oVYz1 3Qqt2xaVPDUd1kILM61SulaAynEZx7w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662792682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TWMB057d75ixVxyjdIFV1XXGOo0lh1V+gLJp5z7ed9M=; b=AHU8PbmhdfXQewGAua2uDxn0q4aOqgFMBy9HYWUjYy6T7vS9zhgeGrhO0n5q1f2Z4po8ZN t72MFvvDiKKpioYBZ/tC6KTutgBigmDyklGxiPWYHDX0nfoIzeqSVjL5xCnDTN5Z3w33fk gSJEgtETKlHJJ4V+EoBRgRx3/mtYfGo= X-Stat-Signature: emdbdoc8apbg7t3p9n5ec14orf9gz5aa X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=NBIurit5; spf=none (imf29.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 668391200B9 X-HE-Tag: 1662792681-748360 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PSI tries to account for the cost of bringing back in pages discarded by the MM LRU management. Currently the prime place for that is hooked into the bio submission path, which is a rather bad place: - it does not actually account I/O for non-block file systems, of which we have many - it adds overhead and a layering violation to the block layer Add the accounting into the two places in the core MM code that read pages into an address space by calling into ->read_folio and ->readahead so that the entire file system operations are covered, to broaden the coverage and allow removing the accounting in the block layer going forward. As psi_memstall_enter can deal with nested calls this will not lead to double accounting even while the bio annotations are still present. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 7 +++++++ mm/readahead.c | 22 ++++++++++++++++++---- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0178b2040ea38..201dc7281640b 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1173,6 +1173,8 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool _workingset; + unsigned long _pflags; }; #define DEFINE_READAHEAD(ractl, f, r, m, i) \ diff --git a/mm/filemap.c b/mm/filemap.c index 15800334147b3..c943d1b90cc26 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2382,6 +2382,8 @@ static void filemap_get_read_batch(struct address_space *mapping, static int filemap_read_folio(struct file *file, filler_t filler, struct folio *folio) { + bool workingset = folio_test_workingset(folio); + unsigned long pflags; int error; /* @@ -2390,8 +2392,13 @@ static int filemap_read_folio(struct file *file, filler_t filler, * fails. */ folio_clear_error(folio); + /* Start the actual read. The read will unlock the page. */ + if (unlikely(workingset)) + psi_memstall_enter(&pflags); error = filler(file, folio); + if (unlikely(workingset)) + psi_memstall_leave(&pflags); if (error) return error; diff --git a/mm/readahead.c b/mm/readahead.c index fdcd28cbd92de..43631c146d6dc 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -122,6 +122,7 @@ #include #include #include +#include #include #include #include @@ -152,6 +153,8 @@ static void read_pages(struct readahead_control *rac) if (!readahead_count(rac)) return; + if (unlikely(rac->_workingset)) + psi_memstall_enter(&rac->_pflags); blk_start_plug(&plug); if (aops->readahead) { @@ -179,6 +182,9 @@ static void read_pages(struct readahead_control *rac) } blk_finish_plug(&plug); + if (unlikely(rac->_workingset)) + psi_memstall_leave(&rac->_pflags); + rac->_workingset = false; BUG_ON(readahead_count(rac)); } @@ -252,6 +258,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, } if (i == nr_to_read - lookahead_size) folio_set_readahead(folio); + ractl->_workingset |= folio_test_workingset(folio); ractl->_nr_pages++; } @@ -480,11 +487,14 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, if (index == mark) folio_set_readahead(folio); err = filemap_add_folio(ractl->mapping, folio, index, gfp); - if (err) + if (err) { folio_put(folio); - else - ractl->_nr_pages += 1UL << order; - return err; + return err; + } + + ractl->_nr_pages += 1UL << order; + ractl->_workingset = folio_test_workingset(folio); + return 0; } void page_cache_ra_order(struct readahead_control *ractl, @@ -826,6 +836,10 @@ void readahead_expand(struct readahead_control *ractl, put_page(page); return; } + if (unlikely(PageWorkingset(page)) && !ractl->_workingset) { + ractl->_workingset = true; + psi_memstall_enter(&ractl->_pflags); + } ractl->_nr_pages++; if (ra) { ra->size++; From patchwork Sat Sep 10 06:50:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12972429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01BA7ECAAD5 for ; Sat, 10 Sep 2022 06:51:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 289F06B0072; Sat, 10 Sep 2022 02:51:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C5636B0074; Sat, 10 Sep 2022 02:51:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDE878D0002; Sat, 10 Sep 2022 02:51:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A34D56B0073 for ; Sat, 10 Sep 2022 02:51:22 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6F18240312 for ; Sat, 10 Sep 2022 06:51:22 +0000 (UTC) X-FDA: 79895254404.04.2A02483 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf10.hostedemail.com (Postfix) with ESMTP id 13834C0079 for ; Sat, 10 Sep 2022 06:51:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=vqMGMaEkOmTqVQylBL+skkFNuiQaEqBjU4MG7KR5LL0=; b=kqYZ5RijV8voK/z0zN+91658ME othEptJ5Jh2KM6dR4Cxso2p8jnG+xQo2M5kHEwh9mo1XVCJT3+LjcQVT86P0gUbIOZqNeFapGyX/h /OdgGVYp+QTVSkxLppYxTFUKdb59n7fqclYWiXunNSvOEQZbHZ7Al4fPyqE+Y7niQJCNcFr1QpNzt E9JUwcNBCafe3zyNEyvt4LB8sGEahpq/lVI09ZRV8hg7amkemarEy1CYLP/bzYggDRP43uf/Xx2Fa L7KwLUlqXcGXEbb1eNQP//XtM7trmDf+dq+zs0ZJyirR+O7PS1uicM9oKe1DTjiDKpApviawkwvvL FHmOX0DA==; Received: from [2001:4bb8:198:38af:e8dc:dbbd:a9d:5c54] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWuKd-006pZX-Fw; Sat, 10 Sep 2022 06:51:12 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 2/5] sched/psi: export psi_memstall_{enter,leave} Date: Sat, 10 Sep 2022 08:50:55 +0200 Message-Id: <20220910065058.3303831-3-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220910065058.3303831-1-hch@lst.de> References: <20220910065058.3303831-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662792682; a=rsa-sha256; cv=none; b=795eDhVfWKKeGECVjmiEjCffapvq1TeoBji/gb7uSvcS7CeKECvRXQ99bLNyvqRTISUwgR AVvjUPLLhtl61iDV5d2CsLkD6vMRBn3cs5YuO27bllEU2Iwdn+jXxi4ATAgPaP/3UuWu2x 8v9xd1ZvHMcz3WLfXIcQrBCMMSuUUIw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=kqYZ5Rij; spf=none (imf10.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662792682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vqMGMaEkOmTqVQylBL+skkFNuiQaEqBjU4MG7KR5LL0=; b=4Wdd2Ft7u+OQgL/ZxucxG6oyenvSYw6xOpoguW26sqnAOvUg5FeX6QRZb0124LHorDKe/1 02z1+LAFBxxkYSHH2m1Q8prWRTzHsrFtpUpU45wX6yKt0jYPTy4FXhr/nlUvDXMmKtrj8c Cuv37t/9SwkweetzPuloq5XKTJP85Oc= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=kqYZ5Rij; spf=none (imf10.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none X-Rspam-User: X-Stat-Signature: 99ka8mfpdc73y4qkojnrasdjodncb3ig X-Rspamd-Queue-Id: 13834C0079 X-Rspamd-Server: rspam08 X-HE-Tag: 1662792681-863876 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To properly account for all refaults from file system logic, file systems need to call psi_memstall_enter directly, so export it. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner --- kernel/sched/psi.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c index ecb4b4ff4ce0a..7f6030091aeee 100644 --- a/kernel/sched/psi.c +++ b/kernel/sched/psi.c @@ -917,6 +917,7 @@ void psi_memstall_enter(unsigned long *flags) rq_unlock_irq(rq, &rf); } +EXPORT_SYMBOL_GPL(psi_memstall_enter); /** * psi_memstall_leave - mark the end of an memory stall section @@ -946,6 +947,7 @@ void psi_memstall_leave(unsigned long *flags) rq_unlock_irq(rq, &rf); } +EXPORT_SYMBOL_GPL(psi_memstall_leave); #ifdef CONFIG_CGROUPS int psi_cgroup_alloc(struct cgroup *cgroup) From patchwork Sat Sep 10 06:50:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12972430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F222EC6FA82 for ; Sat, 10 Sep 2022 06:51:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FAAD6B0074; Sat, 10 Sep 2022 02:51:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AC7E6B0075; Sat, 10 Sep 2022 02:51:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E19AA8D0001; Sat, 10 Sep 2022 02:51:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C65DE6B0074 for ; Sat, 10 Sep 2022 02:51:24 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9607B1407AA for ; Sat, 10 Sep 2022 06:51:24 +0000 (UTC) X-FDA: 79895254488.31.FB95A0A Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf30.hostedemail.com (Postfix) with ESMTP id 482AF800AA for ; Sat, 10 Sep 2022 06:51:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=FkPl9IvUjfBsIN7AIjdm08mTjlqewl1g1KHKU4sEYn8=; b=RMyNmneg26qCu35hD5+W/DxVr4 iVPficcPRLAokYM8ZwsoBLv8dW8VnZmJoK+GDv3k99nhtWxs6z+2MP6KXvBvQGDZX3E3IOlhYWYih 1wSKkmK6byR4m9gCQAvdCO4LFG0XyTL/Wg4/XQ/NxIibvPoIxqr+YNihM/P/pvyv3RLQssprSncca iEiwC1YXUzHQ2sBq+jikNtoTM9Asnl3xpUT3rE6pcjSKj787bsdsV1ETRfYPz/KIjQc2g5iacXJE3 +wyvgt21XtR4GFAfm29jnH92HvRDkK2397b1S1gSfn/wfDIHA8M7HhR6zgyZJzSMSIqFggodqjXIG lsWWrGew==; Received: from [2001:4bb8:198:38af:e8dc:dbbd:a9d:5c54] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWuKg-006paD-79; Sat, 10 Sep 2022 06:51:14 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 3/5] btrfs: add manual PSI accounting for compressed reads Date: Sat, 10 Sep 2022 08:50:56 +0200 Message-Id: <20220910065058.3303831-4-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220910065058.3303831-1-hch@lst.de> References: <20220910065058.3303831-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662792684; a=rsa-sha256; cv=none; b=74dkKQB80RrkePAhDn6hs4aSk+qUe9Xn3Lnyk1pGcKm7pSGinHGxE9rRJFldNrBciXkPNt XzGDjFNMsYegV87MhEninebE/VhRIRna5CVbrDvlvYjyg5Kf54IV6evpQnS5kw7e0MSPod 7MrOY85AvUD5FBeoHdi0z920yNoWbsY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=RMyNmneg; dmarc=none; spf=none (imf30.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662792684; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FkPl9IvUjfBsIN7AIjdm08mTjlqewl1g1KHKU4sEYn8=; b=52NNPBNx7JyYoBphJXkO/wmINUpNsEovsrtXRO9z/0MbgpTH3FcwAI3Ppc3b8c01smO3p7 0R3c+O8qYJXoRuCOs/53v6w/n5vdPSNw+jr18A7emloODUns7XucBRrIEv/33PoNwrDsQZ u5TPmZrTk6adPtkNZ8J45kjtDrAS5Qc= Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=RMyNmneg; dmarc=none; spf=none (imf30.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org X-Rspam-User: X-Rspamd-Queue-Id: 482AF800AA X-Rspamd-Server: rspam11 X-Stat-Signature: u734j784ti9cc7jfcb1puamrz3q1im8d X-HE-Tag: 1662792684-819278 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: btrfs compressed reads try to always read the entire compressed chunk, even if only a subset is requested. Currently this is covered by the magic PSI accounting underneath submit_bio, but that is about to go away. Instead add manual psi_memstall_{enter,leave} annotations. Note that for readahead this really should be using readahead_expand, but the additionals reads are also done for plain ->read_folio where readahead_expand can't work, so this overall logic is left as-is for now. Signed-off-by: Christoph Hellwig Acked-by: David Sterba Acked-by: Johannes Weiner --- fs/btrfs/compression.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index e84d22c5c6a83..f7889a00e0055 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -519,7 +520,8 @@ static u64 bio_end_offset(struct bio *bio) */ static noinline int add_ra_bio_pages(struct inode *inode, u64 compressed_end, - struct compressed_bio *cb) + struct compressed_bio *cb, + unsigned long *pflags) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); unsigned long end_index; @@ -588,6 +590,9 @@ static noinline int add_ra_bio_pages(struct inode *inode, continue; } + if (unlikely(PageWorkingset(page))) + psi_memstall_enter(pflags); + ret = set_page_extent_mapped(page); if (ret < 0) { unlock_page(page); @@ -674,6 +679,8 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, u64 em_len; u64 em_start; struct extent_map *em; + /* initialize to 1 to make skip psi_memstall_leave unless needed */ + unsigned long pflags = 1; blk_status_t ret; int ret2; int i; @@ -729,7 +736,7 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, goto fail; } - add_ra_bio_pages(inode, em_start + em_len, cb); + add_ra_bio_pages(inode, em_start + em_len, cb, &pflags); /* include any pages we added in add_ra-bio_pages */ cb->len = bio->bi_iter.bi_size; @@ -810,6 +817,9 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, } } + if (!pflags) + psi_memstall_leave(&pflags); + if (refcount_dec_and_test(&cb->pending_ios)) finish_compressed_bio_read(cb); return; From patchwork Sat Sep 10 06:50:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12972431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F06A4ECAAD5 for ; Sat, 10 Sep 2022 06:51:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2D396B0075; Sat, 10 Sep 2022 02:51:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDC898D0001; Sat, 10 Sep 2022 02:51:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D542F6B007B; Sat, 10 Sep 2022 02:51:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C3B6A6B0075 for ; Sat, 10 Sep 2022 02:51:27 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 99B44AAE23 for ; Sat, 10 Sep 2022 06:51:27 +0000 (UTC) X-FDA: 79895254614.14.174A6AB Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf17.hostedemail.com (Postfix) with ESMTP id 5403940086 for ; Sat, 10 Sep 2022 06:51:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=fQqaIwOPxSuMna00dQRnvLHVvXRo+zAlkdWprGAd+Cc=; b=TY6vqRLMJhvweO+6nWY/Yk6QCH 2wbbFyPkYsHGjWKYBe7GnKXWBeP7BegsJdll9FnrtOthJp4dUf1Kvsl5a/qlVyFhFhLHEPwEE0WQL wDaXd5YpJFiR0RumNIFaRd1Po+fn9GcxeyCCNfN1fJY7mmjO9rTpn8caq4TXFBBayzRcOc6bu4NKh IoDNIXl5IexFES5+qwaLTG71wyLwN1NbONFzYKyCJUeec2R61kCjzTUcXjZLj0pLvBFd+WW7ty19m JKcjT5Dzix0LV4x4Q6R9PV7MhsSEKNMJR6LZPTnwjwtyUHob7eNK6sy3hKBwQOGcSJCPj2NHLWdC9 4OgUvzQQ==; Received: from [2001:4bb8:198:38af:e8dc:dbbd:a9d:5c54] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWuKj-006pb8-02; Sat, 10 Sep 2022 06:51:17 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 4/5] erofs: add manual PSI accounting for the compressed address space Date: Sat, 10 Sep 2022 08:50:57 +0200 Message-Id: <20220910065058.3303831-5-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220910065058.3303831-1-hch@lst.de> References: <20220910065058.3303831-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662792687; a=rsa-sha256; cv=none; b=qtIdOSX3GvjPng7xit1YXTunx5twe5XqdvoNu/ulUF0CQ8vn0/YlEZfOsIi3mQyhbq44W3 WdePWxFiI8ce+QaNNUishX8RpExbZRYQ/ouOdcNpCozD2kz+g/1WxjU/L5gcUXNC7EHN17 jlV8FOezqQ3hRUhcKcJI6BGDkQpp6dM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=TY6vqRLM; dmarc=none; spf=none (imf17.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662792687; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fQqaIwOPxSuMna00dQRnvLHVvXRo+zAlkdWprGAd+Cc=; b=Ptrqd7rSh8KP+DiMHmk6izL5wiWznPS/HOy/it7MZcSXyn1L1Ljh3379hXBT3/izqe24Fe XOxBC9lai978eexWLRPTPR49w/5Lowl3UI43lVl7Vm8ZRVTwW+EDPYa73wRCz/QSGiFMgM xMBzQfxsjFqsGKjGyPQ6y8Ch9aGFjBY= X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 5403940086 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=TY6vqRLM; dmarc=none; spf=none (imf17.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org X-Stat-Signature: w64h3893boq4y73moio68fnk8n9j9ssc X-Rspam-User: X-HE-Tag: 1662792687-620130 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: erofs uses an additional address space for compressed data read from disk in addition to the one directly associated with the inode. Reading into the lower address space is open coded using add_to_page_cache_lru instead of using the filemap.c helper for page allocation micro-optimizations, which means it is not covered by the MM PSI annotations for ->read_folio and ->readahead, so add manual ones instead. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner Acked-by: Gao Xiang --- fs/erofs/zdata.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index 5792ca9e0d5ef..143a101a36887 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -7,6 +7,7 @@ #include "zdata.h" #include "compress.h" #include +#include #include @@ -1365,6 +1366,8 @@ static void z_erofs_submit_queue(struct z_erofs_decompress_frontend *f, struct block_device *last_bdev; unsigned int nr_bios = 0; struct bio *bio = NULL; + /* initialize to 1 to make skip psi_memstall_leave unless needed */ + unsigned long pflags = 1; bi_private = jobqueueset_init(sb, q, fgq, force_fg); qtail[JQ_BYPASS] = &q[JQ_BYPASS]->head; @@ -1414,10 +1417,15 @@ static void z_erofs_submit_queue(struct z_erofs_decompress_frontend *f, if (bio && (cur != last_index + 1 || last_bdev != mdev.m_bdev)) { submit_bio_retry: + if (!pflags) + psi_memstall_leave(&pflags); submit_bio(bio); bio = NULL; } + if (unlikely(PageWorkingset(page))) + psi_memstall_enter(&pflags); + if (!bio) { bio = bio_alloc(mdev.m_bdev, BIO_MAX_VECS, REQ_OP_READ, GFP_NOIO); @@ -1445,8 +1453,11 @@ static void z_erofs_submit_queue(struct z_erofs_decompress_frontend *f, move_to_bypass_jobqueue(pcl, qtail, owned_head); } while (owned_head != Z_EROFS_PCLUSTER_TAIL); - if (bio) + if (bio) { + if (!pflags) + psi_memstall_leave(&pflags); submit_bio(bio); + } /* * although background is preferred, no one is pending for submission. From patchwork Sat Sep 10 06:50:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12972432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A298ECAAD5 for ; Sat, 10 Sep 2022 06:51:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A725B6B0078; Sat, 10 Sep 2022 02:51:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A20136B007B; Sat, 10 Sep 2022 02:51:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E7826B007D; Sat, 10 Sep 2022 02:51:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7E3626B0078 for ; Sat, 10 Sep 2022 02:51:34 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5AAB21C5F79 for ; Sat, 10 Sep 2022 06:51:34 +0000 (UTC) X-FDA: 79895254908.10.E4155CA Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf18.hostedemail.com (Postfix) with ESMTP id 1D5611C007D for ; Sat, 10 Sep 2022 06:51:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=Bj6Wxth2LNlOrb9ptrXcpFWJGYrnuyA2Icp7ZXfeW/o=; b=K2RnBhW7QUIkDosGrl4GPsn3WL puVVLQPIHR+oiYIc0dCiAWg+BoaPF8fvXmI4e2zGOB0616LPS5EAr/7le5HmA7pYGWsCs5nftgX/M 2hkMrWbjbXBoEZ479tAk2Vo+2eP6ql/aFbRT/g5HHwiGsclbr/+8TZzCo2c78a9DVQWiXf62EGI/c achcb3IFQdWr2jdFOKpg/Yi7UDa3pHtmwSLvTJmAsfwkCPyr5BdhV7dmrIvtZL4aEQHUAVVIVRAS2 bYBTfNySyorNvWtavgLy5TTGayzG7SaZUQo1VrHGyISynz0lpOmzunL//qKfUZN3qe1lO5DVavmbc 5CYi4Y4g==; Received: from [2001:4bb8:198:38af:e8dc:dbbd:a9d:5c54] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oWuKl-006pbo-MI; Sat, 10 Sep 2022 06:51:20 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 5/5] block: remove PSI accounting from the bio layer Date: Sat, 10 Sep 2022 08:50:58 +0200 Message-Id: <20220910065058.3303831-6-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220910065058.3303831-1-hch@lst.de> References: <20220910065058.3303831-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=K2RnBhW7; spf=none (imf18.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662792694; a=rsa-sha256; cv=none; b=m7ZisbK2i71Csbq3GsOHKNR6Y9YyZ4HsYQkQtJAB+20kYpy8wms8zy+TiLODzg+vyxKRgO qi0looGHr46VRwUq7wR2fvO7UbXsDavZCuS6M0XSnKotBk1Kj4hklM75jsP6DJ2vcFvKDP lu4QreZtZNlOlKAso+6jdJTbUKantlU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662792694; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bj6Wxth2LNlOrb9ptrXcpFWJGYrnuyA2Icp7ZXfeW/o=; b=2ZEeRfvt6q/L2sKusdm0EgPKxA1KXKTXoLDaGrGyN/djuA8hNK9BmNpT9dPbeDDodIg2Si d0QbvAsd1d8gsH12deCeMnaAiO9TkSDRSWirwPxxwazM6ILb7osAs2KhI1SRfHuFdN+eo7 FTL03yNifh8ekrVUbr4F412L2kBgKyQ= X-Stat-Signature: otqs7ixf1pkobm3cbp1webns3s55qu9m X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=K2RnBhW7; spf=none (imf18.hostedemail.com: domain of BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+74593d5eb0e0b46c37a4+6957+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1D5611C007D X-HE-Tag: 1662792693-733317 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PSI accounting is now done by the VM code, where it should have been since the beginning. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner --- block/bio.c | 8 -------- block/blk-core.c | 17 ----------------- fs/direct-io.c | 2 -- include/linux/blk_types.h | 1 - 4 files changed, 28 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3d3a2678fea25..d10c4e888cdcf 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1065,9 +1065,6 @@ void __bio_add_page(struct bio *bio, struct page *page, bio->bi_iter.bi_size += len; bio->bi_vcnt++; - - if (!bio_flagged(bio, BIO_WORKINGSET) && unlikely(PageWorkingset(page))) - bio_set_flag(bio, BIO_WORKINGSET); } EXPORT_SYMBOL_GPL(__bio_add_page); @@ -1276,9 +1273,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * fit into the bio, or are requested in @iter, whatever is smaller. If * MM encounters an error pinning the requested pages, it stops. Error * is returned only if 0 pages could be pinned. - * - * It's intended for direct IO, so doesn't do PSI tracking, the caller is - * responsible for setting BIO_WORKINGSET if necessary. */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1294,8 +1288,6 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); - /* don't account direct I/O as memory stall */ - bio_clear_flag(bio, BIO_WORKINGSET); return bio->bi_vcnt ? 0 : ret; } EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); diff --git a/block/blk-core.c b/block/blk-core.c index a0d1104c5590c..9e19195af6f5b 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include @@ -829,22 +828,6 @@ void submit_bio(struct bio *bio) count_vm_events(PGPGOUT, bio_sectors(bio)); } - /* - * If we're reading data that is part of the userspace workingset, count - * submission time as memory stall. When the device is congested, or - * the submitting cgroup IO-throttled, submission can be a significant - * part of overall IO time. - */ - if (unlikely(bio_op(bio) == REQ_OP_READ && - bio_flagged(bio, BIO_WORKINGSET))) { - unsigned long pflags; - - psi_memstall_enter(&pflags); - submit_bio_noacct(bio); - psi_memstall_leave(&pflags); - return; - } - submit_bio_noacct(bio); } EXPORT_SYMBOL(submit_bio); diff --git a/fs/direct-io.c b/fs/direct-io.c index f669163d5860f..03d381377ae10 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -421,8 +421,6 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) unsigned long flags; bio->bi_private = dio; - /* don't account direct I/O as memory stall */ - bio_clear_flag(bio, BIO_WORKINGSET); spin_lock_irqsave(&dio->bio_lock, flags); dio->refcount++; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 1ef99790f6ed3..8b1858df21752 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -321,7 +321,6 @@ enum { BIO_NO_PAGE_REF, /* don't put release vec pages */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ - BIO_WORKINGSET, /* contains userspace workingset pages */ BIO_QUIET, /* Make BIO Quiet */ BIO_CHAIN, /* chained bio, ->bi_remaining in effect */ BIO_REFFED, /* bio has elevated ->bi_cnt */