From patchwork Thu Sep 15 09:41:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12977126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0E16C6FA8B for ; Thu, 15 Sep 2022 09:42:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 365026B0080; Thu, 15 Sep 2022 05:42:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 22AC36B007D; Thu, 15 Sep 2022 05:42:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E85918D0001; Thu, 15 Sep 2022 05:42:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C7F766B0078 for ; Thu, 15 Sep 2022 05:42:24 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 725F940DB7 for ; Thu, 15 Sep 2022 09:42:24 +0000 (UTC) X-FDA: 79913829408.06.66F8367 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf31.hostedemail.com (Postfix) with ESMTP id 3297B200C2 for ; Thu, 15 Sep 2022 09:42:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=SxtVDqdat7BUx/W51wS73TW46FWCaaqoJViREBV0E4g=; b=XCKKlba980yHrEb7U2Z7nxXZcq YZzQTA/3rSfLmxP/itoKa1cbxHecjo6lZ+/daLAoQ54ZywDoNmcYhRKQh7BhjSSHYgAkEROhNV0/F 14GG/i0DGrSzBpEno94+6Ri22JjcTLo6AeQn1wLhshEvNQWMAE80uVSy7lfx0Ic4vGDqbQuJxd+jz TiTrR0NISz7uCi8jJpu5X0J1hYZD+scAZRZY54MvRA9w34a9r6/My0FLliJ93QY1RS4/0GnzTnRsB BSei3Ct+cvW3zZBM7lACxj9tSoD9XtuX3iLfVJvhdqhe9e4BywOrgFoGIoW0PNpYxnkBWmNGmZv+S fkJwuS3g==; Received: from [185.122.133.20] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oYlNk-005b0u-Ql; Thu, 15 Sep 2022 09:42:05 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 1/5] mm: add PSI accounting around ->read_folio and ->readahead calls Date: Thu, 15 Sep 2022 10:41:56 +0100 Message-Id: <20220915094200.139713-2-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220915094200.139713-1-hch@lst.de> References: <20220915094200.139713-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=XCKKlba9; spf=none (imf31.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663234944; a=rsa-sha256; cv=none; b=axONjOr/Mf3t8sXSlHmcWuLkICQKTdmznPFWrVoE6g7C626fgvlCi1GtynEIHYwCu6d9wv 5du1ur8TbDIJyfdgEi2sj+XmaEzCrLxNmZyVEcxlwpRWSfDrJ2W9ZyVMU3INj81FCyio6v ykQsVa9EK6sUH47dDhKGvDT+4H3FjMk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663234944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SxtVDqdat7BUx/W51wS73TW46FWCaaqoJViREBV0E4g=; b=G1CmyYFzqe/3ju1NFhRQTWun5tcgIBS2qkH7VMDAak+72Nv0YWSFoqbglNWpXyNu/ey6Kf 3vzJccTijXVZ8ebUyf8934y2Pev5TieDyi+zC7ytkA3Wu267jQK8WozbtibSBs5QihLe9e KcANzl8IBeWsRgOXi7XgBli5GuVEW/I= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3297B200C2 X-Rspam-User: Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=XCKKlba9; spf=none (imf31.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none X-Stat-Signature: 6jzjb647efnxwx58gt3ff7yhmteb84tm X-HE-Tag: 1663234944-87901 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PSI tries to account for the cost of bringing back in pages discarded by the MM LRU management. Currently the prime place for that is hooked into the bio submission path, which is a rather bad place: - it does not actually account I/O for non-block file systems, of which we have many - it adds overhead and a layering violation to the block layer Add the accounting into the two places in the core MM code that read pages into an address space by calling into ->read_folio and ->readahead so that the entire file system operations are covered, to broaden the coverage and allow removing the accounting in the block layer going forward. As psi_memstall_enter can deal with nested calls this will not lead to double accounting even while the bio annotations are still present. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 7 +++++++ mm/readahead.c | 22 ++++++++++++++++++---- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0178b2040ea38..201dc7281640b 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1173,6 +1173,8 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool _workingset; + unsigned long _pflags; }; #define DEFINE_READAHEAD(ractl, f, r, m, i) \ diff --git a/mm/filemap.c b/mm/filemap.c index 15800334147b3..c943d1b90cc26 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2382,6 +2382,8 @@ static void filemap_get_read_batch(struct address_space *mapping, static int filemap_read_folio(struct file *file, filler_t filler, struct folio *folio) { + bool workingset = folio_test_workingset(folio); + unsigned long pflags; int error; /* @@ -2390,8 +2392,13 @@ static int filemap_read_folio(struct file *file, filler_t filler, * fails. */ folio_clear_error(folio); + /* Start the actual read. The read will unlock the page. */ + if (unlikely(workingset)) + psi_memstall_enter(&pflags); error = filler(file, folio); + if (unlikely(workingset)) + psi_memstall_leave(&pflags); if (error) return error; diff --git a/mm/readahead.c b/mm/readahead.c index fdcd28cbd92de..b10f0cf81d804 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -122,6 +122,7 @@ #include #include #include +#include #include #include #include @@ -152,6 +153,8 @@ static void read_pages(struct readahead_control *rac) if (!readahead_count(rac)) return; + if (unlikely(rac->_workingset)) + psi_memstall_enter(&rac->_pflags); blk_start_plug(&plug); if (aops->readahead) { @@ -179,6 +182,9 @@ static void read_pages(struct readahead_control *rac) } blk_finish_plug(&plug); + if (unlikely(rac->_workingset)) + psi_memstall_leave(&rac->_pflags); + rac->_workingset = false; BUG_ON(readahead_count(rac)); } @@ -252,6 +258,7 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, } if (i == nr_to_read - lookahead_size) folio_set_readahead(folio); + ractl->_workingset |= folio_test_workingset(folio); ractl->_nr_pages++; } @@ -480,11 +487,14 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, if (index == mark) folio_set_readahead(folio); err = filemap_add_folio(ractl->mapping, folio, index, gfp); - if (err) + if (err) { folio_put(folio); - else - ractl->_nr_pages += 1UL << order; - return err; + return err; + } + + ractl->_nr_pages += 1UL << order; + ractl->_workingset |= folio_test_workingset(folio); + return 0; } void page_cache_ra_order(struct readahead_control *ractl, @@ -826,6 +836,10 @@ void readahead_expand(struct readahead_control *ractl, put_page(page); return; } + if (unlikely(PageWorkingset(page)) && !ractl->_workingset) { + ractl->_workingset = true; + psi_memstall_enter(&ractl->_pflags); + } ractl->_nr_pages++; if (ra) { ra->size++; From patchwork Thu Sep 15 09:41:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12977123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE836ECAAA1 for ; Thu, 15 Sep 2022 09:42:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B00C6B0071; Thu, 15 Sep 2022 05:42:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 338F56B0073; Thu, 15 Sep 2022 05:42:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DA628D0001; Thu, 15 Sep 2022 05:42:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0F70C6B0071 for ; Thu, 15 Sep 2022 05:42:21 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E6B7F14081B for ; Thu, 15 Sep 2022 09:42:20 +0000 (UTC) X-FDA: 79913829240.20.F9E663F Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf23.hostedemail.com (Postfix) with ESMTP id BEFE41400BF for ; Thu, 15 Sep 2022 09:42:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=f10ud+4Yk8zuOd0FwaSc+xsLEXHksvzyDOLFsDza1gY=; b=Z0mbKLDUJODd7WIJH7zNtS2L90 vOmMj4hDPPXiSDw+aMBTpUxgy0asgtgfV+4+J+l4oS+R2K/YUyuJ8I/elEQRBtiYerT7+cF5+veE0 As2MLe9Jh+crJYKoXZY8Yx6HDQSgstxsDLr8ptgQnaOoxvx+vyNis0c4HBZ1yJWMJml91LbKDJ6sT o8ABQhBEkp2KFWBYACzfFvirFth7WnnmAhhowLIfySIykeZwWp4JCLieTfsniUw4AZwmamm7MT3nC xxaJOz0cCYfhy3q/EURHz/sxkq0xwdjKLYezjVO8xCGkodJzvbpVbeUUBWYy02JEErH3z9SzjWEdA jKe0ukpg==; Received: from [185.122.133.20] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oYlNn-005b1k-MR; Thu, 15 Sep 2022 09:42:08 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 2/5] sched/psi: export psi_memstall_{enter,leave} Date: Thu, 15 Sep 2022 10:41:57 +0100 Message-Id: <20220915094200.139713-3-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220915094200.139713-1-hch@lst.de> References: <20220915094200.139713-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=Z0mbKLDU; spf=none (imf23.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663234940; a=rsa-sha256; cv=none; b=bOxybFSA/wRdiD8OjQuKeoPxdoC73JDfi7ufMAUtTmX56XyadZAZjE5rO04f2BSAg1pOur Bs90Yh5TybLyoh/gMHkPChIQn5qLQPsRpWnV9mrhjHSTd7jYouLKmck9IGpxpgSFZ2c9He C9AwaAPv1HDQ+uOH6RzQFTXMDIEp9vA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663234940; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f10ud+4Yk8zuOd0FwaSc+xsLEXHksvzyDOLFsDza1gY=; b=YmtNvwAM3TPnQ4RqJ7gVqpRA8tK8+fw9LUD3MW6jC12kZ19svdSZnaloE0R1DAXLqULmaN C/jnXri5BviGIGWIG/n3bStjq8h5YpJd8OLZnI1EeHlHD9KQQeSf19rnQBDRaOeN/K1jww bmiSl08EcTHpoOe6C5tpdJrkYcrjatc= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BEFE41400BF X-Rspam-User: Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=Z0mbKLDU; spf=none (imf23.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none X-Stat-Signature: ome6yes7sdkm1c6p8w5mpdqe1b7we3bq X-HE-Tag: 1663234939-266566 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To properly account for all refaults from file system logic, file systems need to call psi_memstall_enter directly, so export it. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner --- kernel/sched/psi.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c index ecb4b4ff4ce0a..7f6030091aeee 100644 --- a/kernel/sched/psi.c +++ b/kernel/sched/psi.c @@ -917,6 +917,7 @@ void psi_memstall_enter(unsigned long *flags) rq_unlock_irq(rq, &rf); } +EXPORT_SYMBOL_GPL(psi_memstall_enter); /** * psi_memstall_leave - mark the end of an memory stall section @@ -946,6 +947,7 @@ void psi_memstall_leave(unsigned long *flags) rq_unlock_irq(rq, &rf); } +EXPORT_SYMBOL_GPL(psi_memstall_leave); #ifdef CONFIG_CGROUPS int psi_cgroup_alloc(struct cgroup *cgroup) From patchwork Thu Sep 15 09:41:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12977124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E451ECAAD3 for ; Thu, 15 Sep 2022 09:42:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E9666B0074; Thu, 15 Sep 2022 05:42:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 471896B0075; Thu, 15 Sep 2022 05:42:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 277D48D0001; Thu, 15 Sep 2022 05:42:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 028726B0075 for ; Thu, 15 Sep 2022 05:42:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D0FA6404DA for ; Thu, 15 Sep 2022 09:42:21 +0000 (UTC) X-FDA: 79913829282.07.0BB5DD0 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf26.hostedemail.com (Postfix) with ESMTP id 88EA51400CE for ; Thu, 15 Sep 2022 09:42:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=Sn1STluEcmAEl2IDkRqQTtdfy5beUs4WRE2QQKAsD/8=; b=os6+qAq+Yh06rcP0f4OouU/22o oWMxav2lmp8ydZQlV6kYJHiVg8ZVYPX5Z5bUp7SoaxqFj/nUS4D885nff+ExvyS+/ZeTUGV4jsgEU ER77a8FARvGVxYQc08pU28vamvOLA93UJlOYB+wMD6hJYfaFlwU7gVGEBvA4QBbVOudz8qGi3zl6j PD4HWCk7kfMEuJEharZdxwNNq3YjLexhD9ZuJU9XTK/fP5VUj1cnMn1ePeBY44Kxrp22zlpCAneVa pYT72nSZOveSvkojw81g64FzZrU9pT28Z+dU81UCAkWPRoOJZqNkQwmhvF/DotA52n+uG+9hsXvG2 598sGqsQ==; Received: from [185.122.133.20] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oYlNq-005b3C-78; Thu, 15 Sep 2022 09:42:10 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 3/5] btrfs: add manual PSI accounting for compressed reads Date: Thu, 15 Sep 2022 10:41:58 +0100 Message-Id: <20220915094200.139713-4-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220915094200.139713-1-hch@lst.de> References: <20220915094200.139713-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663234941; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Sn1STluEcmAEl2IDkRqQTtdfy5beUs4WRE2QQKAsD/8=; b=jLYAFpWyXISkfBPIXeIpRZQ2IZ1VnOx40LAUDozJTBEb9jO1nWrJF1OzQaiNXDOLZktOp8 TYf/W1rz4pifVNK8CxRsY6ITvdSNKEP017peQ3642SdZjE5HGu0XG5p2erlYQ2+Zj3+6Nt R/Sr3pV94otLxCRNzoho2WHSXv3GDBY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=os6+qAq+; dmarc=none; spf=none (imf26.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663234941; a=rsa-sha256; cv=none; b=EYVRoYtfrgxBG10s1TBUwMWXgiTQTd0RfdemIoVVRRkKtM7jcySgc4U5KxMK5xobvsWMtV etJYIH8Z4wjAerXvFtf2hvcfkhpWUmEETAR0E3G2fFq1F2D28TIBMDIbeg9k+z9s1cKhYk SCqkIv1vdHmhWMYeyZGUnqsZaG3a6iQ= X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 88EA51400CE X-Rspam-User: X-Stat-Signature: gnhokmf9u7kxpg5rqfm8e6xqbycxmmm3 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=os6+qAq+; dmarc=none; spf=none (imf26.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org X-HE-Tag: 1663234941-784155 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: btrfs compressed reads try to always read the entire compressed chunk, even if only a subset is requested. Currently this is covered by the magic PSI accounting underneath submit_bio, but that is about to go away. Instead add manual psi_memstall_{enter,leave} annotations. Note that for readahead this really should be using readahead_expand, but the additionals reads are also done for plain ->read_folio where readahead_expand can't work, so this overall logic is left as-is for now. Signed-off-by: Christoph Hellwig Acked-by: David Sterba Acked-by: Johannes Weiner Reported-by: Thorsten Leemhuis Signed-off-by: Johannes Weiner Tested-by: Thorsten Leemhuis Reported-by: Thorsten Leemhuis Signed-off-by: Johannes Weiner Tested-by: Thorsten Leemhuis --- fs/btrfs/compression.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index e84d22c5c6a83..370788b9b1249 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -519,7 +520,8 @@ static u64 bio_end_offset(struct bio *bio) */ static noinline int add_ra_bio_pages(struct inode *inode, u64 compressed_end, - struct compressed_bio *cb) + struct compressed_bio *cb, + unsigned long *pflags) { struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); unsigned long end_index; @@ -588,6 +590,9 @@ static noinline int add_ra_bio_pages(struct inode *inode, continue; } + if (PageWorkingset(page)) + psi_memstall_enter(pflags); + ret = set_page_extent_mapped(page); if (ret < 0) { unlock_page(page); @@ -674,6 +679,8 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, u64 em_len; u64 em_start; struct extent_map *em; + /* Initialize to 1 to make skip psi_memstall_leave unless needed */ + unsigned long pflags = 1; blk_status_t ret; int ret2; int i; @@ -729,7 +736,7 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, goto fail; } - add_ra_bio_pages(inode, em_start + em_len, cb); + add_ra_bio_pages(inode, em_start + em_len, cb, &pflags); /* include any pages we added in add_ra-bio_pages */ cb->len = bio->bi_iter.bi_size; @@ -810,6 +817,9 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, } } + if (!pflags) + psi_memstall_leave(&pflags); + if (refcount_dec_and_test(&cb->pending_ios)) finish_compressed_bio_read(cb); return; From patchwork Thu Sep 15 09:41:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12977125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63CB6C6FA8A for ; Thu, 15 Sep 2022 09:42:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F23146B0075; Thu, 15 Sep 2022 05:42:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAB1B6B0078; Thu, 15 Sep 2022 05:42:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDACC8D0001; Thu, 15 Sep 2022 05:42:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A7A826B0075 for ; Thu, 15 Sep 2022 05:42:23 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 87C8B40C82 for ; Thu, 15 Sep 2022 09:42:23 +0000 (UTC) X-FDA: 79913829366.30.96FAA2A Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf14.hostedemail.com (Postfix) with ESMTP id 473A11000B3 for ; Thu, 15 Sep 2022 09:42:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=zSqQSYiG/yhGt5jVAkpLYmREyG39C2nKNIUNc2bjsak=; b=LBYTaNgK69SCGNd0NypzW88Xv5 Yx58YrbfYXYM3VU1TTaXmnHJhmAosi9GxG5fHizuaiSpsfQyyTrQq+gWKQKGswDMr3IXN1ZrWzMVP UY1tiT8IDnZgbhBOAQ3xCJ+2VZ7AX9J7984+cTjW67hOwi9a/fxdM4uq9b3MQocffDqEAIF78Qo78 gqCr/pS8eAu+ePAIAL48Nsgbsq1v9+2L+IYizWE+7uXKkal5cK3KhOBwcusamcfxoDVeuG3s8KEDX vjUdj6mQwI2K2tNhBci5SxLXAld8gZc6Vz+gdOMjtgn/GsCCBLZKbXJ5lYzK1oAc2wfdMZD/5hqfX L5xq1muA==; Received: from [185.122.133.20] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oYlNs-005b4u-GP; Thu, 15 Sep 2022 09:42:13 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org, Gao Xiang Subject: [PATCH 4/5] erofs: add manual PSI accounting for the compressed address space Date: Thu, 15 Sep 2022 10:41:59 +0100 Message-Id: <20220915094200.139713-5-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220915094200.139713-1-hch@lst.de> References: <20220915094200.139713-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663234943; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zSqQSYiG/yhGt5jVAkpLYmREyG39C2nKNIUNc2bjsak=; b=R1G0/ChSUKsRa5I/eKideityIxLhYQadL36mGFQtxiB4xTFCsvY360ftUwgBla8vM2zvHI KUu7imo1woJ/6yWw9V0Ff5G05OMM68arHYaekXNV1l/HVpBrlPnJNdGOlTUVuQ5zMH2XL1 6oVgR38V6lsWqzLiSaxPoPTziA7SPtY= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=LBYTaNgK; dmarc=none; spf=none (imf14.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663234943; a=rsa-sha256; cv=none; b=naKDRsMwwM5lRqg9GZaES4KTJK0SBuZ3AiimT/xM69f6k+gvVUzKZwE1LnjYSg2IVi194h gSw6+X1EOJT3H66CGi/3vpUJV22XtfNPWGtLaoMq8Wz8pdtyujqFk7WLJoGbwRlp9k8TJH K5ZyihFL6Tn3J4JSz2gcpifXiFdLKFU= X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 473A11000B3 X-Rspam-User: X-Stat-Signature: acd6u3z5qagued67rckitwsj5ndjds8j Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=LBYTaNgK; dmarc=none; spf=none (imf14.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org X-HE-Tag: 1663234943-540660 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: erofs uses an additional address space for compressed data read from disk in addition to the one directly associated with the inode. Reading into the lower address space is open coded using add_to_page_cache_lru instead of using the filemap.c helper for page allocation micro-optimizations, which means it is not covered by the MM PSI annotations for ->read_folio and ->readahead, so add manual ones instead. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner Acked-by: Gao Xiang --- fs/erofs/zdata.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index 5792ca9e0d5ef..143a101a36887 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -7,6 +7,7 @@ #include "zdata.h" #include "compress.h" #include +#include #include @@ -1365,6 +1366,8 @@ static void z_erofs_submit_queue(struct z_erofs_decompress_frontend *f, struct block_device *last_bdev; unsigned int nr_bios = 0; struct bio *bio = NULL; + /* initialize to 1 to make skip psi_memstall_leave unless needed */ + unsigned long pflags = 1; bi_private = jobqueueset_init(sb, q, fgq, force_fg); qtail[JQ_BYPASS] = &q[JQ_BYPASS]->head; @@ -1414,10 +1417,15 @@ static void z_erofs_submit_queue(struct z_erofs_decompress_frontend *f, if (bio && (cur != last_index + 1 || last_bdev != mdev.m_bdev)) { submit_bio_retry: + if (!pflags) + psi_memstall_leave(&pflags); submit_bio(bio); bio = NULL; } + if (unlikely(PageWorkingset(page))) + psi_memstall_enter(&pflags); + if (!bio) { bio = bio_alloc(mdev.m_bdev, BIO_MAX_VECS, REQ_OP_READ, GFP_NOIO); @@ -1445,8 +1453,11 @@ static void z_erofs_submit_queue(struct z_erofs_decompress_frontend *f, move_to_bypass_jobqueue(pcl, qtail, owned_head); } while (owned_head != Z_EROFS_PCLUSTER_TAIL); - if (bio) + if (bio) { + if (!pflags) + psi_memstall_leave(&pflags); submit_bio(bio); + } /* * although background is preferred, no one is pending for submission. From patchwork Thu Sep 15 09:42:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12977127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54888C6FA86 for ; Thu, 15 Sep 2022 09:42:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 539CA6B0078; Thu, 15 Sep 2022 05:42:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3610B6B007B; Thu, 15 Sep 2022 05:42:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F8286B0078; Thu, 15 Sep 2022 05:42:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D9CDD6B007B for ; Thu, 15 Sep 2022 05:42:24 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B9D77120CE8 for ; Thu, 15 Sep 2022 09:42:24 +0000 (UTC) X-FDA: 79913829408.25.5E7E629 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf05.hostedemail.com (Postfix) with ESMTP id 6D2BA1000B3 for ; Thu, 15 Sep 2022 09:42:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=PzlrQ8oKvfWNjDBiduvx+RzNOBIgqE2Y/9GDFrAa5zA=; b=bXeNhFFDCpfkRWHXmWzdHwlB5U ebzGk7+4exPACeou4fI+Sm6JQFfbX33mPhr7e+OY/vsBOl+7KB4pjS4bZIZmQBKspIrcU6Do7JYgB H66JJ5LEpcNzjlgpmde8OXP0YN93isg92RFIGpYP4eIsiAM2zsRgJ64ui9RH8ZXVhVfDu0YBFF3LE M5dml3Id3wks6W5Ui9PYxUv0W0G0mv5pIbLUdioFpLJQdNk+eUZyJq95PBmrV2Ra3D8ir74eOeDfy DZdcZFhb46yvO66ykAJ0TjIXLo+uKG2Nxo+zFLhSQr8Qeiaf7+ctmRj1h1E2s4mnsAnzj/LV9UNQv 0kRBcInw==; Received: from [185.122.133.20] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oYlNu-005b8U-Qp; Thu, 15 Sep 2022 09:42:15 +0000 From: Christoph Hellwig To: Jens Axboe , Matthew Wilcox , Johannes Weiner , Suren Baghdasaryan , Andrew Morton Cc: Chris Mason , Josef Bacik , David Sterba , Gao Xiang , Chao Yu , linux-block@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-erofs@lists.ozlabs.org, linux-mm@kvack.org Subject: [PATCH 5/5] block: remove PSI accounting from the bio layer Date: Thu, 15 Sep 2022 10:42:00 +0100 Message-Id: <20220915094200.139713-6-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220915094200.139713-1-hch@lst.de> References: <20220915094200.139713-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663234944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PzlrQ8oKvfWNjDBiduvx+RzNOBIgqE2Y/9GDFrAa5zA=; b=TNeJThbDxNtzbHGw3TJuzcmA0tb49sUlNfv/5nMDRY799z2Ol8QW4Uh+3V6hMT+0wW82Ob rPELzs0mQnnTBr7JNJpWIQl8brIkq8+3ceqtIvL699bXmQB1Da0qgUN6/Jl+TxfR43haPF ejVfVXlenavrf/Zt8GZN1umiDtRS7yw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=bXeNhFFD; dmarc=none; spf=none (imf05.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663234944; a=rsa-sha256; cv=none; b=CfJONRaHQik5VwbvHH6C1CTtfakehJMmzkp4NY5UVvfq28tnXBJ0z50rh2bAPAC7NzaYz0 GFA3n8ZnQYk4gJAHDgel5Y+i2wnVEx7paWEzq9d0dAZrpT9SaSso2+XQRVW6FFC2RxPiWS T7Uuu6l7Y2wE9PYLQVJMbPRnRLfsG+U= X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6D2BA1000B3 X-Rspam-User: X-Stat-Signature: gmqs1wwtk84rtfpfay7u8geeps1ty6ut Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=bXeNhFFD; dmarc=none; spf=none (imf05.hostedemail.com: domain of BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+aa90abf7a61f323a8d2f+6962+infradead.org+hch@bombadil.srs.infradead.org X-HE-Tag: 1663234944-16612 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PSI accounting is now done by the VM code, where it should have been since the beginning. Signed-off-by: Christoph Hellwig Acked-by: Johannes Weiner --- block/bio.c | 8 -------- block/blk-core.c | 17 ----------------- fs/direct-io.c | 2 -- include/linux/blk_types.h | 1 - 4 files changed, 28 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3d3a2678fea25..d10c4e888cdcf 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1065,9 +1065,6 @@ void __bio_add_page(struct bio *bio, struct page *page, bio->bi_iter.bi_size += len; bio->bi_vcnt++; - - if (!bio_flagged(bio, BIO_WORKINGSET) && unlikely(PageWorkingset(page))) - bio_set_flag(bio, BIO_WORKINGSET); } EXPORT_SYMBOL_GPL(__bio_add_page); @@ -1276,9 +1273,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * fit into the bio, or are requested in @iter, whatever is smaller. If * MM encounters an error pinning the requested pages, it stops. Error * is returned only if 0 pages could be pinned. - * - * It's intended for direct IO, so doesn't do PSI tracking, the caller is - * responsible for setting BIO_WORKINGSET if necessary. */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1294,8 +1288,6 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); - /* don't account direct I/O as memory stall */ - bio_clear_flag(bio, BIO_WORKINGSET); return bio->bi_vcnt ? 0 : ret; } EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); diff --git a/block/blk-core.c b/block/blk-core.c index a0d1104c5590c..9e19195af6f5b 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include @@ -829,22 +828,6 @@ void submit_bio(struct bio *bio) count_vm_events(PGPGOUT, bio_sectors(bio)); } - /* - * If we're reading data that is part of the userspace workingset, count - * submission time as memory stall. When the device is congested, or - * the submitting cgroup IO-throttled, submission can be a significant - * part of overall IO time. - */ - if (unlikely(bio_op(bio) == REQ_OP_READ && - bio_flagged(bio, BIO_WORKINGSET))) { - unsigned long pflags; - - psi_memstall_enter(&pflags); - submit_bio_noacct(bio); - psi_memstall_leave(&pflags); - return; - } - submit_bio_noacct(bio); } EXPORT_SYMBOL(submit_bio); diff --git a/fs/direct-io.c b/fs/direct-io.c index f669163d5860f..03d381377ae10 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -421,8 +421,6 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) unsigned long flags; bio->bi_private = dio; - /* don't account direct I/O as memory stall */ - bio_clear_flag(bio, BIO_WORKINGSET); spin_lock_irqsave(&dio->bio_lock, flags); dio->refcount++; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 1ef99790f6ed3..8b1858df21752 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -321,7 +321,6 @@ enum { BIO_NO_PAGE_REF, /* don't put release vec pages */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ - BIO_WORKINGSET, /* contains userspace workingset pages */ BIO_QUIET, /* Make BIO Quiet */ BIO_CHAIN, /* chained bio, ->bi_remaining in effect */ BIO_REFFED, /* bio has elevated ->bi_cnt */