From patchwork Mon Jul 1 07:51:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 13717642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C38FC3065B for ; Mon, 1 Jul 2024 07:53:14 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-3.v29.lw.sourceforge.com) by sfs-ml-3.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1sOBqc-00049A-7g; Mon, 01 Jul 2024 07:53:13 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-3.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1sOBqb-000494-0N for linux-f2fs-devel@lists.sourceforge.net; Mon, 01 Jul 2024 07:53:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Type:Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=qMppuDI1wnvEIsKQBj9vDotqx3IBKuxAM1cbtDDXvG0=; b=JXSKd0Kh6mbbD7aEnX82Qg937n 0/ChV/z/neAv0YemjdG4/fBN1BPoxMSq5WQF9yAX9ZUgwdrarxPIoKnHqRJhtORn5CV3dtwPhFFAI sm1xjh8AAb2lK9UQyxPPLRu5mkePcyktsbHUyM4nA+ogjxx/kxWLkgDxS9q+cZxiTRxA=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Type:Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=qMppuDI1wnvEIsKQBj9vDotqx3IBKuxAM1cbtDDXvG0=; b=UBPGflBmeI9qdddqEtcwHOh8pF FySa+VHAUFR8RB88S6AaQHI1IPuf7cdlN5vLrIgmMLRBcg7ZrAv82VrnSbVpS6hlVZVZnRiwTERNZ 5xVC7DmzpyBA3+K1JUmjBmpUqWznzCMOl1Hmjxn1US7FKBPspZZpIfQPosYJhsgamc4g=; Received: from mx1.unisoc.com ([222.66.158.135] helo=SHSQR01.spreadtrum.com) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1sOBqZ-0006v4-Rj for linux-f2fs-devel@lists.sourceforge.net; Mon, 01 Jul 2024 07:53:12 +0000 Received: from dlp.unisoc.com ([10.29.3.86]) by SHSQR01.spreadtrum.com with ESMTP id 4617pill064601; Mon, 1 Jul 2024 15:51:44 +0800 (+08) (envelope-from Yi.Sun@unisoc.com) Received: from SHDLP.spreadtrum.com (bjmbx02.spreadtrum.com [10.0.64.8]) by dlp.unisoc.com (SkyGuard) with ESMTPS id 4WCJ5l68MFz2KQ9Qx; Mon, 1 Jul 2024 15:46:51 +0800 (CST) Received: from tj10379pcu.spreadtrum.com (10.5.32.15) by BJMBX02.spreadtrum.com (10.0.64.8) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Mon, 1 Jul 2024 15:51:41 +0800 From: Yi Sun To: , , , , Date: Mon, 1 Jul 2024 15:51:37 +0800 Message-ID: <20240701075138.1144575-2-yi.sun@unisoc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240701075138.1144575-1-yi.sun@unisoc.com> References: <20240701075138.1144575-1-yi.sun@unisoc.com> MIME-Version: 1.0 X-Originating-IP: [10.5.32.15] X-ClientProxiedBy: SHCAS03.spreadtrum.com (10.0.1.207) To BJMBX02.spreadtrum.com (10.0.64.8) X-MAIL: SHSQR01.spreadtrum.com 4617pill064601 X-Headers-End: 1sOBqZ-0006v4-Rj Subject: [f2fs-dev] [PATCH v2 1/2] workqueue: new struct io_work X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yunlongxing23@gmail.com, ebiggers@google.com, kent.overstreet@linux.dev, linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, yi.sun@unisoc.com, Hao_hao.Wang@unisoc.com Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net Many works will go to submit_bio(), and in many cases the io priority of kworker cannot meet the real-time requirements of this work. So create a new struct io_work, which contains the io priority that the kworker thread can adjust its own io priority according to. And, new function set_io_work_ioprio() to set the io priority of io work, new function may_adjust_io_work_task_ioprio() to adjust kworker's io priority, new function restore_io_work_task_ioprio() to restore kworker's io priority. Signed-off-by: Yi Sun --- include/linux/workqueue.h | 47 ++++++++++++++++++++++++++++++++++++ kernel/workqueue.c | 50 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index d9968bfc8eac..4b2cb54a68b2 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -127,6 +127,21 @@ struct rcu_work { struct workqueue_struct *wq; }; +/* + * If a work may do disk IO, it is recommended to use struct io_work + * instead of struct work_struct. + */ +struct io_work { + struct work_struct work; + + /* If the work does submit_bio, io priority may be needed. */ + unsigned short ioprio; + /* Record kworker's original io priority. */ + unsigned short ori_ioprio; + /* Whether the work has set io priority? */ + long ioprio_flag; +}; + enum wq_affn_scope { WQ_AFFN_DFL, /* use system default */ WQ_AFFN_CPU, /* one pod per CPU */ @@ -218,6 +233,11 @@ static inline struct rcu_work *to_rcu_work(struct work_struct *work) return container_of(work, struct rcu_work, work); } +static inline struct io_work *to_io_work(struct work_struct *work) +{ + return container_of(work, struct io_work, work); +} + struct execute_work { struct work_struct work; }; @@ -347,6 +367,18 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; } #define INIT_RCU_WORK_ONSTACK(_work, _func) \ INIT_WORK_ONSTACK(&(_work)->work, (_func)) +#define INIT_IO_WORK(_work, _func) \ + do { \ + INIT_WORK(&(_work)->work, (_func)); \ + (_work)->ioprio_flag = 0; \ + } while (0) + +#define INIT_IO_WORK_ONSTACK(_work, _func) \ + do { \ + INIT_WORK_ONSTACK(&(_work)->work, (_func)); \ + (_work)->ioprio_flag = 0; \ + } while (0) + /** * work_pending - Find out whether a work item is currently pending * @work: The work item in question @@ -552,6 +584,10 @@ extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq, struct delayed_work *dwork, unsigned long delay); extern bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork); +extern void set_io_work_ioprio(struct io_work *work, unsigned short ioprio); +extern void may_adjust_io_work_task_ioprio(struct io_work *work); +extern void restore_io_work_task_ioprio(struct io_work *work); + extern void __flush_workqueue(struct workqueue_struct *wq); extern void drain_workqueue(struct workqueue_struct *wq); @@ -636,6 +672,17 @@ static inline bool queue_delayed_work(struct workqueue_struct *wq, return queue_delayed_work_on(WORK_CPU_UNBOUND, wq, dwork, delay); } +/** + * queue_io_work - queue io work on a workqueue + * @wq: workqueue to use + * @iowork: io work to queue + */ +static inline bool queue_io_work(struct workqueue_struct *wq, + struct io_work *iowork) +{ + return queue_work(wq, &(iowork->work)); +} + /** * mod_delayed_work - modify delay of or queue a delayed work * @wq: workqueue to use diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 3fbaecfc88c2..a55b74d5f560 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -2652,6 +2652,56 @@ bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork) } EXPORT_SYMBOL(queue_rcu_work); +/** + * set_io_work_ioprio - set io priority for the current io work + * @iowork: the io work to be set + * @ioprio: desired io priority + * + * This function can be called after INIT_IO_WORK if the io priority + * of the io work needs to adjust. And it is recommended to use this + * function together with may_adjust_io_work_task_ioprio() and + * restore_io_work_task_ioprio(). + */ +void set_io_work_ioprio(struct io_work *iowork, unsigned short ioprio) +{ + iowork->ioprio = ioprio; + iowork->ioprio_flag = 1; +} +EXPORT_SYMBOL(set_io_work_ioprio); + +/** + * may_adjust_io_work_task_ioprio - maybe adjust the io priority of kworker + * @iowork: the io work that kworker will do + * + * It is recommended to use this function together with set_io_work_ioprio() + * and restore_io_work_task_ioprio(). + */ +void may_adjust_io_work_task_ioprio(struct io_work *iowork) +{ + if (iowork->ioprio_flag) { + iowork->ori_ioprio = get_current_ioprio(); + set_task_ioprio(current, iowork->ioprio); + } +} +EXPORT_SYMBOL(may_adjust_io_work_task_ioprio); + +/** + * restore_io_work_task_ioprio - restore the io priority of kworker + * @iowork: the io work that kworker just did + * + * When kworker finishes the io work, the original io priority of + * kworker should be restored. It is recommended to use this function + * together with set_io_work_ioprio() and may_adjust_io_work_task_ioprio(). + */ +void restore_io_work_task_ioprio(struct io_work *iowork) +{ + if (iowork->ioprio_flag) { + set_task_ioprio(current, iowork->ori_ioprio); + iowork->ioprio_flag = 0; + } +} +EXPORT_SYMBOL(restore_io_work_task_ioprio); + static struct worker *alloc_worker(int node) { struct worker *worker; From patchwork Mon Jul 1 07:51:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Sun X-Patchwork-Id: 13717643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3742C2BD09 for ; Mon, 1 Jul 2024 07:53:16 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-4.v29.lw.sourceforge.com) by sfs-ml-4.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1sOBqd-0004oi-JL; Mon, 01 Jul 2024 07:53:16 +0000 Received: from [172.30.29.66] (helo=mx.sourceforge.net) by sfs-ml-4.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1sOBqb-0004oc-65 for linux-f2fs-devel@lists.sourceforge.net; Mon, 01 Jul 2024 07:53:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Type:Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=iIHmEFoKcSBN0mKbLF7zuE3vtHJ+me9Lo0+lTC/F6kM=; b=TVqqpyI4xjJeyp2JnJ+Dv8xpYk 3RGANieZxb1yoIafU71auhnMdHBBomNthdF0iK5gh3y/bqxPVod0oWz9aZX5RdfMi760+V3UHIKR2 ei0N96lk4qM4kaP7Ae3oEs0Vn07v8oQjYk9o8G71BwhNaJzauhxAhJsn1sCQfNxUzkQM=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Type:Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=iIHmEFoKcSBN0mKbLF7zuE3vtHJ+me9Lo0+lTC/F6kM=; b=WhyvbdF6mtX8umeJQu9TB3OgVG vPCu3+yMiS7ETjL+T8AmZpRseTgMbRH91ojczWRgjMBbSV9bdcuLSERkyjiM0DsnZ75yXSAh9ptCy /eAdqbO3jP5Rg/Rnqd8gcl0enjYhrkNG9JVIQgVZeKiZO0eJ9FDaodVfMeX9vHGnOSQc=; Received: from mx1.unisoc.com ([222.66.158.135] helo=SHSQR01.spreadtrum.com) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1sOBqb-0006v6-3v for linux-f2fs-devel@lists.sourceforge.net; Mon, 01 Jul 2024 07:53:13 +0000 Received: from dlp.unisoc.com ([10.29.3.86]) by SHSQR01.spreadtrum.com with ESMTP id 4617pjuA064749; Mon, 1 Jul 2024 15:51:45 +0800 (+08) (envelope-from Yi.Sun@unisoc.com) Received: from SHDLP.spreadtrum.com (bjmbx02.spreadtrum.com [10.0.64.8]) by dlp.unisoc.com (SkyGuard) with ESMTPS id 4WCJ5n0qV1z2KKCbl; Mon, 1 Jul 2024 15:46:53 +0800 (CST) Received: from tj10379pcu.spreadtrum.com (10.5.32.15) by BJMBX02.spreadtrum.com (10.0.64.8) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Mon, 1 Jul 2024 15:51:42 +0800 From: Yi Sun To: , , , , Date: Mon, 1 Jul 2024 15:51:38 +0800 Message-ID: <20240701075138.1144575-3-yi.sun@unisoc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240701075138.1144575-1-yi.sun@unisoc.com> References: <20240701075138.1144575-1-yi.sun@unisoc.com> MIME-Version: 1.0 X-Originating-IP: [10.5.32.15] X-ClientProxiedBy: SHCAS03.spreadtrum.com (10.0.1.207) To BJMBX02.spreadtrum.com (10.0.64.8) X-MAIL: SHSQR01.spreadtrum.com 4617pjuA064749 X-Headers-End: 1sOBqb-0006v6-3v Subject: [f2fs-dev] [PATCH v2 2/2] f2fs: set io priority for fsverity work X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yunlongxing23@gmail.com, ebiggers@google.com, kent.overstreet@linux.dev, linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, yi.sun@unisoc.com, Hao_hao.Wang@unisoc.com Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net The io priority of reading the original data of the verity-file and reading hash blocks during verification should be consistent. If the io priority is not set, when a high-task-priority process reads the verity-file, the kworker's io priority may be too low, resulting in delayed reading of hash blocks, which eventually causes the high-task-priority process to response slowly. Signed-off-by: Yi Sun --- fs/f2fs/data.c | 17 ++++++++++------- fs/f2fs/f2fs.h | 1 + fs/f2fs/verity.c | 5 +++++ 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index b9b0debc6b3d..7e8c56a3098b 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -110,7 +110,7 @@ enum bio_post_read_step { struct bio_post_read_ctx { struct bio *bio; struct f2fs_sb_info *sbi; - struct work_struct work; + struct io_work iowork; unsigned int enabled_steps; /* * decompression_attempted keeps track of whether @@ -168,10 +168,12 @@ static void f2fs_finish_read_bio(struct bio *bio, bool in_task) static void f2fs_verify_bio(struct work_struct *work) { struct bio_post_read_ctx *ctx = - container_of(work, struct bio_post_read_ctx, work); + container_of(to_io_work(work), struct bio_post_read_ctx, iowork); struct bio *bio = ctx->bio; bool may_have_compressed_pages = (ctx->enabled_steps & STEP_DECOMPRESS); + may_adjust_io_work_task_ioprio(to_io_work(work)); + /* * fsverity_verify_bio() may call readahead() again, and while verity * will be disabled for this, decryption and/or decompression may still @@ -204,6 +206,7 @@ static void f2fs_verify_bio(struct work_struct *work) } f2fs_finish_read_bio(bio, true); + restore_io_work_task_ioprio(to_io_work(work)); } /* @@ -220,8 +223,8 @@ static void f2fs_verify_and_finish_bio(struct bio *bio, bool in_task) struct bio_post_read_ctx *ctx = bio->bi_private; if (ctx && (ctx->enabled_steps & STEP_VERITY)) { - INIT_WORK(&ctx->work, f2fs_verify_bio); - fsverity_enqueue_verify_work(&ctx->work); + INIT_IO_WORK(&ctx->iowork, f2fs_verify_bio); + f2fs_enqueue_verify_io_work(&ctx->iowork); } else { f2fs_finish_read_bio(bio, in_task); } @@ -270,7 +273,7 @@ static void f2fs_handle_step_decompress(struct bio_post_read_ctx *ctx, static void f2fs_post_read_work(struct work_struct *work) { struct bio_post_read_ctx *ctx = - container_of(work, struct bio_post_read_ctx, work); + container_of(to_io_work(work), struct bio_post_read_ctx, iowork); struct bio *bio = ctx->bio; if ((ctx->enabled_steps & STEP_DECRYPT) && !fscrypt_decrypt_bio(bio)) { @@ -313,8 +316,8 @@ static void f2fs_read_end_io(struct bio *bio) !f2fs_low_mem_mode(sbi)) { f2fs_handle_step_decompress(ctx, intask); } else if (enabled_steps) { - INIT_WORK(&ctx->work, f2fs_post_read_work); - queue_work(ctx->sbi->post_read_wq, &ctx->work); + INIT_IO_WORK(&ctx->iowork, f2fs_post_read_work); + queue_io_work(ctx->sbi->post_read_wq, &ctx->iowork); return; } } diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 1974b6aff397..dc228c50ed30 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -4234,6 +4234,7 @@ void f2fs_unregister_sysfs(struct f2fs_sb_info *sbi); /* verity.c */ extern const struct fsverity_operations f2fs_verityops; +void f2fs_enqueue_verify_io_work(struct io_work *iowork); /* * crypto support diff --git a/fs/f2fs/verity.c b/fs/f2fs/verity.c index f7bb0c54502c..fba6aecba9b6 100644 --- a/fs/f2fs/verity.c +++ b/fs/f2fs/verity.c @@ -292,3 +292,8 @@ const struct fsverity_operations f2fs_verityops = { .read_merkle_tree_page = f2fs_read_merkle_tree_page, .write_merkle_tree_block = f2fs_write_merkle_tree_block, }; + +void f2fs_enqueue_verify_io_work(struct io_work *iowork) +{ + fsverity_enqueue_verify_work(&(iowork->work)); +}