From patchwork Tue Dec 15 00:20:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11973403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BF68C2BBCA for ; Tue, 15 Dec 2020 00:25:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E7DC522258 for ; Tue, 15 Dec 2020 00:25:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728021AbgLOAY6 (ORCPT ); Mon, 14 Dec 2020 19:24:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725976AbgLOAYp (ORCPT ); Mon, 14 Dec 2020 19:24:45 -0500 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D32DC06179C; Mon, 14 Dec 2020 16:24:05 -0800 (PST) Received: by mail-wr1-x441.google.com with SMTP id c5so14509321wrp.6; Mon, 14 Dec 2020 16:24:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LOsfOtb4JF9iqyizeOjm4tVRPPo2GoTdvRn37AEB3XM=; b=kG976mlGUAT54hMTsT4qHl3ePNNRWB9LFy2HmmL+t6mfRSKdk5umZ6WYomCULDQW9E YuWby+j3W4gWivHpVQQOA/7aIwmkSSLngHeQdb6klQaCptqFG/YV5ejnTwIwlz9HFf2m eDUwq7Rwt+h66HV9wnAS1sesppf+Fpl+3qYXxpDR3ZoWJ27/ZiY4pBb1I/35pIXw34kB Lbi1nxTdy0WE+OGKwC0NKqucdKqyJIKuQSjNNW99pLcDkW/m1CtBI/gMiqbPxZ8kxwzH aobpyU6IKfyxV7UT1aOSZWNXNJ0RQdHNyYaHbm5kYkm08wWJNvMk2ZyFR9EbrghRw0A9 lUHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LOsfOtb4JF9iqyizeOjm4tVRPPo2GoTdvRn37AEB3XM=; b=LyijsfZas5MN+fpGuiUTO2FGEottJJAESBvtic/a74zNus+hT0RmsfyFSUK/1AXLtt BwgEPKOIqMUawCsaUeJe51wCCKbEmIToRqcdSeH2QD1JSSYG5p/YBvOxfjeSECtGAXq2 gaBQBFsBPryLWtXPTZAWW9n8eWtbZhDL/LkhpxirHRb5ht4e6rkIoXbqkS3yA8YQjJdd bipuX84mikhLIsUMWpzLhPqdcBsjnQgpRyTwvGIdst0+EEX6rIR3gT1e9LBoNpQi/Hts PnMYShvisPWzWcwRMaO2qozvZkWs5mgJ0KetUkMRSjTv5yD8I8wYmLbSgDK1+fwYNfrs R8Kw== X-Gm-Message-State: AOAM5314kP+8hivUKxxpB6HXY4NsKLXshXfQw7XVXM4vuxIrcGrtOATG Y7GlEtTcTvO6jv7s3urnRORnRks6gjzeGgEL X-Google-Smtp-Source: ABdhPJz/AygAoS74uaniu7lJWQGPmncFabpUEeaVcLaoDdNn4iTUPpmUbiF5aWlFvAES8KDtC5RTLg== X-Received: by 2002:a5d:4c4e:: with SMTP id n14mr31759423wrt.209.1607991843559; Mon, 14 Dec 2020 16:24:03 -0800 (PST) Received: from localhost.localdomain ([85.255.232.163]) by smtp.gmail.com with ESMTPSA id b19sm5362012wmj.37.2020.12.14.16.24.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 16:24:03 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v1 1/6] target/file: allocate the bvec array as part of struct target_core_file_cmd Date: Tue, 15 Dec 2020 00:20:20 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org From: Christoph Hellwig This saves one memory allocation, and ensures the bvecs aren't freed before the AIO completion. This will allow the lower level code to be optimized so that it can avoid allocating another bvec array. Signed-off-by: Christoph Hellwig Signed-off-by: Pavel Begunkov --- drivers/target/target_core_file.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/drivers/target/target_core_file.c b/drivers/target/target_core_file.c index b0cb5b95e892..cce455929778 100644 --- a/drivers/target/target_core_file.c +++ b/drivers/target/target_core_file.c @@ -241,6 +241,7 @@ struct target_core_file_cmd { unsigned long len; struct se_cmd *cmd; struct kiocb iocb; + struct bio_vec bvecs[]; }; static void cmd_rw_aio_complete(struct kiocb *iocb, long ret, long ret2) @@ -268,29 +269,22 @@ fd_execute_rw_aio(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, struct target_core_file_cmd *aio_cmd; struct iov_iter iter = {}; struct scatterlist *sg; - struct bio_vec *bvec; ssize_t len = 0; int ret = 0, i; - aio_cmd = kmalloc(sizeof(struct target_core_file_cmd), GFP_KERNEL); + aio_cmd = kmalloc(struct_size(aio_cmd, bvecs, sgl_nents), GFP_KERNEL); if (!aio_cmd) return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; - bvec = kcalloc(sgl_nents, sizeof(struct bio_vec), GFP_KERNEL); - if (!bvec) { - kfree(aio_cmd); - return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; - } - for_each_sg(sgl, sg, sgl_nents, i) { - bvec[i].bv_page = sg_page(sg); - bvec[i].bv_len = sg->length; - bvec[i].bv_offset = sg->offset; + aio_cmd->bvecs[i].bv_page = sg_page(sg); + aio_cmd->bvecs[i].bv_len = sg->length; + aio_cmd->bvecs[i].bv_offset = sg->offset; len += sg->length; } - iov_iter_bvec(&iter, is_write, bvec, sgl_nents, len); + iov_iter_bvec(&iter, is_write, aio_cmd->bvecs, sgl_nents, len); aio_cmd->cmd = cmd; aio_cmd->len = len; @@ -307,8 +301,6 @@ fd_execute_rw_aio(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, else ret = call_read_iter(file, &aio_cmd->iocb, &iter); - kfree(bvec); - if (ret != -EIOCBQUEUED) cmd_rw_aio_complete(&aio_cmd->iocb, ret, 0); From patchwork Tue Dec 15 00:20:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11973407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4119DC2BBCF for ; Tue, 15 Dec 2020 00:25:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1DDEE22273 for ; Tue, 15 Dec 2020 00:25:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728333AbgLOAY6 (ORCPT ); Mon, 14 Dec 2020 19:24:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727514AbgLOAYq (ORCPT ); Mon, 14 Dec 2020 19:24:46 -0500 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74062C0617A6; Mon, 14 Dec 2020 16:24:06 -0800 (PST) Received: by mail-wm1-x344.google.com with SMTP id g25so9987084wmh.1; Mon, 14 Dec 2020 16:24:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D7uupKi2lTHg1nTREBYfC4U7HbsUj926sgl3xgaqTd0=; b=P81A9Xdxqm6t1KFSSYwxpIM9EzQWTzR8Kz1eIjcTpt9oDQteS1+WmwytiySg9Ve3ck NH3x0eCT2MFsM6i7mpTaRmDkUKmx5yR84cDkT/K6YRM1+dJDKNadBpjefTeOpL7Jq0sU hLnzYpj6MkeIUvLlzsAR9PAA75XqsccnOt5Wb44pkE1iqEyfYLJT7jvppOb0ArghCB3U aiiSZEELjay6jLHL3mD8WZJ+DeJ9kVoBdCzXJAJemKWaykjXM3T+6gNyZ4vQ7eIbTDsU Sd4LMtYWOt+LJEqXarqBcbVqn+H4L7PALmdqkkXdPbMC2NyrSHYlmYwwX0FoD6I8p81c 3blQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D7uupKi2lTHg1nTREBYfC4U7HbsUj926sgl3xgaqTd0=; b=Z/ayUguHDbfm8n3WQuLImH2uFxf5GburTC7IEyzsKXPgocNsvSznCrgs05gP3Yta1Z 1a5p8nlFzl8DEI7kMT7yX1+f1+pyCrhtRUId4GAVzqJsRUL4Iy44KJXhcZ8thfD9uxXt bKp8G6bSqsPNmD7AgbL7eNHshUygksaKs6kxmoQ/FRQDfWG7hmhGj4wj0cHg3IdkW94i XunxaqfAgb+qxqTjTg0zSeeGHXSfN0WaYn/0Ixf/sgiSQ6OR2V1m2APyZPFthkV4JYB+ gvvpol74yrBRNbJwvd/tk39MoGd3y6aJFaY41AhCi7ZDMkIwrIHoSAd6bqZdu2Ccz+JN kBKA== X-Gm-Message-State: AOAM531JhHgqEkxk5Ez+q+dIJB6eXoEtG2p+ugqOVu9NZlJS7tvnO27E w3h9XFr+4x6MxDJ6wgqBb3y+5XPtKQ40/rV5 X-Google-Smtp-Source: ABdhPJykC0JY5d7rIaOq+MAqm+QaT5pR/4R471Z55M1imOLkxialCK8BI686kFFIPWNpqaYfjLcRPQ== X-Received: by 2002:a1c:4c14:: with SMTP id z20mr30684129wmf.149.1607991845077; Mon, 14 Dec 2020 16:24:05 -0800 (PST) Received: from localhost.localdomain ([85.255.232.163]) by smtp.gmail.com with ESMTPSA id b19sm5362012wmj.37.2020.12.14.16.24.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 16:24:04 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v1 2/6] iov_iter: optimise bvec iov_iter_advance() Date: Tue, 15 Dec 2020 00:20:21 +0000 Message-Id: <5c9c22dbeecad883ca29b31896c262a8d2a77132.1607976425.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org iov_iter_advance() is heavily used, but implemented through generic iteration. As bvecs have a specifically crafted advance() function, i.e. bvec_iter_advance(), which is faster and slimmer, use it instead. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- lib/iov_iter.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1635111c5bd2..5b186dc2c9ea 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1067,6 +1067,21 @@ static void pipe_advance(struct iov_iter *i, size_t size) pipe_truncate(i); } +static void iov_iter_bvec_advance(struct iov_iter *i, size_t size) +{ + struct bvec_iter bi; + + bi.bi_size = i->count; + bi.bi_bvec_done = i->iov_offset; + bi.bi_idx = 0; + bvec_iter_advance(i->bvec, &bi, size); + + i->bvec += bi.bi_idx; + i->nr_segs -= bi.bi_idx; + i->count = bi.bi_size; + i->iov_offset = bi.bi_bvec_done; +} + void iov_iter_advance(struct iov_iter *i, size_t size) { if (unlikely(iov_iter_is_pipe(i))) { @@ -1077,6 +1092,10 @@ void iov_iter_advance(struct iov_iter *i, size_t size) i->count -= size; return; } + if (iov_iter_is_bvec(i)) { + iov_iter_bvec_advance(i, size); + return; + } iterate_and_advance(i, size, v, 0, 0, 0) } EXPORT_SYMBOL(iov_iter_advance); From patchwork Tue Dec 15 00:20:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11973405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A81AC2BBCD for ; Tue, 15 Dec 2020 00:25:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3E40F22209 for ; Tue, 15 Dec 2020 00:25:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728341AbgLOAZB (ORCPT ); Mon, 14 Dec 2020 19:25:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727535AbgLOAYs (ORCPT ); Mon, 14 Dec 2020 19:24:48 -0500 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07F1DC0617A7; Mon, 14 Dec 2020 16:24:08 -0800 (PST) Received: by mail-wr1-x443.google.com with SMTP id a12so18140478wrv.8; Mon, 14 Dec 2020 16:24:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jQVTvgGa1LjUWqvTxBIf73PGX5t0yWv/7jb/FOyHpnQ=; b=MNxOq+/uE7DIwyNRUYewGw+cHYV3pClHNKkFNyBIVdI3EmUW/iPsPfP7yyzvq8u6HW y+G5urnHOa4dHakH8fNwlven+K4tiRfbWXhhmxGgEeMzIgON/V/3KPvIIE1STFr4wlIj SgeoTLBn1XPnx5Gw54WO9Ail0Pbu+qn1ntWpDgf7sX2nl46zzOUPoASz3AKqbQTRe5C8 qjhO3fZhfFlR+QPZhkUFrJzZojsifXm8XUJO1MGsQth6ATu4tYHY/Ma1viQrwsp2EP4o 7pb0PcP510viQINAxb0x6SwFrTsMMbzg0xzT5YV55QJkl2tVrPkM8FYFuof/4cC/EixF gDtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jQVTvgGa1LjUWqvTxBIf73PGX5t0yWv/7jb/FOyHpnQ=; b=J7JY5rXW+LgkCyM9vmy8ceYE1Ie5zjb1QzLInwkdkQ+4mDT1CaZvWxLwroq6RuI+a4 0fKrMddE6xUScoIbOKsAi/tPVR9D+R5n9gRNSavJCKmr0pXJ6Ui1d1Zx3bn5PEd0zx7A BVlcoB7J/vPV9VJaDBK4rB6hbY3++rFzlD3iXlBekc/LZBMMd8YrAAJSFf6KDxF83L7w VEFOzsOS6QhT2/VJCtRXU/YZH78GPViVz+axdSWlYey6S2hsZfHIsDb+Iogeuo1dwpKI ddzzZ8tlQXp4O9j3eFJlG0WXyiit/4PHCYGmVjpzHBJuazbvVuOVcKaliluycZTkCLkZ sv5A== X-Gm-Message-State: AOAM5315zvBDRXh8mizkM+XJoT+Go0B8je2Qh7SnW8L9wzO1FOgzGUR1 eP08cRZFgrI2xnY1e7gA+7jghzjQQ6OB0/ZV X-Google-Smtp-Source: ABdhPJzkeBXbuB0e70wPSfv0YNEY+IY3u+FsW+X1sRWABcHyKLngnn7Dx2/0Lsq9Kl4ceWUMiJY3iQ== X-Received: by 2002:adf:f590:: with SMTP id f16mr194470wro.40.1607991846606; Mon, 14 Dec 2020 16:24:06 -0800 (PST) Received: from localhost.localdomain ([85.255.232.163]) by smtp.gmail.com with ESMTPSA id b19sm5362012wmj.37.2020.12.14.16.24.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 16:24:06 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v1 3/6] bio: deduplicate adding a page into bio Date: Tue, 15 Dec 2020 00:20:22 +0000 Message-Id: <189cae47946fa49318f85678def738d358e8298b.1607976425.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org Both bio_add_hw_page() mimics bio_add_page() and has a hand-coded version of appending a page into bio's bvec. DRY Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- block/bio.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/block/bio.c b/block/bio.c index 1f2cc1fbe283..4a8f77bb3956 100644 --- a/block/bio.c +++ b/block/bio.c @@ -734,6 +734,22 @@ const char *bio_devname(struct bio *bio, char *buf) } EXPORT_SYMBOL(bio_devname); +static void bio_add_page_noaccount(struct bio *bio, struct page *page, + unsigned int len, unsigned int off) +{ + struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt]; + + WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)); + WARN_ON_ONCE(bio_full(bio, len)); + + bv->bv_page = page; + bv->bv_offset = off; + bv->bv_len = len; + + bio->bi_iter.bi_size += len; + bio->bi_vcnt++; +} + static inline bool page_is_mergeable(const struct bio_vec *bv, struct page *page, unsigned int len, unsigned int off, bool *same_page) @@ -818,12 +834,7 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, if (bio->bi_vcnt >= queue_max_segments(q)) return 0; - bvec = &bio->bi_io_vec[bio->bi_vcnt]; - bvec->bv_page = page; - bvec->bv_len = len; - bvec->bv_offset = offset; - bio->bi_vcnt++; - bio->bi_iter.bi_size += len; + bio_add_page_noaccount(bio, page, len, offset); return len; } @@ -903,18 +914,7 @@ EXPORT_SYMBOL_GPL(__bio_try_merge_page); void __bio_add_page(struct bio *bio, struct page *page, unsigned int len, unsigned int off) { - struct bio_vec *bv = &bio->bi_io_vec[bio->bi_vcnt]; - - WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)); - WARN_ON_ONCE(bio_full(bio, len)); - - bv->bv_page = page; - bv->bv_offset = off; - bv->bv_len = len; - - bio->bi_iter.bi_size += len; - bio->bi_vcnt++; - + bio_add_page_noaccount(bio, page, len, off); if (!bio_flagged(bio, BIO_WORKINGSET) && unlikely(PageWorkingset(page))) bio_set_flag(bio, BIO_WORKINGSET); } From patchwork Tue Dec 15 00:20:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11973409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56CCEC2BB48 for ; Tue, 15 Dec 2020 00:25:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1E59A22258 for ; Tue, 15 Dec 2020 00:25:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728715AbgLOAZR (ORCPT ); Mon, 14 Dec 2020 19:25:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728437AbgLOAZI (ORCPT ); Mon, 14 Dec 2020 19:25:08 -0500 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B6BFC0617B0; Mon, 14 Dec 2020 16:24:09 -0800 (PST) Received: by mail-wr1-x443.google.com with SMTP id r7so18152809wrc.5; Mon, 14 Dec 2020 16:24:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KwoEZ6JyxQmbBCmxLJQ+KWQFWoJwlaOL+90wK7NFOco=; b=CCqnIhGkYp5ddpdcS1UPTb39Ra4D+5WscVMtb6oidexSCYeRHWEJdco70zMnNXbweP Pn9+E/zWBZSVxQulIBksAPt2eJsrV+hcX5QXQ5e/rNpsVTCUypnB4O6PqOB1cPyrdig5 Z2W2sZoDSXpf3t6IsKdOTrqvZNe7gphiU7OKr3DzkP3ai/bVVx+ROIKjzGv/MjUZreFw UJYaTjClZtkxL3Qk4unb4sOP/qIqrNNTCyOMK+oyZy/vMPEVYMwnvSidQGP0uwVdVXKT RUMiYdgenV+xw4kpOcRJAQoq5G7ku8XOkHkg7hXSJdQunmwauFc9IIpAaXUTAN1K0Asq asiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KwoEZ6JyxQmbBCmxLJQ+KWQFWoJwlaOL+90wK7NFOco=; b=C8fnFApzFssakdswfeTqlZYEDLAwW0mE3Cf7u83WCfUK6eyspdxWPpSaTNRo3UoGkX FeX1xUJSCQpDmh8LpaS1FU4VQ59cwXfamSZy1KiXVoOBeJJyVHwfaiT1qIeUC0PjVxxl QzpMzww5uSYPblj9pMsZJKTWsfk0uGE/yyqrnsgVt4tm3+ldHH7zV+1VCpNSJT2GiTTI Io937H2qY53E8qXBKNQ3GjG/alxwtF5gsMok9xxeZLQo5sFWMeR+v85jKfHfs+GNJlaw kDOGaKMTT2deo9XTDFk4MRfypSPovgPm4WYyilNdAr5nwpNeNm0HusbVLMNt9vj1zxor hu0w== X-Gm-Message-State: AOAM531GiGUTbCY6fXyoB7aV6Gz1gY5LCKVEp+5A45yMbeS7EsVlWLl0 3aRIoihqCycesAxeOgqY/C+JyHDYx+fXZGcR X-Google-Smtp-Source: ABdhPJy2l/O+oho3Jx+i8vPBRwZoxDCJHPX4ZJfTmMVoBuUmrUSRDXK9gRQP/XvrS0UN33z/eOUSAg== X-Received: by 2002:a5d:6708:: with SMTP id o8mr190459wru.64.1607991847796; Mon, 14 Dec 2020 16:24:07 -0800 (PST) Received: from localhost.localdomain ([85.255.232.163]) by smtp.gmail.com with ESMTPSA id b19sm5362012wmj.37.2020.12.14.16.24.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 16:24:07 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v1 4/6] block/psi: remove PSI annotations from direct IO Date: Tue, 15 Dec 2020 00:20:23 +0000 Message-Id: <1d3cf86668e44b3a3d35b5dbe759a086a157e434.1607976425.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org As reported, we must not do pressure stall information accounting for direct IO, because otherwise it tells that it's thrashing a page when actually doing IO on hot data. Apparently, bio_iov_iter_get_pages() is used only by paths doing direct IO, so just make it avoid setting BIO_WORKINGSET, it also saves us CPU cycles on doing that. For fs/direct-io.c just clear the flag before submit_bio(), it's not of much concern performance-wise. Reported-by: Christoph Hellwig Suggested-by: Christoph Hellwig Suggested-by: Johannes Weiner Signed-off-by: Pavel Begunkov --- block/bio.c | 25 ++++++++++++++++--------- fs/direct-io.c | 2 ++ 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/block/bio.c b/block/bio.c index 4a8f77bb3956..3192358c411f 100644 --- a/block/bio.c +++ b/block/bio.c @@ -963,18 +963,22 @@ EXPORT_SYMBOL_GPL(bio_release_pages); static int __bio_iov_bvec_add_pages(struct bio *bio, struct iov_iter *iter) { const struct bio_vec *bv = iter->bvec; - unsigned int len; - size_t size; + struct page *page = bv->bv_page; + bool same_page = false; + unsigned int off, len; if (WARN_ON_ONCE(iter->iov_offset > bv->bv_len)) return -EINVAL; len = min_t(size_t, bv->bv_len - iter->iov_offset, iter->count); - size = bio_add_page(bio, bv->bv_page, len, - bv->bv_offset + iter->iov_offset); - if (unlikely(size != len)) - return -EINVAL; - iov_iter_advance(iter, size); + off = bv->bv_offset + iter->iov_offset; + + if (!__bio_try_merge_page(bio, page, len, off, &same_page)) { + if (bio_full(bio, len)) + return -EINVAL; + bio_add_page_noaccount(bio, page, len, off); + } + iov_iter_advance(iter, len); return 0; } @@ -1023,8 +1027,8 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) put_page(page); } else { if (WARN_ON_ONCE(bio_full(bio, len))) - return -EINVAL; - __bio_add_page(bio, page, len, offset); + return -EINVAL; + bio_add_page_noaccount(bio, page, len, offset); } offset = 0; } @@ -1099,6 +1103,9 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * fit into the bio, or are requested in @iter, whatever is smaller. If * MM encounters an error pinning the requested pages, it stops. Error * is returned only if 0 pages could be pinned. + * + * It also doesn't set BIO_WORKINGSET, so is intended for direct IO. If used + * otherwise the caller is responsible to do that to keep PSI happy. */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { diff --git a/fs/direct-io.c b/fs/direct-io.c index d53fa92a1ab6..914a7f600ecd 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -426,6 +426,8 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) unsigned long flags; bio->bi_private = dio; + /* PSI is only for paging IO */ + bio_clear_flag(bio, BIO_WORKINGSET); spin_lock_irqsave(&dio->bio_lock, flags); dio->refcount++; From patchwork Tue Dec 15 00:20:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11973421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C270C2BB40 for ; Tue, 15 Dec 2020 00:28:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 62CFE22273 for ; Tue, 15 Dec 2020 00:28:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729177AbgLOA1w (ORCPT ); Mon, 14 Dec 2020 19:27:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728450AbgLOAZI (ORCPT ); Mon, 14 Dec 2020 19:25:08 -0500 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B547C06138C; Mon, 14 Dec 2020 16:24:10 -0800 (PST) Received: by mail-wm1-x343.google.com with SMTP id k10so15342145wmi.3; Mon, 14 Dec 2020 16:24:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v0AauUnwA0hTGdBXNj6ZzKlUOrb4jF47iEjWqFK5L8s=; b=DmTQasMwcOvVC2HGyReHTCHsrRm49zPi6M3MRtAMOhqhX/MN7tDKz9tL+2pg6+QqYB CXap+KpU5xs24Y+WYCf1WmIMcbkqlrNyus789gDHv7ZcPFhHbkPbeOMS/zQtebq3NWL0 g6sKVXm18j9BbKfBTXinHBaJ1rkacOcYlffL1aBWZLdVYiAedGshDJV8apYteChI1dp3 kwkWBENwuxx4wSeb6aRKG2sPyMMMglhBwVrK8hYRjGz+bUIbsu2XwQ7RGM4HF38LI006 yaGms8xc0u3XMms3xpgR52HhDX5ov0KL7K0BuvZS+UZXQZh49414R/7FwDK2hlXJMadg TuBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v0AauUnwA0hTGdBXNj6ZzKlUOrb4jF47iEjWqFK5L8s=; b=QyfMUBFuUm3Mfyzzb9fuP3ipEImcyr8tKELyi/+ZUCAPLuav8EL3G3tOKM6We2PZrY CDBzdgYzC9ahxAT08eTJvoIiwsvaUkBL+9cpE3J6H0OTXb9bK4d7ZC6UydHWeIH07MYH ieIy06LpjjGkc+9Uz5RsRbeoMUOIZwE4NXk87W6ONhfdcd2LyI5YyYzetneG8P59lxEF M/gHmyvOaKoo5Q6am/x0QUD6CSzlkusH1mIGRh/N4lGilN07WdrWGtMcSl0mV9jvdksi YfBU+SPPtRC+empqUcE2SSmUiFYFQCsuvcnoMihSj7tXGrC593wuEk0JOdJM9tjufmog vY7A== X-Gm-Message-State: AOAM532MMyZUprFztZ57rCLu6/Q7N6PwiBCyihtG8e/hb0mRPbzNL51n zTDXQVeQXJgEW2pqcu7AGBXRcE4THF/DkzlL X-Google-Smtp-Source: ABdhPJxINkanVy2QGQhY1u+YaY38qOJLZ6n+hZXE+kGwMf8ljRg3x4j8lZkvZFRa2g0tfnchLTq01Q== X-Received: by 2002:a1c:64c4:: with SMTP id y187mr25944685wmb.165.1607991848997; Mon, 14 Dec 2020 16:24:08 -0800 (PST) Received: from localhost.localdomain ([85.255.232.163]) by smtp.gmail.com with ESMTPSA id b19sm5362012wmj.37.2020.12.14.16.24.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 16:24:08 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v1 5/6] bio: add a helper calculating nr segments to alloc Date: Tue, 15 Dec 2020 00:20:24 +0000 Message-Id: <94b6f76d2d47569742ee47caede1504926f9807a.1607976425.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org A preparation patch. It adds a simple helper which abstracts out number of segments we're allocating for a bio from iov_iter_npages(). Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- fs/block_dev.c | 7 ++++--- fs/iomap/direct-io.c | 9 ++++----- include/linux/bio.h | 6 ++++++ 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 9e56ee1f2652..1cbbc794edc6 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -417,7 +417,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) dio->size += bio->bi_iter.bi_size; pos += bio->bi_iter.bi_size; - nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES); + nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_PAGES); if (!nr_pages) { bool polled = false; @@ -482,9 +482,10 @@ blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) { int nr_pages; - nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES + 1); - if (!nr_pages) + if (!iov_iter_count(iter)) return 0; + + nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_PAGES + 1); if (is_sync_kiocb(iocb) && nr_pages <= BIO_MAX_PAGES) return __blkdev_direct_IO_simple(iocb, iter, nr_pages); diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 933f234d5bec..ea1e8f696076 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -250,11 +250,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, orig_count = iov_iter_count(dio->submit.iter); iov_iter_truncate(dio->submit.iter, length); - nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES); - if (nr_pages <= 0) { - ret = nr_pages; + if (!iov_iter_count(dio->submit.iter)) goto out; - } if (need_zeroout) { /* zero out from the start of the block to the write offset */ @@ -263,6 +260,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, iomap_dio_zero(dio, iomap, pos - pad, pad); } + nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter, BIO_MAX_PAGES); do { size_t n; if (dio->error) { @@ -308,7 +306,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, dio->size += n; copied += n; - nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES); + nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter, + BIO_MAX_PAGES); iomap_dio_submit_bio(dio, iomap, bio, pos); pos += n; } while (nr_pages); diff --git a/include/linux/bio.h b/include/linux/bio.h index 1edda614f7ce..2a9f3f0bbe0a 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -10,6 +10,7 @@ #include /* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */ #include +#include #define BIO_DEBUG @@ -441,6 +442,11 @@ static inline void bio_wouldblock_error(struct bio *bio) bio_endio(bio); } +static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) +{ + return iov_iter_npages(iter, max_segs); +} + struct request_queue; extern int submit_bio_wait(struct bio *bio); From patchwork Tue Dec 15 00:20:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11973411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A26C2BB48 for ; Tue, 15 Dec 2020 00:26:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7248322273 for ; Tue, 15 Dec 2020 00:26:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728773AbgLOAZg (ORCPT ); Mon, 14 Dec 2020 19:25:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728755AbgLOAZY (ORCPT ); Mon, 14 Dec 2020 19:25:24 -0500 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C80DFC061282; Mon, 14 Dec 2020 16:24:11 -0800 (PST) Received: by mail-wr1-x441.google.com with SMTP id y17so18129652wrr.10; Mon, 14 Dec 2020 16:24:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=atVGnGnKX0+QWRcHgwbeiUpqoMrtsuGF8OD/ap3yMV0=; b=u+zjC4RjWtSd1EVEtSggrw5x/fzyJXtB05ocK2DmCyHbvHdbAjSYxqLG/aQQ/1SN7+ v8zcHxWf3G+8vVngFW8HMhPd4JY2HkvboHm3n5Pb04oWacCx+zManfJ+/QEd3911HPgi 5VgZYb+6nxuhmIlsEbRI8AUA0I1FGrZZlsVfA5jTjyqLOp/nV2w29yuWXPkRlUcJWDpP 4d7OeUsozBCiZNZFfhlcizeVjluwsy8qgAwJf3Y/uJOLfdRI7DjoIRBGKd2S+0Yv9ddS nhAy6A+Ovk//nWdeGQ/vYQc5ldTzOEXOsG45nqJlWDD4IABVlHzYKw6sdiv00OUaDbOo oTlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=atVGnGnKX0+QWRcHgwbeiUpqoMrtsuGF8OD/ap3yMV0=; b=P4b0RQrWgJOlf89pTppOa6bd58Ig4/i9gNEELgnL2ndcqrypiHf62srVG6cEhD0rzB tgHUAVsUeEt2YikzzToYTAxO0J4utwCDVNcu09m/lLYrLiz0wi3KDSmxMM9yv39qdvlC o4S7hEJPsPmbGX2QO0Je2zPCZKcQ+fU6MSnOiNPLUimpSxrj6CW2Xc2faqhkCTuDRVXh f7/aeHLyWr7VwTbLuNJ9seutqYaT5hsYxG6M3XpqWcSBnisP7BNimczqSwk6EIPPDZEC VrROe5dakfiEjFNDa53iNcccrC0tdET9jCsgG3i+2UXvCRXqXSMxfDj09HtgxBdRJD8r g3Tg== X-Gm-Message-State: AOAM53262akUQov4R5/c2dtwtkg7s5/8CtK1FGeG2oTHR2USIeFcbz1e 9bg2Qd0Dphtlp8ggyxGjHbERX3tQdCAOBeID X-Google-Smtp-Source: ABdhPJzD6TNgMT/l5rfeMAyPUUuqa5EHPZa+zhYb6kKMMDi0EGqLG8qd52Qk85eB5WU9zWrxCo1MOg== X-Received: by 2002:adf:f18a:: with SMTP id h10mr33264191wro.244.1607991850205; Mon, 14 Dec 2020 16:24:10 -0800 (PST) Received: from localhost.localdomain ([85.255.232.163]) by smtp.gmail.com with ESMTPSA id b19sm5362012wmj.37.2020.12.14.16.24.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Dec 2020 16:24:09 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v1 6/6] block/iomap: don't copy bvec for direct IO Date: Tue, 15 Dec 2020 00:20:25 +0000 Message-Id: <498b34d746627e874740d8315b2924880c46dbc3.1607976425.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org The block layer spends quite a while in blkdev_direct_IO() to copy and initialise bio's bvec. However, if we've already got a bvec in the input iterator it might be reused in some cases, i.e. when new ITER_BVEC_FLAG_FIXED flag is set. Simple tests show considerable performance boost, and it also reduces memory footprint. Suggested-by: Matthew Wilcox Signed-off-by: Pavel Begunkov --- Documentation/filesystems/porting.rst | 9 ++++ block/bio.c | 64 +++++++++++---------------- include/linux/bio.h | 3 ++ 3 files changed, 38 insertions(+), 38 deletions(-) diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index 867036aa90b8..47a622879952 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -865,3 +865,12 @@ no matter what. Everything is handled by the caller. clone_private_mount() returns a longterm mount now, so the proper destructor of its result is kern_unmount() or kern_unmount_array(). + +--- + +**mandatory** + +For bvec based itererators bio_iov_iter_get_pages() now doesn't copy bvecs but +uses the one provided. Anyone issuing kiocb-I/O should ensure that the bvec and +page references stay until I/O has completed, i.e. until ->ki_complete() has +been called or returned with non -EIOCBQUEUED code. diff --git a/block/bio.c b/block/bio.c index 3192358c411f..f8229be24562 100644 --- a/block/bio.c +++ b/block/bio.c @@ -960,25 +960,16 @@ void bio_release_pages(struct bio *bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(bio_release_pages); -static int __bio_iov_bvec_add_pages(struct bio *bio, struct iov_iter *iter) +static int bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { - const struct bio_vec *bv = iter->bvec; - struct page *page = bv->bv_page; - bool same_page = false; - unsigned int off, len; - - if (WARN_ON_ONCE(iter->iov_offset > bv->bv_len)) - return -EINVAL; - - len = min_t(size_t, bv->bv_len - iter->iov_offset, iter->count); - off = bv->bv_offset + iter->iov_offset; - - if (!__bio_try_merge_page(bio, page, len, off, &same_page)) { - if (bio_full(bio, len)) - return -EINVAL; - bio_add_page_noaccount(bio, page, len, off); - } - iov_iter_advance(iter, len); + WARN_ON_ONCE(BVEC_POOL_IDX(bio) != 0); + bio->bi_vcnt = iter->nr_segs; + bio->bi_max_vecs = iter->nr_segs; + bio->bi_io_vec = (struct bio_vec *)iter->bvec; + bio->bi_iter.bi_bvec_done = iter->iov_offset; + bio->bi_iter.bi_size = iter->count; + + iov_iter_advance(iter, iter->count); return 0; } @@ -1092,12 +1083,13 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * This takes either an iterator pointing to user memory, or one pointing to * kernel pages (BVEC iterator). If we're adding user pages, we pin them and * map them into the kernel. On IO completion, the caller should put those - * pages. If we're adding kernel pages, and the caller told us it's safe to - * do so, we just have to add the pages to the bio directly. We don't grab an - * extra reference to those pages (the user should already have that), and we - * don't put the page on IO completion. The caller needs to check if the bio is - * flagged BIO_NO_PAGE_REF on IO completion. If it isn't, then pages should be - * released. + * pages. If we're adding kernel pages, it doesn't take extra page references + * and reuses the provided bvec, so the caller must ensure that the bvec isn't + * freed and page references remain to be taken until I/O has completed. If + * the I/O is completed asynchronously, the bvec must not be freed before + * ->ki_complete() has been called. The caller needs to check if the bio is + * flagged BIO_NO_PAGE_REF on IO completion. If it isn't, then pages should + * be released. * * The function tries, but does not guarantee, to pin as many pages as * fit into the bio, or are requested in @iter, whatever is smaller. If @@ -1109,27 +1101,23 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { - const bool is_bvec = iov_iter_is_bvec(iter); int ret; - if (WARN_ON_ONCE(bio->bi_vcnt)) - return -EINVAL; + if (iov_iter_is_bvec(iter)) { + if (WARN_ON_ONCE(bio_op(bio) == REQ_OP_ZONE_APPEND)) + return -EINVAL; + bio_iov_bvec_set(bio, iter); + bio_set_flag(bio, BIO_NO_PAGE_REF); + return 0; + } do { - if (bio_op(bio) == REQ_OP_ZONE_APPEND) { - if (WARN_ON_ONCE(is_bvec)) - return -EINVAL; + if (bio_op(bio) == REQ_OP_ZONE_APPEND) ret = __bio_iov_append_get_pages(bio, iter); - } else { - if (is_bvec) - ret = __bio_iov_bvec_add_pages(bio, iter); - else - ret = __bio_iov_iter_get_pages(bio, iter); - } + else + ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); - if (is_bvec) - bio_set_flag(bio, BIO_NO_PAGE_REF); return bio->bi_vcnt ? 0 : ret; } EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); diff --git a/include/linux/bio.h b/include/linux/bio.h index 2a9f3f0bbe0a..337f4280b639 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -444,6 +444,9 @@ static inline void bio_wouldblock_error(struct bio *bio) static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) { + /* reuse iter->bvec */ + if (iov_iter_is_bvec(iter)) + return 0; return iov_iter_npages(iter, max_segs); }