From patchwork Sat Jan 9 16:02:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 496BAC433E6 for ; Sat, 9 Jan 2021 16:08:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2062E23A23 for ; Sat, 9 Jan 2021 16:08:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726447AbhAIQH2 (ORCPT ); Sat, 9 Jan 2021 11:07:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726073AbhAIQH1 (ORCPT ); Sat, 9 Jan 2021 11:07:27 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13106C06179F; Sat, 9 Jan 2021 08:06:47 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id e25so11050762wme.0; Sat, 09 Jan 2021 08:06:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lGO3N0dJRSDXwKGKd9FDaeuDl4ejZglaZH9ZU4+s/tY=; b=PNS20XMr5oY5wsCd5IWCatRGYgKhqOGRWD42cUzqU3npgYmGMzrjMlQaKrMuRzUyc8 7le18rv1rLvmFEUClV6PEHtvQK3aPEQ5Eavs+vj4yHGO7YyYQ6gOpjJRPPtMJXuCzzy7 0SMvGb+teD4g2KnZITU9ti3EZ5Z0APu7/3lwhMTAld3hg2khQV8kramlUuW9yuI/bIyk rKQb1fd9HbjFeAz1LdRXFtgYiqhE3kSck9Y9NZIP08qyIULBtDjcvn9ateqYsZQmm8n5 0O4OxKK1lyWfrvqjIu713MV1I9fpuCZEVnKGze8Rz95P5bj+PaD4IIN8RriANInJtb5U ifug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lGO3N0dJRSDXwKGKd9FDaeuDl4ejZglaZH9ZU4+s/tY=; b=BYYhz9CI58obEjQWt9TQLShaDMtO2DLRg1WGUD9Hg7fTo3yLksRCysBp151BEWVjK6 K6vJ1B6t5Lz3L/sByRAJvmxBa3bYNaPJaRT4sIa5t0k9jcxo+eMAt22Aa73gS7S+Xws4 sA0GVw8Wg0MKDik8yiGRf1G8wdWz6uKvsI1DUtPKEaNwd5myEzSnVBGn04OXkQvHcPqe t4eEhfAxPxK8AX5F1yvEhBCtPfPjUJZQCRDOBLbbNWme5yHLZ2MBeV/iXuWK0FXPLkH9 gXSobW8qN9sqYknHG4bkc2014iAbwatWoNFYZfi9vfhFjUZNs9zvl+mSQW5AoI82Voi1 GBMg== X-Gm-Message-State: AOAM532YiM8OcP0W1YjUALKWfYLi3Y+ElhbQpp3c3/Pzx4jKIeXXA81s s68rb5l1Dt6XnUoBMAzKQpJWaW9J7s45xnvG X-Google-Smtp-Source: ABdhPJyPJNi1iMc9SCE+oR3bxu64pyOQeD5A8QFIrcf1TR3/G6IczfLuhp47YWEuOLfIpgEf/eM15g== X-Received: by 2002:a1c:790f:: with SMTP id l15mr7831379wme.188.1610208405336; Sat, 09 Jan 2021 08:06:45 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:44 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 1/7] splice: don't generate zero-len segement bvecs Date: Sat, 9 Jan 2021 16:02:57 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org iter_file_splice_write() may spawn bvec segments with zero-length. In preparation for prohibiting them, filter out by hand at splice level. Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- fs/splice.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 866d5c2367b2..474fb8b5562a 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -662,12 +662,14 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, /* build the vector */ left = sd.total_len; - for (n = 0; !pipe_empty(head, tail) && left && n < nbufs; tail++, n++) { + for (n = 0; !pipe_empty(head, tail) && left && n < nbufs; tail++) { struct pipe_buffer *buf = &pipe->bufs[tail & mask]; size_t this_len = buf->len; - if (this_len > left) - this_len = left; + /* zero-length bvecs are not supported, skip them */ + if (!this_len) + continue; + this_len = min(this_len, left); ret = pipe_buf_confirm(pipe, buf); if (unlikely(ret)) { @@ -680,6 +682,7 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, array[n].bv_len = this_len; array[n].bv_offset = buf->offset; left -= this_len; + n++; } iov_iter_bvec(&from, WRITE, array, n, sd.total_len - left); From patchwork Sat Jan 9 16:02:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEA51C4360C for ; Sat, 9 Jan 2021 16:08:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9C0A323A04 for ; Sat, 9 Jan 2021 16:08:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726864AbhAIQHt (ORCPT ); Sat, 9 Jan 2021 11:07:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726527AbhAIQH3 (ORCPT ); Sat, 9 Jan 2021 11:07:29 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92BF5C0617A2; Sat, 9 Jan 2021 08:06:48 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id g185so11020610wmf.3; Sat, 09 Jan 2021 08:06:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q8d8dG9UCf5e8vbipZsYPEj7r5MDYaj3bnreiQQXGps=; b=flzNBu2Q3t3/ImhTptnUpWZq2eOx/8D444lV2HOdKskJiBvPeDRYD15kOiU4YWJuig a2s3X5nS7c5Gu0oZ6799cSPGNNhYVL/DNuj6K8u45rZ2OSMGtpibL7htfny+gBGn1Ir7 2ChwG/1/1TCbzPQMHL+TfEpG3iCnbxF6sZ8AsQ/02eXp20D6QL3CKQRqDDmBkSVfC0SV f6IuYJToioYrFrS8j23EyvK/Odm1iOgFzB2J7lVZzbO4RXL0oEFMFsAlNXU2n+23bBVy YEfcRNr25KYQJTnYxjfV0xLx+bWyP/gAF04dZSua4crVLyAEMveSQ92gBaEVPLnZzdfo HAFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q8d8dG9UCf5e8vbipZsYPEj7r5MDYaj3bnreiQQXGps=; b=fDhjNFVvismg2ilzciurJ1os2toxyyXDUDRyQrj3lhiDS6Mgc4yyuvotJ+H4tD9X2C jg8TKJpfXXn/71+xqK5nY/ln//3Vws5Thbp7kxNw5cFt+VmBatb7iSNbbemAOWU86Hdm XK4PtK8l3aSZ/uDYN4dQLNOTjUGMEfArn8WztcdwGVFnKe/vjU8cdqMkSsza4g1Op3PK ojIfXcyFxs3yFwsVwT/yJEsIH0lDd4xXJcdbazOqn50Azzu7qKKD/i/HZodQ53I2DE0i fMdynDTe3vFWXQziO8s4hAEPuRMe9PsA1Ud2pHtWpQa7SkoAWyE/bLF4VJZftYLry7XQ CNvA== X-Gm-Message-State: AOAM533KEY5/uj+7F2C7l29RxQWtx+UIvDMu/PIMUZ8PUcinUtd/f9y8 GofKHMdX0rZGATXdSH8ntCCb1OjZHfgL47pS X-Google-Smtp-Source: ABdhPJzXFJcFemHVSsCyCdF3SY1fQfhQCrpun0BXv+Pm10PNgRweNCoNTn+FNvIFpVE+sVQIwpzbuQ== X-Received: by 2002:a05:600c:cc:: with SMTP id u12mr7982009wmm.42.1610208407039; Sat, 09 Jan 2021 08:06:47 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:46 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 2/7] bvec/iter: disallow zero-length segment bvecs Date: Sat, 9 Jan 2021 16:02:58 +0000 Message-Id: <4570836cc62137a9ee788d9c820f58ed8efe9b37.1610170479.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org zero-length bvec segments are allowed in general, but not handled by bio and down the block layer so filtered out. This inconsistency may be confusing and prevent from optimisations. As zero-length segments are useless and places that were generating them are patched, declare them not allowed. Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- Documentation/block/biovecs.rst | 2 ++ Documentation/filesystems/porting.rst | 7 +++++++ lib/iov_iter.c | 2 -- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Documentation/block/biovecs.rst b/Documentation/block/biovecs.rst index 36771a131b56..ddb867e0185b 100644 --- a/Documentation/block/biovecs.rst +++ b/Documentation/block/biovecs.rst @@ -40,6 +40,8 @@ normal code doesn't have to deal with bi_bvec_done. There is a lower level advance function - bvec_iter_advance() - which takes a pointer to a biovec, not a bio; this is used by the bio integrity code. +As of 5.12 bvec segments with zero bv_len are not supported. + What's all this get us? ======================= diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index 867036aa90b8..c722d94f29ea 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -865,3 +865,10 @@ no matter what. Everything is handled by the caller. clone_private_mount() returns a longterm mount now, so the proper destructor of its result is kern_unmount() or kern_unmount_array(). + +--- + +**mandatory** + +zero-length bvec segments are disallowed, they must be filtered out before +passed on to an iterator. diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1635111c5bd2..7de304269641 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -72,8 +72,6 @@ __start.bi_bvec_done = skip; \ __start.bi_idx = 0; \ for_each_bvec(__v, i->bvec, __bi, __start) { \ - if (!__v.bv_len) \ - continue; \ (void)(STEP); \ } \ } From patchwork Sat Jan 9 16:02:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF4B0C43142 for ; Sat, 9 Jan 2021 16:07:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF85A23AAC for ; Sat, 9 Jan 2021 16:07:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726676AbhAIQHc (ORCPT ); Sat, 9 Jan 2021 11:07:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726642AbhAIQHa (ORCPT ); Sat, 9 Jan 2021 11:07:30 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A348C0617A3; Sat, 9 Jan 2021 08:06:50 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id 190so10216019wmz.0; Sat, 09 Jan 2021 08:06:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ouhtlCdJ/DY280urrHIcZiPcXncXrhPOyIjDJZKeP64=; b=qy5YzKqSVfj/n61inmw754WS9TPAvCacovuEf3ll3twxCUwvazuZ1XoReQ9iWnKikI LIBWzWQu1z7SOxwi++WGQbPTnkcjlVXSg4idyU2Ael20Dmj6WscsKo4BjDK+k+cOCgOK AgfVg6ARPHkkdURVSbRB2cyeIG87ftyMhN6S30APxSo/ZGFubuF9E5plaXWQr0TFgm2D VsA8KwTn+D9JUhJaSNS3Mw5LO3TAbxBMcYZQXlGHMUhB8w7TPdNwk3Ey4bBoG+lGeFur D+r+jIEFATqrHDTKCAaQGzJYt+gXNh++yGeJvODQVW4uyj2JRglkfeBakz1FKdtZN7vP y9Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ouhtlCdJ/DY280urrHIcZiPcXncXrhPOyIjDJZKeP64=; b=C1AIrKH+g2WH1MzdATOmKtXNmbXvLspjVSSyqaBcNX+jKHQkHA5yVRrWhXY2djjFyx Sjn9Rna0PmaDNZL2QXuoLMnkIS/Ep1VsxA/51ApOkH+IAOfCHL5iSt8twYC8s8N8x4lf QAvufHD525y9T9U35mbu6GaEpLsCWh92JbpGgxKBkzw7WX4m7aylF+Thu8Ms9CdGpviy hO+kbm/wQiVAI7rS79I1F3XamUW6qojr0MJIkjRDBzJd6FrhKI3Kc+4ECqIXzGzvNYry ScO+LYO/wmmsbRjQPM5d1l2qTqCHrCyWYZFYmr0Q6sB5aYPuOup2XfGOMMLLu5jxPEtP iGiw== X-Gm-Message-State: AOAM530QFRSCf/Ym/npTfsaeFMRQixOab8mGdHlzGeb94pH12SDpYyQd ukyK7K/dQDsj5gAnbq2bpfhKSuKcpG4L+8zq X-Google-Smtp-Source: ABdhPJw4gnrLCSdkD/iV43iWJWvb3fyCSRNNfo8n4OuJ5w224968euoNuQ/4NpxpqxQQbSZa0wIJ1A== X-Received: by 2002:a05:600c:2042:: with SMTP id p2mr7943949wmg.152.1610208408536; Sat, 09 Jan 2021 08:06:48 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:47 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 3/7] block/psi: remove PSI annotations from direct IO Date: Sat, 9 Jan 2021 16:02:59 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Direct IO does not operate on the current working set of pages managed by the kernel, so it should not be accounted as memory stall to PSI infrastructure. The block layer and iomap direct IO use bio_iov_iter_get_pages() to build bios, and they are the only users of it, so to avoid PSI tracking for them clear out BIO_WORKINGSET flag. Do same for dio_bio_submit() because fs/direct_io constructs bios by hand directly calling bio_add_page(). Reported-by: Christoph Hellwig Suggested-by: Christoph Hellwig Suggested-by: Johannes Weiner Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- block/bio.c | 6 ++++++ fs/direct-io.c | 2 ++ 2 files changed, 8 insertions(+) diff --git a/block/bio.c b/block/bio.c index 1f2cc1fbe283..9f26984af643 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1099,6 +1099,9 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * fit into the bio, or are requested in @iter, whatever is smaller. If * MM encounters an error pinning the requested pages, it stops. Error * is returned only if 0 pages could be pinned. + * + * It's intended for direct IO, so doesn't do PSI tracking, the caller is + * responsible for setting BIO_WORKINGSET if necessary. */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1123,6 +1126,9 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) if (is_bvec) bio_set_flag(bio, BIO_NO_PAGE_REF); + + /* don't account direct I/O as memory stall */ + bio_clear_flag(bio, BIO_WORKINGSET); return bio->bi_vcnt ? 0 : ret; } EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); diff --git a/fs/direct-io.c b/fs/direct-io.c index d53fa92a1ab6..0e689233f2c7 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -426,6 +426,8 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) unsigned long flags; bio->bi_private = dio; + /* don't account direct I/O as memory stall */ + bio_clear_flag(bio, BIO_WORKINGSET); spin_lock_irqsave(&dio->bio_lock, flags); dio->refcount++; From patchwork Sat Jan 9 16:03:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74C02C43603 for ; Sat, 9 Jan 2021 16:07:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4CB8723A1C for ; Sat, 9 Jan 2021 16:07:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726723AbhAIQHd (ORCPT ); Sat, 9 Jan 2021 11:07:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726073AbhAIQHb (ORCPT ); Sat, 9 Jan 2021 11:07:31 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57198C0617A4; Sat, 9 Jan 2021 08:06:51 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id 190so10216049wmz.0; Sat, 09 Jan 2021 08:06:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LOsfOtb4JF9iqyizeOjm4tVRPPo2GoTdvRn37AEB3XM=; b=Naupq7MH/QGjgYPNZ6nDh8HfOkE4n+5H8WjdQV22O4U+GN5oR8Ixv+Gl0qgQdk/j5c ror5a9etqnkRwbLZtFecJMcJzPkfKyMpprD4UzKGG7SSMj0Wi+vldVDQD03UCr6cRo7n XslEbcpU4b29/jvNvkfRRV425pGdOV5ri/XWO0dOpB0xSe9yp0uF+JUkgxBHlG0EVsSJ vh6OV5OVGJxasK4c8MsZ0f0UaF7d5DdwAcf3iB4vuMr/Bo932nWihR9VgM3blH1dJpvR guhwKvbCGPS3CqatmW254sYwWbBHH7dCEoXtGD9igJm7GWNOpYe5oppH34AB0Kbgxc3c vGbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LOsfOtb4JF9iqyizeOjm4tVRPPo2GoTdvRn37AEB3XM=; b=j8MFP6gGghrGEaOUgrKW1uX0oott1K/p4BNb2FJJ0065VQQx3j5jkUZQi7w/wbrQP/ FUtKUfIYToAM0F9cuUW7AJ6hiKlRyILiXMNouEkq+wHiF5LVZSjWBwDH8knhgzBRU+8E 5pMrsTasAqD6HQXWrfwlVIzGSDwl1DJy+tk2zubp3Bs2JJCIUjmyv9tr1+7e8nEiaoj3 5EILJK82RIuvQLAp6oXrlP7q84pJklWplG5InfkScdHKV+HgTBnxGvxfiJDajG54QnlO Ufewxd+ilTy77L5vUYyMuRQE2Mq0AbF90ROlokFfSpa0s4soiOHe7lyvAJ/Tlfa/RmB0 VPcg== X-Gm-Message-State: AOAM531ziyyhi+zFizpCjKv3M6qZGMRFl3fAdT1FnoLFHusc9eCmmq6S q4xEnuX6u127MMxxRxodSeBhp6cvq3bI2GGx X-Google-Smtp-Source: ABdhPJwTh8SgrJgP2KLlExaCfgCAEBrNfjWruyuIlYMsHFTOuOfOCftaIufrH3TPb7KFf9qH55pOLg== X-Received: by 2002:a1c:7d94:: with SMTP id y142mr7810148wmc.105.1610208409857; Sat, 09 Jan 2021 08:06:49 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:49 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 4/7] target/file: allocate the bvec array as part of struct target_core_file_cmd Date: Sat, 9 Jan 2021 16:03:00 +0000 Message-Id: <2650722037cd756690f2e398468420bbaa26ed7f.1610170479.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Christoph Hellwig This saves one memory allocation, and ensures the bvecs aren't freed before the AIO completion. This will allow the lower level code to be optimized so that it can avoid allocating another bvec array. Signed-off-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- drivers/target/target_core_file.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/drivers/target/target_core_file.c b/drivers/target/target_core_file.c index b0cb5b95e892..cce455929778 100644 --- a/drivers/target/target_core_file.c +++ b/drivers/target/target_core_file.c @@ -241,6 +241,7 @@ struct target_core_file_cmd { unsigned long len; struct se_cmd *cmd; struct kiocb iocb; + struct bio_vec bvecs[]; }; static void cmd_rw_aio_complete(struct kiocb *iocb, long ret, long ret2) @@ -268,29 +269,22 @@ fd_execute_rw_aio(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, struct target_core_file_cmd *aio_cmd; struct iov_iter iter = {}; struct scatterlist *sg; - struct bio_vec *bvec; ssize_t len = 0; int ret = 0, i; - aio_cmd = kmalloc(sizeof(struct target_core_file_cmd), GFP_KERNEL); + aio_cmd = kmalloc(struct_size(aio_cmd, bvecs, sgl_nents), GFP_KERNEL); if (!aio_cmd) return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; - bvec = kcalloc(sgl_nents, sizeof(struct bio_vec), GFP_KERNEL); - if (!bvec) { - kfree(aio_cmd); - return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; - } - for_each_sg(sgl, sg, sgl_nents, i) { - bvec[i].bv_page = sg_page(sg); - bvec[i].bv_len = sg->length; - bvec[i].bv_offset = sg->offset; + aio_cmd->bvecs[i].bv_page = sg_page(sg); + aio_cmd->bvecs[i].bv_len = sg->length; + aio_cmd->bvecs[i].bv_offset = sg->offset; len += sg->length; } - iov_iter_bvec(&iter, is_write, bvec, sgl_nents, len); + iov_iter_bvec(&iter, is_write, aio_cmd->bvecs, sgl_nents, len); aio_cmd->cmd = cmd; aio_cmd->len = len; @@ -307,8 +301,6 @@ fd_execute_rw_aio(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, else ret = call_read_iter(file, &aio_cmd->iocb, &iter); - kfree(bvec); - if (ret != -EIOCBQUEUED) cmd_rw_aio_complete(&aio_cmd->iocb, ret, 0); From patchwork Sat Jan 9 16:03:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E86AC433E0 for ; Sat, 9 Jan 2021 16:08:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2FFCF23A1C for ; Sat, 9 Jan 2021 16:08:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726960AbhAIQIK (ORCPT ); Sat, 9 Jan 2021 11:08:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726527AbhAIQII (ORCPT ); Sat, 9 Jan 2021 11:08:08 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8A73C0617A5; Sat, 9 Jan 2021 08:06:52 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id k10so10197014wmi.3; Sat, 09 Jan 2021 08:06:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BfcAWicfY3ps8Cp/gsA4qs7Avo53ajYGpXmNYv+JbSY=; b=AeQssdOPinNt7VEfPxpzOS0PFW0o40Y9EV+3MG63+Dd8wEjtQhPls6aUI/zqMC4gsv IW3I6GjVy1dOi5SnlEp+YRA1/gtBdhH5asZs1Vc8BwgkTu/OQL9SiwZQeTeh6MEK9Ytx oKJWgI/oCxvRIUq3vpAnYYeXiwoUDBD2DaSwvxqPoCeBCIOxe63PTPOjy+qnYmafOzLt pOInVokeOb4fjqx5nf6WKgyOT4BJn7nKX37LNZAUETQywherYZWmn6EShmOq2aNMEzsS +u1A8+eL2dHUF/rHQOXJTo6YWxB4tASlksAkwh8nvZUnfMoL5BQpSeLSg0/B3npBgBaE OwzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BfcAWicfY3ps8Cp/gsA4qs7Avo53ajYGpXmNYv+JbSY=; b=BHGQA2UIqbpfCASUyFgEzop2pJwTum3A7MttmuA/MrYk2Y3Fziq9Bxy0/lqLKHk8zc rmTsFH9Vfb8m96q1zpOJCLRfsPkZthPsR6OAgXBay6d/22KO0DS0yhZEb+Q4SI6RFFZH G+pEqVuPrPfrrFloNkvEB8DItyqJbwtX/jL4sphIZwoEPXDZl9P6Tg7nXmOERVZgiliM BpbYZEdabwzwe1QfRoKpDrbyVV6nkpslp4HXbWZriYkGIgeFk84UXfclgF1cSV1b8N6U 2BoSt0luBZm/j/zdFs1MkZA065atDw4ledBsvR2A6An6s9RYWPw4DVeyc+q1PrW+BDPP Y2uQ== X-Gm-Message-State: AOAM531nrSjIY+5yn6vWpErUL/Le9z/2dp1MNepGCXcwuMNQQIwKYPUt HVseCUtTEabpC7s8yQRqJaR90G0HzrRuyCZC X-Google-Smtp-Source: ABdhPJx4y4lbIa9CtCF5CrCkTJk2J7iIX4ikX7/zFLt/oXjOr5IOUBU+qaI+PSRkIWiQi2yqw+zLjA== X-Received: by 2002:a1c:4384:: with SMTP id q126mr7875358wma.84.1610208411272; Sat, 09 Jan 2021 08:06:51 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:50 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 5/7] iov_iter: optimise bvec iov_iter_advance() Date: Sat, 9 Jan 2021 16:03:01 +0000 Message-Id: <58552e3ba333650ccd425823cb9dc0b949350959.1610170479.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org iov_iter_advance() is heavily used, but implemented through generic means. For bvecs there is a specifically crafted function for that, so use bvec_iter_advance() instead, it's faster and slimmer. Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- lib/iov_iter.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 7de304269641..9b1c109dc8a9 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1065,6 +1065,21 @@ static void pipe_advance(struct iov_iter *i, size_t size) pipe_truncate(i); } +static void iov_iter_bvec_advance(struct iov_iter *i, size_t size) +{ + struct bvec_iter bi; + + bi.bi_size = i->count; + bi.bi_bvec_done = i->iov_offset; + bi.bi_idx = 0; + bvec_iter_advance(i->bvec, &bi, size); + + i->bvec += bi.bi_idx; + i->nr_segs -= bi.bi_idx; + i->count = bi.bi_size; + i->iov_offset = bi.bi_bvec_done; +} + void iov_iter_advance(struct iov_iter *i, size_t size) { if (unlikely(iov_iter_is_pipe(i))) { @@ -1075,6 +1090,10 @@ void iov_iter_advance(struct iov_iter *i, size_t size) i->count -= size; return; } + if (iov_iter_is_bvec(i)) { + iov_iter_bvec_advance(i, size); + return; + } iterate_and_advance(i, size, v, 0, 0, 0) } EXPORT_SYMBOL(iov_iter_advance); From patchwork Sat Jan 9 16:03:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D11FC43381 for ; Sat, 9 Jan 2021 16:08:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6FBA123A1C for ; Sat, 9 Jan 2021 16:08:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726932AbhAIQIK (ORCPT ); Sat, 9 Jan 2021 11:08:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726058AbhAIQII (ORCPT ); Sat, 9 Jan 2021 11:08:08 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C274C0617A6; Sat, 9 Jan 2021 08:06:54 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id i9so11920379wrc.4; Sat, 09 Jan 2021 08:06:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RwZZxMhI82qZ1NGjjWs3i9PNzY6l2vY31jXhvq/2jzI=; b=Ttko+RVGU3n5aJcKu+1ljJRmO0ZoHSASRWTCu9NUBLQwvAMVtJm4xDctv0HVgar+fN j1+FVNn31Se10pLKpcqpfLj8k/U/bCPpLxmbmyG9XhOQyBjS0P2EzY+bpgmi+CuQUrhJ WIGEIL+T1d4wzj8RPH1EjE4HYVpzd83UevIJJNOzJAvvhyRCV9gTTPu/DFakNUBhjPUE 6YAc1X5PQF2GEWeLQfCtAjB499ILo9qLgoWgjQm1uq5kYEoxQ6UI0jSz1HOqcEDSpjJd uY01F6EMQsqPI6ykkfpm8TNw/LNJIbIiD1vvLsZRCX6bzgxA5GWTjxPV4bsYNY4Ny2/9 EEYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RwZZxMhI82qZ1NGjjWs3i9PNzY6l2vY31jXhvq/2jzI=; b=fLFT8G3T/77dDHC+cFDrKkF1EG+/9I/IJ4lTztV4/r5D1H+7Y3jRtoajbsurVSWJ2f W5raIE9/GDGDeWVZYd47sTHa9dqKRY1qp/YaGFLoZqbwKNSfAJSzKKpGa1GUSIV0x9ai mPn0YWOAfysDskRRoFeMNTEzjTddoeJTgR2V0tieJSAyycH/i/eYS0k3h+jH0c4R4fEw U8V3yXSKZoPtmI1+rvyqhtO9++QHbYfTZtm03wWaBFvTgPeJcB5ydet/mTSu0GWAA6GO KjKjeOnns01Nv9mYsN2qx15vo7qALiV3rq3OyVa1l2PQXgd8CcY2gYzBL+zz9quzgVlW zn8w== X-Gm-Message-State: AOAM530ZIUsnWvTA29TU4Qn5NUjtMO5Dtzut524QEOGriHtEIbkBRe8T ZUJaU5l0PP1zV0spzizLFiB3n9DUC9Fa1sxM X-Google-Smtp-Source: ABdhPJzoUjsfyBMMnqeq2r+2TVEKl5UD3aKgPfvVrdJFCjILudq2jNC955I3gsW4LTeZMtylLSyYdw== X-Received: by 2002:a5d:4a44:: with SMTP id v4mr8956687wrs.106.1610208412908; Sat, 09 Jan 2021 08:06:52 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:52 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 6/7] bio: add a helper calculating nr segments to alloc Date: Sat, 9 Jan 2021 16:03:02 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add a helper function calculating the number of bvec segments we need to allocate to construct a bio. It doesn't change anything functionally, but will be used to not duplicate special cases in the future. Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- fs/block_dev.c | 7 ++++--- fs/iomap/direct-io.c | 9 ++++----- include/linux/bio.h | 10 ++++++++++ 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 3b8963e228a1..6f5bd9950baf 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -416,7 +416,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) dio->size += bio->bi_iter.bi_size; pos += bio->bi_iter.bi_size; - nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES); + nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_PAGES); if (!nr_pages) { bool polled = false; @@ -481,9 +481,10 @@ blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) { int nr_pages; - nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES + 1); - if (!nr_pages) + if (!iov_iter_count(iter)) return 0; + + nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_PAGES + 1); if (is_sync_kiocb(iocb) && nr_pages <= BIO_MAX_PAGES) return __blkdev_direct_IO_simple(iocb, iter, nr_pages); diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 933f234d5bec..ea1e8f696076 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -250,11 +250,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, orig_count = iov_iter_count(dio->submit.iter); iov_iter_truncate(dio->submit.iter, length); - nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES); - if (nr_pages <= 0) { - ret = nr_pages; + if (!iov_iter_count(dio->submit.iter)) goto out; - } if (need_zeroout) { /* zero out from the start of the block to the write offset */ @@ -263,6 +260,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, iomap_dio_zero(dio, iomap, pos - pad, pad); } + nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter, BIO_MAX_PAGES); do { size_t n; if (dio->error) { @@ -308,7 +306,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, dio->size += n; copied += n; - nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES); + nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter, + BIO_MAX_PAGES); iomap_dio_submit_bio(dio, iomap, bio, pos); pos += n; } while (nr_pages); diff --git a/include/linux/bio.h b/include/linux/bio.h index 1edda614f7ce..d8f9077c43ef 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -10,6 +10,7 @@ #include /* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */ #include +#include #define BIO_DEBUG @@ -441,6 +442,15 @@ static inline void bio_wouldblock_error(struct bio *bio) bio_endio(bio); } +/* + * Calculate number of bvec segments that should be allocated to fit data + * pointed by @iter. + */ +static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) +{ + return iov_iter_npages(iter, max_segs); +} + struct request_queue; extern int submit_bio_wait(struct bio *bio); From patchwork Sat Jan 9 16:03:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12008483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B82AC432C3 for ; Sat, 9 Jan 2021 16:08:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 227A523A04 for ; Sat, 9 Jan 2021 16:08:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726985AbhAIQIL (ORCPT ); Sat, 9 Jan 2021 11:08:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726913AbhAIQII (ORCPT ); Sat, 9 Jan 2021 11:08:08 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1482C0617A7; Sat, 9 Jan 2021 08:06:55 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id e25so11051015wme.0; Sat, 09 Jan 2021 08:06:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yD8zYkali7maLviDx2WTi8i72ihDIWnJKSJbAu6xfJ0=; b=Ts4OoM5bqISNbtmG5sPDsmpk3BTHhh/QjboJ9FjJbTRW6os44a6JWrPaQeIq9a4gzB ytPoZ/GW+c8ZRLyy3+gy3UFzrpFfiklRK34zsaVHar6VPLoh+t6didLQ79On2YaXPI61 pJYMvy+biZXWpv873DXVw0VcAA4BWzd8gzcFFTJKXfdRrqQ5DS5I/WBs9CIv7jzQeFMI Hq3iECPsMcVtJgXRMv4wn1o2Hlj3wnnkuWosFYpHehQe35KojPpv+ALObFOb9wY8RLyx lHcnE4uGyPIaA5yYJvbSruf6mQkaPTFCw8BRrqy91Lzpf7zZilpiDzluyMd0xwOv21xY uFYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yD8zYkali7maLviDx2WTi8i72ihDIWnJKSJbAu6xfJ0=; b=oeBGYqW+EAhOqFqgxPFX6WJCGqHZ+JDX1EsheS3j67f4MhBWYTIiiTY9KB5CDWa/QG X+tXP0vw+Jncwg8KrXpIyQnb4lt5YS+30VHVVA883M0jTkIx1ylyXFndEK92PXxam2mq sCQazQVcFUsJ9XgJ0vNrCpwmdleYMqeMHpiR1PAYODWze1ntqKMj8qGR8DnRuDBvmueD h+1WEhQUDR/P0eYGfBRM/7rMEMAwPgvy77c/dnRVcR/3RSuMOAv+efWW3nuxMJABSosr Jo9i+rJIFBmAbkkGea5ZPEDIjrkHS3mQQRyyHOhdCFY6PxTDvlsK4cR7sao3APTZcicS hjSQ== X-Gm-Message-State: AOAM53204fXKCtpTwPObRcJPb2FH/gWzxkmLYwDz8CGiVqwcdkYXJbfX P8dhy1JfCBMUi8UkTmutV+aBbiqLyusg13AY X-Google-Smtp-Source: ABdhPJyDJ7ivMuFntSJFW5N+/HuavTp+fJKVSfLb/Mv/WA3X5yhkivHrCndukCuTwkfjMzb7+EAADg== X-Received: by 2002:a1c:1d1:: with SMTP id 200mr7900279wmb.98.1610208414381; Sat, 09 Jan 2021 08:06:54 -0800 (PST) Received: from localhost.localdomain ([185.69.144.125]) by smtp.gmail.com with ESMTPSA id j9sm17403866wrm.14.2021.01.09.08.06.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 08:06:53 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v3 7/7] bio: don't copy bvec for direct IO Date: Sat, 9 Jan 2021 16:03:03 +0000 Message-Id: <69fef253b37fc44dd28c43398715e27cee5e0fe0.1610170479.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org The block layer spends quite a while in blkdev_direct_IO() to copy and initialise bio's bvec. However, if we've already got a bvec in the input iterator it might be reused in some cases, i.e. when new ITER_BVEC_FLAG_FIXED flag is set. Simple tests show considerable performance boost, and it also reduces memory footprint. Suggested-by: Matthew Wilcox Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- Documentation/filesystems/porting.rst | 9 ++++ block/bio.c | 67 ++++++++++++--------------- include/linux/bio.h | 5 +- 3 files changed, 42 insertions(+), 39 deletions(-) diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index c722d94f29ea..1f8cf8e10b34 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -872,3 +872,12 @@ its result is kern_unmount() or kern_unmount_array(). zero-length bvec segments are disallowed, they must be filtered out before passed on to an iterator. + +--- + +**mandatory** + +For bvec based itererators bio_iov_iter_get_pages() now doesn't copy bvecs but +uses the one provided. Anyone issuing kiocb-I/O should ensure that the bvec and +page references stay until I/O has completed, i.e. until ->ki_complete() has +been called or returned with non -EIOCBQUEUED code. diff --git a/block/bio.c b/block/bio.c index 9f26984af643..6f031a04b59a 100644 --- a/block/bio.c +++ b/block/bio.c @@ -960,21 +960,17 @@ void bio_release_pages(struct bio *bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(bio_release_pages); -static int __bio_iov_bvec_add_pages(struct bio *bio, struct iov_iter *iter) +static int bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { - const struct bio_vec *bv = iter->bvec; - unsigned int len; - size_t size; - - if (WARN_ON_ONCE(iter->iov_offset > bv->bv_len)) - return -EINVAL; - - len = min_t(size_t, bv->bv_len - iter->iov_offset, iter->count); - size = bio_add_page(bio, bv->bv_page, len, - bv->bv_offset + iter->iov_offset); - if (unlikely(size != len)) - return -EINVAL; - iov_iter_advance(iter, size); + WARN_ON_ONCE(BVEC_POOL_IDX(bio) != 0); + + bio->bi_vcnt = iter->nr_segs; + bio->bi_max_vecs = iter->nr_segs; + bio->bi_io_vec = (struct bio_vec *)iter->bvec; + bio->bi_iter.bi_bvec_done = iter->iov_offset; + bio->bi_iter.bi_size = iter->count; + + iov_iter_advance(iter, iter->count); return 0; } @@ -1088,12 +1084,12 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * This takes either an iterator pointing to user memory, or one pointing to * kernel pages (BVEC iterator). If we're adding user pages, we pin them and * map them into the kernel. On IO completion, the caller should put those - * pages. If we're adding kernel pages, and the caller told us it's safe to - * do so, we just have to add the pages to the bio directly. We don't grab an - * extra reference to those pages (the user should already have that), and we - * don't put the page on IO completion. The caller needs to check if the bio is - * flagged BIO_NO_PAGE_REF on IO completion. If it isn't, then pages should be - * released. + * pages. For bvec based iterators bio_iov_iter_get_pages() uses the provided + * bvecs rather than copying them. Hence anyone issuing kiocb based IO needs + * to ensure the bvecs and pages stay referenced until the submitted I/O is + * completed by a call to ->ki_complete() or returns with an error other than + * -EIOCBQUEUED. The caller needs to check if the bio is flagged BIO_NO_PAGE_REF + * on IO completion. If it isn't, then pages should be released. * * The function tries, but does not guarantee, to pin as many pages as * fit into the bio, or are requested in @iter, whatever is smaller. If @@ -1105,27 +1101,22 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { - const bool is_bvec = iov_iter_is_bvec(iter); - int ret; - - if (WARN_ON_ONCE(bio->bi_vcnt)) - return -EINVAL; + int ret = 0; - do { - if (bio_op(bio) == REQ_OP_ZONE_APPEND) { - if (WARN_ON_ONCE(is_bvec)) - return -EINVAL; - ret = __bio_iov_append_get_pages(bio, iter); - } else { - if (is_bvec) - ret = __bio_iov_bvec_add_pages(bio, iter); + if (iov_iter_is_bvec(iter)) { + if (WARN_ON_ONCE(bio_op(bio) == REQ_OP_ZONE_APPEND)) + return -EINVAL; + bio_iov_bvec_set(bio, iter); + bio_set_flag(bio, BIO_NO_PAGE_REF); + return 0; + } else { + do { + if (bio_op(bio) == REQ_OP_ZONE_APPEND) + ret = __bio_iov_append_get_pages(bio, iter); else ret = __bio_iov_iter_get_pages(bio, iter); - } - } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); - - if (is_bvec) - bio_set_flag(bio, BIO_NO_PAGE_REF); + } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); + } /* don't account direct I/O as memory stall */ bio_clear_flag(bio, BIO_WORKINGSET); diff --git a/include/linux/bio.h b/include/linux/bio.h index d8f9077c43ef..1d30572a8c53 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -444,10 +444,13 @@ static inline void bio_wouldblock_error(struct bio *bio) /* * Calculate number of bvec segments that should be allocated to fit data - * pointed by @iter. + * pointed by @iter. If @iter is backed by bvec it's going to be reused + * instead of allocating a new one. */ static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) { + if (iov_iter_is_bvec(iter)) + return 0; return iov_iter_npages(iter, max_segs); }