From patchwork Tue Feb 14 17:13:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140600 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37A1FC05027 for ; Tue, 14 Feb 2023 17:13:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 194A26B0078; Tue, 14 Feb 2023 12:13:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 144106B007B; Tue, 14 Feb 2023 12:13:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00C436B007D; Tue, 14 Feb 2023 12:13:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E2A316B007B for ; Tue, 14 Feb 2023 12:13:47 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B18ED16085F for ; Tue, 14 Feb 2023 17:13:47 +0000 (UTC) X-FDA: 80466544494.04.BD19F79 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id E71FCC000E for ; Tue, 14 Feb 2023 17:13:45 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ab+l4l18; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zSLOPyPBTyGpP+6+bR7yKU/VhAXOTSaAngOTYdMp3ic=; b=7r89XYxTiHBiptPfbzF1f6W/cxXEcojGmwsl/Kpo1gteW4SCGkkbtq3o3TZQ2gJsFAyY1Y 2xEHmS5ZKmGS/PciALLezzX/qH8P5pII2nvNQikf2MLrEEQoQov6ugtkcBB9Pe1B/JweLd kZVxQQieV6aZym+gmqI3K9DWY56/Uxk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Ab+l4l18; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394826; a=rsa-sha256; cv=none; b=ER/6cLzD5RoUQQF82B3lBqBAsEVTtifKgvaFnFqbpwL8cps0iwpX1HodpzSoUh+4w+Ql5I Pu1PuM/l5VBxL9OHVEq/jg4ZIw/4fOH3M9c9YOM3lbXHEg5NZXFfJEc4Zlvlz3bnc5TMGb 50/zNAna3B6NTgF9VTL2OYg9b0f3fbo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zSLOPyPBTyGpP+6+bR7yKU/VhAXOTSaAngOTYdMp3ic=; b=Ab+l4l18HMer0UkCotZtzJxCXsKRuUDnXLPb/4rHVHLifcBrFqsKXqxySP6cswV7sLkDck 2hs+qnVxlyLU3QJYtubGRRlmT0wBuJw1Op7reV82gEP/TjU2F2KTWj4BgNJQMVCacziW49 7+DxLuXi3QXLbzqB9TP0UUlm7mMqXQU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237-r6zz-Qt7OtqamK4CCCrB6g-1; Tue, 14 Feb 2023 12:13:41 -0500 X-MC-Unique: r6zz-Qt7OtqamK4CCCrB6g-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 095C2101A521; Tue, 14 Feb 2023 17:13:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D5A4492B03; Tue, 14 Feb 2023 17:13:35 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 01/17] mm: Pass info, not iter, into filemap_get_pages() Date: Tue, 14 Feb 2023 17:13:14 +0000 Message-Id: <20230214171330.2722188-2-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E71FCC000E X-Rspam-User: X-Stat-Signature: ga51k8qgtyh51bdodmmjhcsc5bh437ro X-HE-Tag: 1676394825-190680 X-HE-Meta: U2FsdGVkX18ZliY72Z/hMAAVjUM3giNOyHMn8YB6Mn1y93lSqfX0OE2SvqQM6Qrymy/O3fVhZz6LXs/JOoVBU6zrhLXxkqOzxasn6/VKb4fXnlNutuA2A87PVEQ7TMezvpW3rSkKuKifi8B4mFkNpJ2AgiNuV8A1YceKCy0NxrS3gTuKaT8mdUsmVe/Zx+vrhoZt6pD0+k3OTFqDng+tn0CyPBVhQgMt2smSx1Jx6AH6wDjpAStvETTJgVK9NFMg3ogwlarLZa8dsv56pek89KFxPHFKlTeeu46NpO+3btHBjCQKj5RFq5m1jd2I1AuPg/RdLTPfqLQiSxB0LU4f+IVZGb5O6j8Hl4fnXUsRBElqs9ArF/ESbTf8fJHV753ayRF1Nei7pP40t1bJbeMjQrCzasp1VY1duZgCykxtJ9XbyDEORCLyvLM9EMK6zVFElatnKOgzY909awpumSm5hhlSyP+TSKUdvnZ89iBC11KhunxngUqXY7mm0EoExymq1hCOpJnKpgt1cAwbFvmTh4eYxlKqZHVtri+ho+KQkew8HY2WuBhoPTld58z3rthTA29Ufz/TkUchql3ueys6RTJN4yLfjmjXQ8SRhWI1KEE4jVCPFWKkvp/YE2M1JhXgi7nL1Xs+6dPGAT1Q4Qp+MCY3YhPULJAcU77ExV+e29bmmmDHtYyD317snQmVBRSOp+Hp0iOwhb/ieXIyFEwFqk5QPiLaSRdTBOZP8HrZQ8NGicibdJdliD9IY0X8zNGtjC0ef72L7V4cG+cdH0XABfa36dxgDVHVghLs/G5LLImDu8cA7CjoQa65/IulvIYmXtrSto32ZqEDGtr0IW2yCHdcKR0I/56pYp0TtU7QDx3muz11aWmC8lSWmvJ0IqItApSX7DMlWeyaQPcU5EZP6kyqOS4mtslDtJA8x/XfG+ukGFQFuUZEhjhBOXsqPDu5JxENyaE9A/MmEX3zg3n i/HTVlhi vBXy9Po8P5bJEmwDaP9pkLy7tpAnE78KPtND9/G+BhohYjDhdDRq/OjbYHTp+vJF+rulSvkwaYpjiG0eLlYR8DoSBz5Q/raTVWHLee/pY59g+vdqesZIfzoliW7I6Uq1HwsQmuuAi+KaqlB1YfL5klc6oYNxsWnN8KavzsG+0ZYzWt7aM8d5Gm7ePNpVLrq6w4kVVzdLSBeXFFQwQdYWJB8vUBd0iHdM2w425Z7kOt3QpXiuzRWjqw/4LddZnBMXURhYuVq9jUy84e1MfBZTaGhwk5o+7ZbD8gRxyI/+Vp4ZyCbGJTxh7lYTHRscHVbKhUhStdiGS1I+I2elvmt85MZgJ593fHukp1eLCXmRQwdptw3+YYayZS1OYGEFCeL62JvcLKPlaw9SSHzb+Dtx6NtXYA821PT4Ajalt++4w6lS+gqFJVj1AF0xzlkNXNVHG5s0k1tQ9LTuWCFgYR5jWvn2FdrWzgHKWkMg9g4otaP2UsFqbR5ygH2DEYg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: filemap_get_pages() and a number of functions that it calls take an iterator to provide two things: the number of bytes to be got from the file specified and whether partially uptodate pages are allowed. Change these functions so that this information is passed in directly. This allows it to be called without having an iterator to hand. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: Christoph Hellwig cc: Matthew Wilcox cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org --- mm/filemap.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index c4d4ace9cc70..876e77278d2a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2440,21 +2440,19 @@ static int filemap_read_folio(struct file *file, filler_t filler, } static bool filemap_range_uptodate(struct address_space *mapping, - loff_t pos, struct iov_iter *iter, struct folio *folio) + loff_t pos, size_t count, struct folio *folio, + bool need_uptodate) { - int count; - if (folio_test_uptodate(folio)) return true; /* pipes can't handle partially uptodate pages */ - if (iov_iter_is_pipe(iter)) + if (need_uptodate) return false; if (!mapping->a_ops->is_partially_uptodate) return false; if (mapping->host->i_blkbits >= folio_shift(folio)) return false; - count = iter->count; if (folio_pos(folio) > pos) { count -= folio_pos(folio) - pos; pos = 0; @@ -2466,8 +2464,8 @@ static bool filemap_range_uptodate(struct address_space *mapping, } static int filemap_update_page(struct kiocb *iocb, - struct address_space *mapping, struct iov_iter *iter, - struct folio *folio) + struct address_space *mapping, size_t count, + struct folio *folio, bool need_uptodate) { int error; @@ -2501,7 +2499,8 @@ static int filemap_update_page(struct kiocb *iocb, goto unlock; error = 0; - if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, folio)) + if (filemap_range_uptodate(mapping, iocb->ki_pos, count, folio, + need_uptodate)) goto unlock; error = -EAGAIN; @@ -2577,8 +2576,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, return 0; } -static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, - struct folio_batch *fbatch) +static int filemap_get_pages(struct kiocb *iocb, size_t count, + struct folio_batch *fbatch, bool need_uptodate) { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; @@ -2588,7 +2587,7 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, struct folio *folio; int err = 0; - last_index = DIV_ROUND_UP(iocb->ki_pos + iter->count, PAGE_SIZE); + last_index = DIV_ROUND_UP(iocb->ki_pos + count, PAGE_SIZE); retry: if (fatal_signal_pending(current)) return -EINTR; @@ -2621,7 +2620,8 @@ static int filemap_get_pages(struct kiocb *iocb, struct iov_iter *iter, if ((iocb->ki_flags & IOCB_WAITQ) && folio_batch_count(fbatch) > 1) iocb->ki_flags |= IOCB_NOWAIT; - err = filemap_update_page(iocb, mapping, iter, folio); + err = filemap_update_page(iocb, mapping, count, folio, + need_uptodate); if (err) goto err; } @@ -2691,7 +2691,8 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, if (unlikely(iocb->ki_pos >= i_size_read(inode))) break; - error = filemap_get_pages(iocb, iter, &fbatch); + error = filemap_get_pages(iocb, iter->count, &fbatch, + iov_iter_is_pipe(iter)); if (error < 0) break; From patchwork Tue Feb 14 17:13:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58CC0C61DA4 for ; Tue, 14 Feb 2023 17:13:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDE986B0075; Tue, 14 Feb 2023 12:13:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8F016B0078; Tue, 14 Feb 2023 12:13:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B56DF6B007B; Tue, 14 Feb 2023 12:13:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A8A166B0075 for ; Tue, 14 Feb 2023 12:13:47 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7CBFF140D2B for ; Tue, 14 Feb 2023 17:13:47 +0000 (UTC) X-FDA: 80466544494.03.CEC2A05 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id B5B84A0002 for ; Tue, 14 Feb 2023 17:13:44 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dog3BdOh; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H0CTUnwfVEWUZBHu5fA2fJ8y0quxYsONFQfvaH+Rj5U=; b=W9cM2lsRWk5yJJT8HoZZPQTXruwy3RrVGTjhML3QBDHYdCNRf78UfBdgbAaSW+EP44Qydk fJpBT06wKcNpAPQef3oraSsww8K429qotEXvH8ZTGqMyV6W9Oru+4ID3ZXsb1Y0NuL78Ci HNPepXEvF+PhRTTWmLfCVuRj2AU4i5U= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dog3BdOh; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394824; a=rsa-sha256; cv=none; b=0h5e56tYVC/eWNOEJf6xladgy+y8LYSh3dXtiGTywSMq9vnw24RfwMc93mzDBkJRsyOxeb JllA2vlvipXr4DLYj/5fYnc0NmyywX9mnjspXm590XcLGHP92/D1aWsam3sBkUmTMOhwCL 18aDBDtKtPluJDG1HobwaVStcr68b5E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394824; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H0CTUnwfVEWUZBHu5fA2fJ8y0quxYsONFQfvaH+Rj5U=; b=dog3BdOhoHCFuBk9FxwgeGNA0Pfi8t2hZy5PVslr8XPTdzg07C+S53HzZ54vZxCpM3f9A9 fJZS2GaqxF01QYyNRsHrzOLQRBnlk2t4em+0oU5KGn1ndwT+Rf6UcqrzIiVuE1SPiBxK5P IsfIDi/WdtnGX6GGw6NFNlywiUg5KNo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-49-fW_ij3QZO56kIwrqyILfmQ-1; Tue, 14 Feb 2023 12:13:42 -0500 X-MC-Unique: fW_ij3QZO56kIwrqyILfmQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A9D5F1C08796; Tue, 14 Feb 2023 17:13:40 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1699140EBF6; Tue, 14 Feb 2023 17:13:38 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 02/17] splice: Add a func to do a splice from a buffered file without ITER_PIPE Date: Tue, 14 Feb 2023 17:13:15 +0000 Message-Id: <20230214171330.2722188-3-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Stat-Signature: bio66f679ccmmwmixq93aw1wcsbdy4n6 X-Rspam-User: X-Rspamd-Queue-Id: B5B84A0002 X-Rspamd-Server: rspam06 X-HE-Tag: 1676394824-267192 X-HE-Meta: U2FsdGVkX1/7bZcJVV5siysjCzsAiF10c3t4oeOjNkP74yONGhqQFR23Pi0y6IMw6nzQO9UEBiqjCohXBt3Fu/1l2xD+Z/i8nVbIO8+nsTRe4fWzgYBf0WrQL71c5kbSdtlN7mdtLAT7523uu6kDx8JQJQ8SCaATtAwql3iu0fZXLBIuYAiP4jsdMgxtDC90Q7yS5LzOK5tuHU9Y8krJ1WZUkWqowAwwLkghE82ukehfGhCyytRR3N9yHvLiDIF6d3esSZixcxFEmvUNHCn8KJI9RtisaLQYJJbIbvfzjREmpBVHXcnEfopz1o7dc+yKprFMO4DP8OFcF8GdK1DPqv3iLVnkAQROHVcG4P+eb9Ui/gd5NtMdt5V0C2MIsB/3WIgP/xSU0HPg14Qgj+Rb3CdUE8kCKOUK8he9VUNnf1cs5+K4ROIi0/ZoBqC8zGzw6AiPxpK8IumgYOa6rqg3ZZOa4D3PwjA0yuRVMgtuie4HJzel3MComApVQlgEzy8589l6Ef8IbVTSVc7BGp7CpznpwUo6olT+IN1YnxKWuH9FpNEIdB7CBD1MTixnoNa56rgyEJDp1Q+ZDdOgEqIk5ugz8phLuz4E9XLOii3vXY/0e3xAi9cmjPcZiyr1ErbXAg/LLFcW5IwJizu1GA0KA974q7TOzQN5YmCa5DAkKsD3IsoT24+VTLIXR9UNcUgjVzCH5jckHhiiQkFe/3Ocf4VDVBCJoW+5G20BOfFMedbHkuBevkRV2x49HRPSz8RA2EAEPfrYoQeHfXIQCE/r7EPKs1SuQ91X5WU9xI2ZQt0wDtihCnois0nuFqM7q5l6Mp29spiSGD8n9LKyFGt8Pm6pE6cd/3KktfaZVG65x5JAbArD7vOkYWBlJ4AdIE07S+BXhH2mKIdmu6kcNF7+veJhNydiioRQuMsvBiw7dBfT7pEiOabGtxC2CeqeYNcDZ7rfrO7a74id1eD6jIz YGqgD37I l+HqP5ObYd18+MWwWppE4IkqLN1IfyGHQj/NXQNbqbQSxTw02fQ2JGROydW/pdfBqzc5FceOyNa5ctXqQImZdr7vxCrHv3slREQo8X9NxyBYXuxRHJtknzPlUeHjopVpvz2gPOHc/x80PeaoMeRhF74LRLKhgOSKpy6CVrU6ZbpBt8YkkfJ2uWlwtZValE7/fBI7xUxxOLkPA1YH4gXy9Q/ssXdCVzMd5eFgjgApjnPKubD/Fk4le8n2cYzhv9f8cZn5gIoSntzqZAI04S78BrsnGr+aw2cajJPv9Z0n3jIQn9c8rT2zhSvGZfP4qPbVUZrpolrPe138gN+O3KUWISYbiq1UKQZeWxLRHM9N1KogpIrPpRB01pl9QDE/MUPxdXrp0gIt4m6EnH+zvrsxl9UtwHdt0/2hiQDISqJFIHGnznUXd/Frll6kjMzCH46lWVmdf7gUnJHCuLEUUv/kWKxPsu4NxooxgEndnrGLjukbhWmMzZSQ0QoE5u/YyRle267fn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide a function to do splice read from a buffered file, pulling the folios out of the pagecache directly by calling filemap_get_pages() to do any required reading and then pasting the returned folios into the pipe. A helper function is provided to do the actual folio pasting and will handle multipage folios by splicing as many of the relevant subpages as will fit into the pipe. The code is loosely based on filemap_read() and might belong in mm/filemap.c with that as it needs to use filemap_get_pages(). Signed-off-by: David Howells cc: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org --- Notes: ver #14) - Rename to filemap_splice_read(). - Create a helper, pipe_head_buf(), to get the head buffer. - Use init_sync_kiocb(). - Move to mm/filemap.c. - Split the implementation of filemap_splice_read() from the patch to make generic_file_splice_read() use it and direct_splice_read(). include/linux/fs.h | 3 ++ mm/filemap.c | 128 +++++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 6 +++ 3 files changed, 137 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index c1769a2c5d70..28743e38df91 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3163,6 +3163,9 @@ ssize_t vfs_iocb_iter_write(struct file *file, struct kiocb *iocb, struct iov_iter *iter); /* fs/splice.c */ +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, diff --git a/mm/filemap.c b/mm/filemap.c index 876e77278d2a..8c7b135c8e23 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -42,6 +42,8 @@ #include #include #include +#include +#include #include #include #include "internal.h" @@ -2842,6 +2844,132 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) } EXPORT_SYMBOL(generic_file_read_iter); +/* + * Splice subpages from a folio into a pipe. + */ +size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, + struct folio *folio, loff_t fpos, size_t size) +{ + struct page *page; + size_t spliced = 0, offset = offset_in_folio(folio, fpos); + + page = folio_page(folio, offset / PAGE_SIZE); + size = min(size, folio_size(folio) - offset); + offset %= PAGE_SIZE; + + while (spliced < size && + !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); + + *buf = (struct pipe_buffer) { + .ops = &page_cache_pipe_buf_ops, + .page = page, + .offset = offset, + .len = part, + }; + folio_get(folio); + pipe->head++; + page++; + spliced += part; + offset = 0; + } + + return spliced; +} + +/* + * Splice folios from the pagecache of a buffered (ie. non-O_DIRECT) file into + * a pipe. + */ +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct folio_batch fbatch; + struct kiocb iocb; + size_t total_spliced = 0, used, npages; + loff_t isize, end_offset; + bool writably_mapped; + int i, error = 0; + + init_sync_kiocb(&iocb, in); + iocb.ki_pos = *ppos; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + + folio_batch_init(&fbatch); + + do { + cond_resched(); + + if (*ppos >= i_size_read(file_inode(in))) + break; + + iocb.ki_pos = *ppos; + error = filemap_get_pages(&iocb, len, &fbatch, true); + if (error < 0) + break; + + /* + * i_size must be checked after we know the pages are Uptodate. + * + * Checking i_size after the check allows us to calculate + * the correct value for "nr", which means the zero-filled + * part of the page is not copied back to userspace (unless + * another truncate extends the file - this is desired though). + */ + isize = i_size_read(file_inode(in)); + if (unlikely(*ppos >= isize)) + break; + end_offset = min_t(loff_t, isize, *ppos + len); + + /* + * Once we start copying data, we don't want to be touching any + * cachelines that might be contended: + */ + writably_mapped = mapping_writably_mapped(in->f_mapping); + + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + size_t n; + + if (folio_pos(folio) >= end_offset) + goto out; + folio_mark_accessed(folio); + + /* + * If users can be writing to this folio using arbitrary + * virtual addresses, take care of potential aliasing + * before reading the folio on the kernel side. + */ + if (writably_mapped) + flush_dcache_folio(folio); + + n = splice_folio_into_pipe(pipe, folio, *ppos, len); + if (!n) + goto out; + len -= n; + total_spliced += n; + *ppos += n; + in->f_ra.prev_pos = *ppos; + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + goto out; + } + + folio_batch_release(&fbatch); + } while (len); + +out: + folio_batch_release(&fbatch); + file_accessed(in); + + return total_spliced ? total_spliced : error; +} + static inline loff_t folio_seek_hole_data(struct xa_state *xas, struct address_space *mapping, struct folio *folio, loff_t start, loff_t end, bool seek_data) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..6d4ca98f3844 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -794,6 +794,12 @@ struct migration_target_control { gfp_t gfp_mask; }; +/* + * mm/filemap.c + */ +size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, + struct folio *folio, loff_t fpos, size_t size); + /* * mm/vmalloc.c */ From patchwork Tue Feb 14 17:13:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A8AC64ED6 for ; Tue, 14 Feb 2023 17:13:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 899CA6B007E; Tue, 14 Feb 2023 12:13:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 84A3C6B0080; Tue, 14 Feb 2023 12:13:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EA236B0081; Tue, 14 Feb 2023 12:13:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5D85E6B007E for ; Tue, 14 Feb 2023 12:13:52 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 017A31401F5 for ; Tue, 14 Feb 2023 17:13:51 +0000 (UTC) X-FDA: 80466544704.18.8353A0D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 43EA880011 for ; Tue, 14 Feb 2023 17:13:50 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ECxwjEg7; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Wrfy5DSUofzooKuydOZLN29ccYXYPtSMXDanySrmvRI=; b=HO8J+s82TREJJM9B7F4JpPnCxe2ze+t+0aY1cNS3ps/4bdTbvZSGrIiVW5pXr5+AkmtFj4 m3b8erUBpW3jI8Du2ts7n1zR63Qh4GFnNXxsN1u09XzG3TAL9/PiMH5H6/wAeGUOjWbX4l N+RJi1X9/8D2MmvXYrVlAGFIUb3DJX0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ECxwjEg7; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394830; a=rsa-sha256; cv=none; b=PM92b/VdaP6TMBLDtfawjg/PNzaODuh1dFHHYLUKKgKlYip2UQV7DxmfeiXcCwV4Zf7xhJ bBrHmMr9Bfj7jwXsgX37jCqMkpHxX0jL+dyRYJNDdBZRuhLTH2VG0AAmMXZCnR6dU/bpfH CAfuC5+fXun8LFO7a5EjgG+sgq6COjw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394829; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wrfy5DSUofzooKuydOZLN29ccYXYPtSMXDanySrmvRI=; b=ECxwjEg72Aygmn9QqzZA8toC1d6o9uldlZEsTn4ThgDYTO+eQLPgsyqeNTS4s3J0h6tURC mOTcphw7r0NvOEu69u+JhFjJt1qYFuqdSInzlJalRL8NxWN1k0RRN+UtD0+olnv5ZVajsv S5522d92MnFjjDKCKaKhZlc3cMtCryU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-593-llfpyhjxOKyRI8fnUf8UoA-1; Tue, 14 Feb 2023 12:13:44 -0500 X-MC-Unique: llfpyhjxOKyRI8fnUf8UoA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6EDA11C08795; Tue, 14 Feb 2023 17:13:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 53202140EBF6; Tue, 14 Feb 2023 17:13:41 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com, Christoph Hellwig , John Hubbard Subject: [PATCH v14 03/17] splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE Date: Tue, 14 Feb 2023 17:13:16 +0000 Message-Id: <20230214171330.2722188-4-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 43EA880011 X-Stat-Signature: i3f464w7d9w6kozqd6wa9sdg5b6uejdc X-HE-Tag: 1676394830-188795 X-HE-Meta: U2FsdGVkX1+aLYzUyYqdZc+XgSq4AO3fo0eC3QuJIJs0CeTaj52yzcb3IqgoxJXT36T7KxgeWX4vTZzIEqnrTSEZmoQFO5wimmaK6Z3N/dpeW7Tac01yDuxeTCAarN6KpJwmAM4Gw0kvcqxxo31vWJ4IOcqQjCR5pUt2/OSJBQozaslQF1Y+s0emZWYwOoI/2wF7+PtENUxF/ts8izqIFRiw0rjLR9ytnnQmEcn6RChdAvr4FEYN0hIfHGJAhdsjNe9lKccO1Z88xoq5AGGAHta7mwBjucV2HuE/E2KdqnG3ugd65zVUzpawJ6xORP7w5efFklLMPvLPHbsfgbE0qFkjiKi7oKI4nl+Sh4UUVboOqvgcPFo+C5VhsT/La8ZxNJmMNfHwTP4OBvWwi8wUJYRSbtujFmqamiWk347hlPkDmgV0jwYOzOp3X7opSlvfAg9mPgVZeT9L1urXcMzuva/K/S+WvO2Sz09KMJrRB871kRKKEOV/EzaEwnF3wXMSz+t64KP4BdjYrm1p7WBUnsHd05ldyT/sXFl+Jod7jt8PV7znfs3+c8XKkt1xAzimB86S9A5G4E0AxBnBnyEGan3OUSFZLAKb9mZdpMoJES2//pH90goyxaqvRWFfbYy9P+4A6qMmEjLgocjaLwrEUIjAqDy9gpk6zKBe+yb1D2DZWI/0sThujYmqyYiWMOYJHW/0d1cgXFZc1Pb0FfYYSUNpgLBO5/kdLZ0RyLPbQJwlIqpcU9Cz9HpdneT1Ghcc7LLbXlITju0Snqs0cIEMotNZe3ikXFqEdI2BTbha9cvT187UDrEI2rk0VuRBIRm+1m00ZaBhidd7ZQl9CXrETZYAwjZ88K0aoWPGZd8sw7cnDNaPQD0EjpvbKQfc9iCIxcu4Z0rLJJ8v1qysj1R7NRa7CuV9bZosgX6KDrJXmCFuecE1+X0UW2sIWjvqSdGp2uaOrJvWTPtaTF/pMDv QXsmCSum iS3fXON5FcPN5BMUdmR/NqHDdP9hpNt1syLGN+O4zFzXLSrET9J53l7LVBprHaCVIG5dUWUksFKwSKEKZ51KAimdjnV+nO0EaojAn4qEjiB1GDDjAO8FJ5O+XP+mG9jMdBQzQ8XjK0K7/GlALfqrFEaSBWhsBXS69hbN/CuNRHwyLb021H/QGbcio7UcagPz6TrH4cAzTZCxuO4zHN/hLXzYQMhmcR4YYlD/2+mxH1dD/OLCoY0BesmsKJKGjabVHYWi1iyrnA63is9I2OdD0ybzzOYysaZ9Dllo7QO3YowSFsqMYN/77HWQCHRVeVl7yLZrPn0fMQhyhGFp2cnBr+V9JQ0ZyKljgj2uUJs3fpiNkPh5LZ3YP0XBX/EYpT8EwdMILgrCXEgyqwUWC8ydnpBpjCeRhuhrvoF82S5RzC3kT3vUWEAnN1J598xCw9eSJpASxLtI7gDMeCL4roJlJI92QuolPVs7H6664e3AD1teiyN2GMZLR1w9iPvUziO+yo9zcNRMVCyPrMthkR08q9qw/CSA1BpFKMpTf8eEMvvj5+IVY6wls9qnPH0jnJFFnguUMPHVX0H4MrXplrO0EoUHysuHKC7gPjwZslF7ruIi10DiaOLJOoGvo/66WlFsOHn3ngOM4yj142L4nCiMIsNyyfoZrMPchISpPxTJBzNNEL7VR7EdB6t8Qxj4RD5xn3NoKEyYIGqWsxFcR7GtIX21R1CVqd/PRIJBNpZz0qciiQLUeZ17Z07eOmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement a function, direct_file_splice(), that deals with this by using an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't free its buffers when reverted. The function bulk allocates all the buffers it thinks it is going to use in advance, does the read synchronously and only then trims the buffer down. The pages we did use get pushed into the pipe. This fixes a problem with the upcoming iov_iter_extract_pages() function, whereby pages extracted from a non-user-backed iterator such as ITER_PIPE aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to shorten the iterator to just the bufferage it is going to use - which has the side-effect of freeing the excess pipe buffers, even though they're attached to a bio and may get written to by DMA (thanks to Hillf Danton for spotting this[1]). This then causes memory corruption that is particularly noticable when the syzbot test[2] is run. The test boils down to: out = creat(argv[1], 0666); ftruncate(out, 0x800); lseek(out, 0x200, SEEK_SET); in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW); sendfile(out, in, NULL, 0x1dd00); run repeatedly in parallel. What I think is happening is that ftruncate() occasionally shortens the DIO read that's about to be made by sendfile's splice core by reducing i_size. This should be more efficient for DIO read by virtue of doing a bulk page allocation, but slightly less efficient by ignoring any partial page in the pipe. Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com Signed-off-by: David Howells cc: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1] Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2] --- Notes: ver #14) - Use alloc_pages_bulk_array() rather than alloc_pages_bulk_list(). - Use release_pages() rather than a loop calling __free_page(). - Rename to direct_splice_read(). - Don't call from generic_file_splice_read() yet. ver #13) - Don't completely replace generic_file_splice_read(), but rather only use this if we're doing a splicing from an O_DIRECT file fd. fs/splice.c | 92 +++++++++++++++++++++++++++++++++++++++ include/linux/fs.h | 3 ++ include/linux/pipe_fs_i.h | 20 +++++++++ lib/iov_iter.c | 6 --- 4 files changed, 115 insertions(+), 6 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 5969b7a1d353..4c6332854b63 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -282,6 +282,98 @@ void splice_shrink_spd(struct splice_pipe_desc *spd) kfree(spd->partial); } +/* + * Splice data from an O_DIRECT file into pages and then add them to the output + * pipe. + */ +ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct iov_iter to; + struct bio_vec *bv; + struct kiocb kiocb; + struct page **pages; + ssize_t ret; + size_t used, npages, chunk, remain, reclaim; + int i; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + npages = DIV_ROUND_UP(len, PAGE_SIZE); + + bv = kzalloc(array_size(npages, sizeof(bv[0])) + + array_size(npages, sizeof(struct page *)), GFP_KERNEL); + if (!bv) + return -ENOMEM; + + pages = (void *)(bv + npages); + npages = alloc_pages_bulk_array(GFP_USER, npages, pages); + if (!npages) { + kfree(bv); + return -ENOMEM; + } + + remain = len = min_t(size_t, len, npages * PAGE_SIZE); + + for (i = 0; i < npages; i++) { + chunk = min_t(size_t, PAGE_SIZE, remain); + bv[i].bv_page = pages[i]; + bv[i].bv_offset = 0; + bv[i].bv_len = chunk; + remain -= chunk; + } + + /* Do the I/O */ + iov_iter_bvec(&to, ITER_DEST, bv, npages, len); + init_sync_kiocb(&kiocb, in); + kiocb.ki_pos = *ppos; + ret = call_read_iter(in, &kiocb, &to); + + reclaim = npages * PAGE_SIZE; + remain = 0; + if (ret > 0) { + reclaim -= ret; + remain = ret; + *ppos = kiocb.ki_pos; + file_accessed(in); + } else if (ret < 0) { + /* + * callers of ->splice_read() expect -EAGAIN on + * "can't put anything in there", rather than -EFAULT. + */ + if (ret == -EFAULT) + ret = -EAGAIN; + } + + /* Free any pages that didn't get touched at all. */ + reclaim /= PAGE_SIZE; + if (reclaim) { + npages -= reclaim; + release_pages(pages + npages, reclaim); + } + + /* Push the remaining pages into the pipe. */ + for (i = 0; i < npages; i++) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + + chunk = min_t(size_t, remain, PAGE_SIZE); + *buf = (struct pipe_buffer) { + .ops = &default_pipe_buf_ops, + .page = bv[i].bv_page, + .offset = 0, + .len = chunk, + }; + pipe->head++; + remain -= chunk; + } + + kfree(bv); + return ret; +} + /** * generic_file_splice_read - splice data from file to a pipe * @in: file to splice from diff --git a/include/linux/fs.h b/include/linux/fs.h index 28743e38df91..551c9403f9b3 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3166,6 +3166,9 @@ ssize_t vfs_iocb_iter_write(struct file *file, struct kiocb *iocb, ssize_t filemap_splice_read(struct file *in, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); +ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h index 6cb65df3e3ba..d2c3f16cf6b1 100644 --- a/include/linux/pipe_fs_i.h +++ b/include/linux/pipe_fs_i.h @@ -156,6 +156,26 @@ static inline bool pipe_full(unsigned int head, unsigned int tail, return pipe_occupancy(head, tail) >= limit; } +/** + * pipe_buf - Return the pipe buffer for the specified slot in the pipe ring + * @pipe: The pipe to access + * @slot: The slot of interest + */ +static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, + unsigned int slot) +{ + return &pipe->bufs[slot & (pipe->ring_size - 1)]; +} + +/** + * pipe_head_buf - Return the pipe buffer at the head of the pipe ring + * @pipe: The pipe to access + */ +static inline struct pipe_buffer *pipe_head_buf(const struct pipe_inode_info *pipe) +{ + return pipe_buf(pipe, pipe->head); +} + /** * pipe_buf_get - get a reference to a pipe_buffer * @pipe: the pipe that the buffer belongs to diff --git a/lib/iov_iter.c b/lib/iov_iter.c index f9a3ff37ecd1..47c484551c59 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -186,12 +186,6 @@ static int copyin(void *to, const void __user *from, size_t n) return res; } -static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, - unsigned int slot) -{ - return &pipe->bufs[slot & (pipe->ring_size - 1)]; -} - #ifdef PIPE_PARANOIA static bool sanity(const struct iov_iter *i) { From patchwork Tue Feb 14 17:13:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8435C05027 for ; Tue, 14 Feb 2023 17:13:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95D006B007D; Tue, 14 Feb 2023 12:13:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 932F56B007E; Tue, 14 Feb 2023 12:13:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FE0D6B0080; Tue, 14 Feb 2023 12:13:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 712CD6B007D for ; Tue, 14 Feb 2023 12:13:51 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D83FCA0C2B for ; Tue, 14 Feb 2023 17:13:50 +0000 (UTC) X-FDA: 80466544620.29.BB5B3A4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 291EC40007 for ; Tue, 14 Feb 2023 17:13:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=b32oe4v7; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oNTxwJtD7ZehgoJ1QPpdyGLg4+saLDUsiyn+vcqXfIw=; b=bDBNRcUucVJMNmt5CWGDxqTxyi8Hi4bdsG57p4bOiThpcWbcegbkFug1l6gxD8k/Qg/5ni hojBNp8ehHO2+8nCYBmkBLVYqILAVSo2Fif+1EwTXGmaex+DkiL+wv41bVCN/VASXCxgpf XPg5UrwcDcQvIMLwLKEPF9Qt2lXnyWE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=b32oe4v7; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394829; a=rsa-sha256; cv=none; b=kA+fE+4aKUN9gIvmtpLMt4ZD6fPh0v1ZS9y4SvwbP/WooNI5/JokpL93FxoeNJ9Ic+AGDt Xt+j1JVuHlbU754l3FuA5Uc/y+JqVIVFM/eVBNYEzTozgDV7rgbE+oWgRVNQ6QPjIc80m1 uTNqE7OUduzgIhilIrbiXxK8C2jh8lM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oNTxwJtD7ZehgoJ1QPpdyGLg4+saLDUsiyn+vcqXfIw=; b=b32oe4v7NqgP1ggI001c52KSDFSHnlOrSZmDw2fOIzRQjeUp3dlwp4eD2Kk7/i/6icRiT/ rU+5GFjVTDT/8/awApJ2/689NnJ3m0Ef7XnEaoYem2xICLD6l2o41UbqLtMwxlIixT+8my JvV/W4BKIJTw/J4KNs+Nr6cZVuGhkAg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-344-D5m4tvNWOIeGlA5oelilFg-1; Tue, 14 Feb 2023 12:13:47 -0500 X-MC-Unique: D5m4tvNWOIeGlA5oelilFg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 69FDE800050; Tue, 14 Feb 2023 17:13:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2684F2166B26; Tue, 14 Feb 2023 17:13:44 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Daniel Golle , Guenter Roeck , Christoph Hellwig , John Hubbard , Hugh Dickins Subject: [PATCH v14 04/17] shmem: Implement splice-read Date: Tue, 14 Feb 2023 17:13:17 +0000 Message-Id: <20230214171330.2722188-5-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: sjo3qxq6qcqasz54h9x1tp3za79mmpzk X-Rspamd-Queue-Id: 291EC40007 X-HE-Tag: 1676394829-976194 X-HE-Meta: U2FsdGVkX1/LlzTsegtE594El+0f4FeKCuU9x9RIdCx7t2xIWTJcrK/2wSoNdHhS4Kp3biURWLgo3os7sA+8lqJM2iHos9KHsNfUxoryXbyYTS5i7p4lAXunoIoEuDxUNVpzeUritNE5MfbS/rALcLFrnKO6ufCbOrCRuPKoFR6/96F2H+2y2DQYq4y2vV6tp2SDBurfGpIQytdn8ocXaz5WED9DTiR/TIn2NAASiD6sQItOIYVA0ylWK1RPXlUPKcs3b8XGWzifpYbL7FGI9BKRFB/nh6G0Pg1spZa4LlmgAR7aZ+h1w9+SZaqIb5983nLzZyx+3AQ1jK7KN8lKGcwvt6KHOSqqFXa/ANVwIsdE2l+Jjq8i9HrMWuh2rlnRk01LB3BzIucadJfNW69xSKKWSbYwRAiuPiqxtPpfCCrmmpcMD5A7dJN9CsWOqrb8psSCghEhbtwSaBAcLcUQhu2/I6ET8NEldCYjRR68qV5J7w7XODMBeUZ3yi3qUg0lnuhpwodSAez/YlpVTnMH2G7fc4f7WrEaKbNMaRmvr8vzaUtaHUhWE6R8PfFn+pLY58IulOLuQSDTHKipw+k+dPsK0VPgBbMt4zChody55Ght5QCldZ5I6Ao+7yjOSmeKSmM4I4VAfA9D+x5Wid17tSANLIq1TG4ZggtlYatv/zFBaZAX5UG10SrmDVPgAdcges7likdrDtJesUYRYPIdxWG3jIhDcrbfQMlnPPG/c9x7mK830tf7bIepk48bCzs6ijTcLpgTRasexehWXzwOi4gRJPCHk52GbKNmKD2Idy3JHZXeZ+/LdSvaB65ICmowf8H6bQ++A/xOqGJn6cK1gPhJhrgBZDdRd8uuB6VAI30YTemcVHYdBT3DppJ16MVS9rT6UYltzO4lzDcQGhIjSMK3sDXQodt85k9QxNFmyh7DqlFtprIAghisqCEGboKIWJog4hqbekZMciKQhhp maC7CYNC r5RbAt0CXBcS1fVV8uo9o+J9hchVMSeqG2H5DrGRBKR5bZWtmRTxT/asK6lfWcW30q08Nk6Chvu3j2vbrxPJ0qKF8IoqNL/pWNjaSYePeOioDt/DZXT7xnS/Mbo0IjZtUWTpTMy8+7PAgU0znJfDOOyGEDs4Yd9AR4oNjRknUUlm6+MLOGlPqGmV2gJ8u70uSdPj338fsJhTCQ4npRahWU9RklG6LVeg01lskTAhAKnW/+jtsJAVV90iYYuxSk1/qwIRUJWBO26eZcO4WtL2G35xKcyRhzYcVQnWRS1zEQYrN9jtowxB97bPcjfmyAfsUHIKpUZd+GhH9ol07KKepGETzBHWSFnuW1P3H7owCPR9Bf/IEuE4OhZmoEf9UtYVRpRCIXYSc7qllnXx0IG2TMNc+r5OCScEp2NnvqPrcXUYt3jAmZFYCoeB2yWt+wRZzYL3XRa5JypFlVb7jswlTDZgFb2a4rjfPg6+2Zl4qOJveN/ZgnmQ9z/QIu0IwgkjwGgwJ3xCxMLoHHOLOS1YmjC5SJTHVn3OpmJRpaatVhTxDsBhIPev2RhF1lA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new filemap_splice_read() has an implicit expectation via filemap_get_pages() that ->read_folio() exists if ->readahead() doesn't fully populate the pagecache of the file it is reading from[1], potentially leading to a jump to NULL if this doesn't exist. shmem, however, (and by extension, tmpfs, ramfs and rootfs), doesn't have ->read_folio(), Work around this by equipping shmem with its own splice-read implementation, based on filemap_splice_read(), but able to paste in zero_page when there's a page missing. Signed-off-by: David Howells cc: Daniel Golle cc: Guenter Roeck cc: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: Hugh Dickins cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/Y+pdHFFTk1TTEBsO@makrotopia.org/ [1] --- mm/shmem.c | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 123 insertions(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index 0005ab2c29af..7145a5345f4d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2711,6 +2711,128 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) return retval ? retval : error; } +static bool zero_pipe_buf_try_steal(struct pipe_inode_info *pipe, + struct pipe_buffer *buf) +{ + return false; +} + +static const struct pipe_buf_operations zero_pipe_buf_ops = { + .release = generic_pipe_buf_release, + .try_steal = zero_pipe_buf_try_steal, + .get = generic_pipe_buf_get, +}; + +static size_t splice_zeropage_into_pipe(struct pipe_inode_info *pipe, + loff_t fpos, size_t size) +{ + size_t offset = fpos & ~PAGE_MASK; + + size = min_t(size_t, size, PAGE_SIZE - offset); + + if (!pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + + *buf = (struct pipe_buffer) { + .ops = &zero_pipe_buf_ops, + .page = ZERO_PAGE(0), + .offset = offset, + .len = size, + }; + get_page(buf->page); + pipe->head++; + } + + return size; +} + +static ssize_t shmem_file_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct inode *inode = file_inode(in); + struct address_space *mapping = inode->i_mapping; + struct folio *folio = NULL; + size_t total_spliced = 0, used, npages, n, part; + loff_t isize; + int error = 0; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + + do { + if (*ppos >= i_size_read(inode)) + break; + + error = shmem_get_folio(inode, *ppos / PAGE_SIZE, &folio, SGP_READ); + if (error) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + + if (folio_test_hwpoison(folio)) { + error = -EIO; + break; + } + } + + /* + * i_size must be checked after we know the pages are Uptodate. + * + * Checking i_size after the check allows us to calculate + * the correct value for "nr", which means the zero-filled + * part of the page is not copied back to userspace (unless + * another truncate extends the file - this is desired though). + */ + isize = i_size_read(inode); + if (unlikely(*ppos >= isize)) + break; + part = min_t(loff_t, isize - *ppos, len); + + if (folio) { + /* + * If users can be writing to this page using arbitrary + * virtual addresses, take care about potential aliasing + * before reading the page on the kernel side. + */ + if (mapping_writably_mapped(mapping)) + flush_dcache_folio(folio); + folio_mark_accessed(folio); + /* + * Ok, we have the page, and it's up-to-date, so we can + * now splice it into the pipe. + */ + n = splice_folio_into_pipe(pipe, folio, *ppos, part); + folio_put(folio); + folio = NULL; + } else { + n = splice_zeropage_into_pipe(pipe, *ppos, len); + } + + if (!n) + break; + len -= n; + total_spliced += n; + *ppos += n; + in->f_ra.prev_pos = *ppos; + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + break; + + cond_resched(); + } while (len); + + if (folio) + folio_put(folio); + + file_accessed(in); + return total_spliced ? total_spliced : error; +} + static loff_t shmem_file_llseek(struct file *file, loff_t offset, int whence) { struct address_space *mapping = file->f_mapping; @@ -3929,7 +4051,7 @@ static const struct file_operations shmem_file_operations = { .read_iter = shmem_file_read_iter, .write_iter = generic_file_write_iter, .fsync = noop_fsync, - .splice_read = generic_file_splice_read, + .splice_read = shmem_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = shmem_fallocate, #endif From patchwork Tue Feb 14 17:13:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B601C61DA4 for ; Tue, 14 Feb 2023 17:14:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDA4E6B0080; Tue, 14 Feb 2023 12:14:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E8A336B0081; Tue, 14 Feb 2023 12:14:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D79356B0082; Tue, 14 Feb 2023 12:14:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C81816B0080 for ; Tue, 14 Feb 2023 12:14:06 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9E7F8C0E4E for ; Tue, 14 Feb 2023 17:14:06 +0000 (UTC) X-FDA: 80466545292.03.4C8F44D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id DAD651C0029 for ; Tue, 14 Feb 2023 17:14:03 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MA2Tp0y4; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394844; a=rsa-sha256; cv=none; b=qxlxXO+5kWlZVgneytXyl4Aiy7aUCpJcTF/aJxHrowlqPZYVV1z2xwm0TJHyYU/gbDGfgn 8dIAObkxEjIJpW5kD8ioltMslGQ+a7+mr5KXD5BAZL7TSeRWbtWx37focxa7rNwazMtV2S BOLARWVHqPT/Pkz+mLpyCckKPwqfGHI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MA2Tp0y4; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394844; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LVjIujk8d1N0k6RGniY3UP+0o4vFHY30HdwwNIbIhak=; b=euZFv6WBM6JchXkz4Jgbv0CGrGVTU8XX281ytRagfsMZRBdIWf9u1Af46KFCBN8+776BYS rUCrfAT/+oVRudMezu0tPT894lA1tr82xq+QZWuTMAuPgaWZxCyB2YP48bBEpWBRb06nJJ pbYcyXrmYOOw3MrtT6KcWxo79ZNdZXI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LVjIujk8d1N0k6RGniY3UP+0o4vFHY30HdwwNIbIhak=; b=MA2Tp0y4F6kIgwUR6MJz0HS9ffds8A1m3jEaav9Jj+wqi2zJd/v719q2TeR3tm8nKY50Ut HhvvM/2V3IkPs0zUVv4+Q17Ev0X1hJefoF8cCFLzO4MOKRShGH59LtNBlRTErsH8ZlBdCA e/X3QsjeoH7S1U0rjAZD9p+SOljZF9A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-608-VwTllVNdOty5IfHcJbDwmw-1; Tue, 14 Feb 2023 12:14:00 -0500 X-MC-Unique: VwTllVNdOty5IfHcJbDwmw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4BBF285CCE0; Tue, 14 Feb 2023 17:13:59 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0784C40C10FA; Tue, 14 Feb 2023 17:13:46 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard , Miklos Szeredi , linux-unionfs@vger.kernel.org Subject: [PATCH v14 05/17] overlayfs: Implement splice-read Date: Tue, 14 Feb 2023 17:13:18 +0000 Message-Id: <20230214171330.2722188-6-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspam-User: X-Rspamd-Queue-Id: DAD651C0029 X-Rspamd-Server: rspam01 X-Stat-Signature: uaeib3d95opiabegiqt61zgf6gqrodby X-HE-Tag: 1676394843-372679 X-HE-Meta: U2FsdGVkX19GysZ1TV/lxMJWUm5gkToCZOZj5hIkAKNJY60J8pz78O8HAq1oq2L819Y1qib0bHA35LgpRKzUKO2AZIBlUH+hRap5ZIT5RGdMWGtkR4MlPjA0gUfjHs4hkZSvxGX8OvdtgN7bhUGumV1bUevNwp2Aip1rMA6IiaS/d4dzGbnWAw8uaFgw6f/HD928Jg+YPzjXJXdFOs5TYIhnaLjY4wxwGclsImHgy6tHTqOGqO8mGt3wVjabiL9Pen4gCYRvHifWB20eqVcgGh1ymkIhAcnyWimVX8y1OSQpNI+JGNPIoeEMTOOMKJpRmvKUzhMHhpFmEUtZUVp71NxvssLH5R0ljJV80xyjl41XcLZDTzFEz6ObtiW6Vhhc3QCaLUWxo18KO82PVNvy3W8G43xv33eSUlgU9ajGm3/VstYpR3yb/gKib0/M1yYoG78G2HWQX4YEpoE03UcBoNiRjTz8Ic6n0TQt7eJoCEoQkGoxC4CR7SgZgQ0MaHEQ5Q/u0CEJfpyKmqeO07Dn9KT7IPbLKa60cEVdY0R2RZDuQWTmBwiYvaCm/jjUrFQidIL1dOg/odHJEwULZ3cQyzMmZ56nQ3iGjNLM5Two8F/nBv8Ubg8oqp7LEPy3rgC2/9xL7Bf4mSav55Pu6H8hgRNSwHB0uD//mp9rWZRke8XvAJgX81PHvxVFZHqFCeIoLAuJ7C9u/JaujhbVNh7kgb7wdeSAimFPCbXSbkOjnuv25WM4b5C49b3DrMPuZajHW8cV6rAnhfU1rRvqC44NxBuk02NxJ1zEOWXaN3UhoAGt7TBQz05AonW3jY3rontV21k3h20sxAk09dd6csZNz9PIItDxB/8FFJ2z2ZTGgUovc6O2ugzLsstPzYQDR678R1fzVeRyPoxSkGjts2I4zbsfdMYTFAyLezE3PnIRIL6s2iNWRdKeBvRTdLaRvz5lzBlzK8Tqs6HMIMJfCpY ZDFXiDKi LuEE+QLC7qnkaeXU99M8nMcmG87u12M1zEA1Uk8D+b4SIDSfrASJVdj3V8AHXmw7+O9Bi4AjViC2+ZALiR1oEPZlDe7yeH+OMyj9xh97/HLI8ID49FSCQsIINmQMmCP0Pv5cc8KeyhChwhxzhIk2aA3+5azRuTd4TJh0kcvEVzDdgNZSwCwFg1TEPmeTjsx1OdFlI1ZcKfnDTFwlrE4G50djsLsqv3RFR/OqVIZPXAGQk8d/8W1qfRErSJVuqwpc64Z3Eio3b7oxIhrnNB6rCBjI5v/ztjPaXxcAN2tkHnx2MJIhOz2q6aX3sdXLMBj4SGqhmAYRF1eVMzE5YObxibXrLqPQsYyXct/hVxQE1sdwyd1XcndH9w6LihIw7A0NH2fAi/OuJpADVGYzxql08quzMvzpomPdZiCJZ/EpDXhGRZaCAW/v48pO0T0fNhIDQTFB3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement splice-read for overlayfs by passing the request down a layer rather than going through generic_file_splice_read() which is going to be changed to assume that ->read_folio() is present on buffered files. Signed-off-by: David Howells cc: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: Miklos Szeredi cc: linux-unionfs@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- fs/overlayfs/file.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c index c9d0c362c7ef..267b61df6fcd 100644 --- a/fs/overlayfs/file.c +++ b/fs/overlayfs/file.c @@ -419,6 +419,40 @@ static ssize_t ovl_write_iter(struct kiocb *iocb, struct iov_iter *iter) return ret; } +static ssize_t ovl_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, size_t len, + unsigned int flags) +{ + const struct cred *old_cred; + struct fd real; + ssize_t ret; + + ret = ovl_real_fdget(in, &real); + if (ret) + return ret; + + ret = -EINVAL; + if (in->f_flags & O_DIRECT && + !(real.file->f_mode & FMODE_CAN_ODIRECT)) + goto out_fdput; + if (!real.file->f_op->splice_read) + goto out_fdput; + + ret = rw_verify_area(READ, in, ppos, len); + if (unlikely(ret < 0)) + return ret; + + old_cred = ovl_override_creds(file_inode(in)->i_sb); + ret = real.file->f_op->splice_read(real.file, ppos, pipe, len, flags); + + revert_creds(old_cred); + ovl_file_accessed(in); +out_fdput: + fdput(real); + + return ret; +} + /* * Calling iter_file_splice_write() directly from overlay's f_op may deadlock * due to lock order inversion between pipe->mutex in iter_file_splice_write() @@ -695,7 +729,7 @@ const struct file_operations ovl_file_operations = { .fallocate = ovl_fallocate, .fadvise = ovl_fadvise, .flush = ovl_flush, - .splice_read = generic_file_splice_read, + .splice_read = ovl_splice_read, .splice_write = ovl_splice_write, .copy_file_range = ovl_copy_file_range, From patchwork Tue Feb 14 17:13:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 247A2C05027 for ; Tue, 14 Feb 2023 17:14:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B59256B0078; Tue, 14 Feb 2023 12:14:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ABB266B0085; Tue, 14 Feb 2023 12:14:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8975C6B0087; Tue, 14 Feb 2023 12:14:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 70A446B0078 for ; Tue, 14 Feb 2023 12:14:19 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 218CEC0E7E for ; Tue, 14 Feb 2023 17:14:19 +0000 (UTC) X-FDA: 80466545838.28.0AA1D26 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id BD082C0026 for ; Tue, 14 Feb 2023 17:14:15 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TAQLjtQJ; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394855; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9CqUl4uoQ+LRGKUoO3X6p5TYca6H7tIaOYxOpOq6akk=; b=UWP85/7VHhIj2zyEO4a5IM0wiKNifNU3JXxmVu/DF2OuWAhVUpWDn/IO38tp20+Om9kwNL u/g1n+wOG+6LDDgfcUqrPxeL/gnJ2ySW7Jyx24I6RYLTKlRszvvK1UiaG0C8wpFj1QxZx4 N2bZpWMYqdA11B1yGiXoDyuQyU6MJ9E= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TAQLjtQJ; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394855; a=rsa-sha256; cv=none; b=R/yw4aovNyNlFTegoNiK2LtN9nMVPtU1vS9pVtiJVmXWPb9mp5CN3wMz5ovkG21UGjmyVU eXuOU1+VFCmh/vBDNH+WspGWbiUQo1cqCw5s2FcQuYd9hpe+YVHYUAZ6sPR/oBDI51/jfB 4uwUIS5z/PgHXAXDT7a+vwxl8J9YvzE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9CqUl4uoQ+LRGKUoO3X6p5TYca6H7tIaOYxOpOq6akk=; b=TAQLjtQJBeo6nHoCYBN0mY0gxeo5tFFkXqoA1wVsvGB8i0ee1HoEbXri3+XWOOhfjZmwsU 5yJjq7kRHP9i/Ajyk5QBQrklliN3FJsafRsha+dqffw4jaeo/uY0SHV5pOMerYqUX1C27U HQvPV4oz3HnYutAJ/6gH5vL4DsojUQE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-253-cDT_luHMOIyAlH60JlEtxw-1; Tue, 14 Feb 2023 12:14:04 -0500 X-MC-Unique: cDT_luHMOIyAlH60JlEtxw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A98F52817248; Tue, 14 Feb 2023 17:14:02 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D058492B15; Tue, 14 Feb 2023 17:13:59 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard , Jan Harkes , coda@cs.cmu.edu, codalist@coda.cs.cmu.edu, linux-unionfs@vger.kernel.org Subject: [PATCH v14 06/17] coda: Implement splice-read Date: Tue, 14 Feb 2023 17:13:19 +0000 Message-Id: <20230214171330.2722188-7-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: p1heggpduftxq8jo4gq5mh47431x6fjf X-Rspamd-Queue-Id: BD082C0026 X-HE-Tag: 1676394855-202932 X-HE-Meta: U2FsdGVkX1/nTkjrUTn5nB2rghcvDeoCKsSsUs2XBuzSiXuxtWVRNMYBHhUFKz3hl8Ubg2Pfoeduw/1pjLcFzaAUm0rxCXc03nNgOWD8uVitHb846VT2AQ8iOuUD4K0e05i/yxB78qQMAih6Ii470U+GNpau4mIqkshNLu19bnq6skpLJxZY6fKCixorwCxtQdrBR2U07y6CuqdruSQZZfSXyDUkjT1+m/bACUbRdXR6Z/F+OX4cvrX2x04JcayGsU8p+PcaFkZO6kSB46hfAak3xxnLbyTi3HMwG1OfMwhNfWmddUuRjXDxHRVUkKWFpBZyeUMz7FZt1e4ARsfxB3XOg1dQ8d01OevqCcEXo5RwRn4v7jBF5OlVruXPbNNAdGzMSpvqcfPo03APVlVsY2NC2tHnKNcsW20e76ucJ3lxUcyko5zRBXUYNIJmocnXyT153IGw5Smk6unaUZ8jS6GZZsr4oru2VOsG/y7kTjsN24PMLlwMeNg0GOVrd9Xr/UbSorINnlSyZi+oIcN3wbqtNOh4+F4506THTowzDD48ZFBOIrb5apZHmX7bOq9d6yuquThoAjJCTfrcnqkT0BYGkXt8YWh9C68rJTKlloa2Ua1thVs47TW92reALz3S/oo5cdWqxDjNUXdonuotR46MHGWn1C230nqiS4piA4bwUdsIZwq0hJur03DGBbTQ7VlralF9e8hs0I9Stb9R5wcqJlmXg8rQ5sN4hMw/B3WrbvBM3go2R0EljNLJYnyCgHNBLGlT8iFEOl/FZKB28AR3tjRehexAS6PNX5KGaNfy0dlxy6ZWCgHgEOOnMjRAu0HuXWTu1Aw20scAJylg0ZgkPiVu60kPzNh4i19z56ewWcsAGzZXwcHeCcpwWLweneRHRHdKc4YMMmvHIp1HtkTVj6IUd/y6s+F2372LewN8E8bfuNGhoBV/gQBUc2iHTNRirBozT2gdoYpW3jy CK4E8Doh 2MaJExWezCr/vtV39AXwpEjwWJaWb5iDeCIcgajhxea3v1zumVogyjEtAt1y3uZhjU+X5JnuDRtNx59M8IvdNNZtglCJd4zY+GOqZ65wXk3gWnBor9ohmuMnVtVD3CdpeCp0V6eH+mt1SnnUZB4v5yIkMfrguXpbkA1grpu6Cnh5rhelUn260WTqzqQ/PuFuYXL5jDkt/qq887huzKmYoRHU9rMF1zZLhotQ/OW8jFvu7DSCOyXiJG5WKpvWUgUA1tRLCu4t9liAf4TZMzAvJ5ZgrkaRCOdunAQoDcD4Rxnj+j6oRhT5hOWqniHqv9KFmKfrpJ6nAbeqjH9avwp5dd6SkPkYTucMEi8UTBPrsUPRPGs+9AncRxISmKJ3Jm+GzYr6vNWhP4GI8Kx+3hIZXzfU6Ehtmr/PsFRJLXchy3QqXQrMjsZgyxy3/MgSkhmJSLUW3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement splice-read for coda by passing the request down a layer rather than going through generic_file_splice_read() which is going to be changed to assume that ->read_folio() is present on buffered files. Signed-off-by: David Howells cc: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: Jan Harkes cc: coda@cs.cmu.edu cc: codalist@coda.cs.cmu.edu cc: linux-unionfs@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Acked-by: Jan Harkes --- fs/coda/file.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/fs/coda/file.c b/fs/coda/file.c index 3f3c81e6b1ab..33cd7880d30e 100644 --- a/fs/coda/file.c +++ b/fs/coda/file.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include "coda_psdev.h" @@ -94,6 +95,39 @@ coda_file_write_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t +coda_file_splice_read(struct file *coda_file, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct inode *coda_inode = file_inode(coda_file); + struct coda_file_info *cfi = coda_ftoc(coda_file); + struct file *in = cfi->cfi_container; + loff_t ki_pos = *ppos; + ssize_t ret; + + if (!in->f_op->splice_read) + return -EINVAL; + + ret = rw_verify_area(READ, in, ppos, len); + if (unlikely(ret < 0)) + return ret; + + ret = venus_access_intent(coda_inode->i_sb, coda_i2f(coda_inode), + &cfi->cfi_access_intent, + len, ki_pos, CODA_ACCESS_TYPE_READ); + if (ret) + goto finish_read; + + ret = in->f_op->splice_read(in, ppos, pipe, len, flags); + +finish_read: + venus_access_intent(coda_inode->i_sb, coda_i2f(coda_inode), + &cfi->cfi_access_intent, + len, ki_pos, CODA_ACCESS_TYPE_READ_FINISH); + return ret; +} + static void coda_vm_open(struct vm_area_struct *vma) { @@ -302,5 +336,5 @@ const struct file_operations coda_file_operations = { .open = coda_open, .release = coda_release, .fsync = coda_fsync, - .splice_read = generic_file_splice_read, + .splice_read = coda_file_splice_read, }; From patchwork Tue Feb 14 17:13:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 690E1C64ED6 for ; Tue, 14 Feb 2023 17:14:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8CCA6B0083; Tue, 14 Feb 2023 12:14:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEFB06B0085; Tue, 14 Feb 2023 12:14:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF3E56B0087; Tue, 14 Feb 2023 12:14:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9A5BB6B0083 for ; Tue, 14 Feb 2023 12:14:17 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0D515AACC2 for ; Tue, 14 Feb 2023 17:14:17 +0000 (UTC) X-FDA: 80466545754.16.8EBFF83 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id A1AD81C001E for ; Tue, 14 Feb 2023 17:14:13 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q7G7aVCn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tsT35xW/A1CGBEVLxMBn7T6yW2S9XrBAkcV2v0SNR0w=; b=WObk58mmLyZ+TLBZfY7tyiZ4hob2Td1eDKmv7EH3P8yTlKRDKgD2yHqa2t2iaTgRMMrYSs j+lF3ynbNnyT3q5Jmdxs4DwLlhZMkvm+rWek+nEImo0+IuzCr7PYs3z//T74oZYwrBchwX 3NJJt83EeFupZFQAIwNO8fo52L2El7E= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Q7G7aVCn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394853; a=rsa-sha256; cv=none; b=nVgdcJfmd8WE8pJzhhKOteUwE6Q7Kf4oSeG0UfPBfzBFURqlRpmjngLgHDr202DcuwYZyQ aLY7aB3MRVk/C6Vdp98qmh754hVC/riLtgPiNTYxAYDdGb/Xg3KwySk0W1129IZQKdqSGd byh/o4W6l03EztEaNCgLmN5nq4+fAxM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tsT35xW/A1CGBEVLxMBn7T6yW2S9XrBAkcV2v0SNR0w=; b=Q7G7aVCnEw52I0HWimG31Mw8crhZAGqmEJQ7rj6Gz94wPp5/2LAmeP4knTlzfxwyzTuymg bPui5ob+0/jgxXj1yrgkoR2rwSs3/+fgzTfyyqgYJP/hoyc97pqvwmX2htzDb3nDWtFiZq 2slspbF2ow2c17hfNfNJ5monv7g+zU0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-619--tEiHNGSOsmK7bn8sDqt9Q-1; Tue, 14 Feb 2023 12:14:08 -0500 X-MC-Unique: -tEiHNGSOsmK7bn8sDqt9Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8CC982817223; Tue, 14 Feb 2023 17:14:05 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4B91B2166B29; Tue, 14 Feb 2023 17:14:03 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard , Miklos Szeredi , Arnd Bergmann , Greg Kroah-Hartman Subject: [PATCH v14 07/17] tty, proc, kernfs, random: Use direct_splice_read() Date: Tue, 14 Feb 2023 17:13:20 +0000 Message-Id: <20230214171330.2722188-8-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A1AD81C001E X-Stat-Signature: g44y97cjo7d448shf739s1to7j7746sx X-HE-Tag: 1676394853-931185 X-HE-Meta: U2FsdGVkX19US61hBMZ5EaTN9cqcVfOplCPnBjsBkoXozFY5ggDZ73cHGxIyiEKmiPcDcA8e0gyy6ivHca9tJPuIr+8fvhwAYyGeoJOM8SvfqtkHFt71Z7zsCnl6YqyC00spIm6NYJpfzgSsWVXu2jhumCZkeB72Dk32ull4RaG8KSTS5jAGtwO1qWGKLsEKeQQ9ml3ertRVVmU7tuwU5qblikINGqqlOLb0iadur6p1+IuXnBogJ8Kj9JYyoE7IjQoAmD06oFgCfDgaO0Rpgw80PTvH1XhEfOVWQHZ1A5RqZ3Sz2bv+gVtxhC/5Jsl3DyUIP0qFuf3KKhhmR7mrtMqyCxPOBk7dfCVlE14ysynqNawTuiRHKAsSAQ0eQyWi+mnL1sA/0+nelPDQFYkrzFIctVc+ePpVXyHe1Py4Z8YhmIUkA3oA+liwkpTZXt+6lhsUD7XB2McgAjMgW2cSpN8QEOSkUZzl0dr4VvyVFIgYN0KCJuQB7qsKZt4C0uNOTuPkJdEn8yLmUy+pJdU+svSTao6TxLtbMovYsQiSLDJQAIBDB2L93zOvJNhZWYo1RcIOsyGEkMDkN2kckKAFa3foemqlaKkHPKnVV/fX8pS/qcnF9zdCI1jSOS6UFGVuDTQE2KeCK3/EXQLb4H+L+wFcsimYJRydxc021VXhLKNmZGoR4q1MAjxU1iPLgqm36V217bqN26ITdT7U86hU03yaIFXbJQB03L46uzQl4LD4AoDNczX0CQ5nlA44XvcedGYpU/5q57cm3BbKVo9FsfPaWGjWUlITQi2V4si3zt3OBx7G1dJgdIv36jvCAIgwEwDEQK+DDt1L7aP9LJoJ/zc2qTdPhGuZaNUF72tXPisqmGNo4IvRg6niENog3uuJmYreptCn+zep8zfEHiDAJGmsM0+0d5wQNN6JEDma+0y5iROiB5eKUUJrb7GC3CM5qG5wmlqX3MxSN7s3dt+ x8vfGuIK 44gO51XfOJax6c7UwZB1mxl2D88LIQqHquhOnZpX50ha3GmtkgkKf4NnqZioU7+IOBzGg3sPb+7MJEXV1CLpPIIjRZaSq79/UZ8oy2QiRBFJatIARXPe2gjLA8ye43G57sNKTPSQX7sIjMu1qSDaX3VNJd6EMtaPS9GrT+VEA56Qtj0r+Cg5OGnFeMnxdwQkJMRFRJ6tLp0rXbk47zdYHbrBBxPjz+gJ8n7mdU7krFkWicCLudiff9vQnPboObThekzIqcvZHcNPsEw6HycaSYQmPygp6JRKyYcZE5UiycKv8Yc/i/UC70xrosy3ArTeqhaml+qeNAUMqPiyL4r3w39y/vq15pEjfK5nKsJdmdk3AlxKBWQUZgbsvS1K0mSCl1eYXUt07Of4hz6i3wtdg+t0Iasx4Tc3O0A2J9zi1JEc0MQblSD6/CPxykQ4ZGZVFqLQtBV9Mtm57Ut2+zOpZDRX8pe/6WsYZhwsurHvpOfErUXdoqF1sU0mSTwQPCqOIKiEpT1ve5y/MADMngxwccQkBYX+XYzvhUNkgxEbBDCx/k2yZBQMFhxf+oA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use direct_splice_read() for tty, procfs, kernfs and random files rather than going through generic_file_splice_read() as they just copy the file into the output buffer and don't splice pages. This avoids the need for them to have a ->read_folio() to satisfy filemap_splice_read(). Signed-off-by: David Howells cc: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: Miklos Szeredi cc: Arnd Bergmann cc: Greg Kroah-Hartman cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- drivers/char/random.c | 4 ++-- drivers/tty/tty_io.c | 4 ++-- fs/kernfs/file.c | 2 +- fs/proc/inode.c | 4 ++-- fs/proc/proc_sysctl.c | 2 +- fs/proc_namespace.c | 6 +++--- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index ce3ccd172cc8..792713616ba8 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1546,7 +1546,7 @@ const struct file_operations random_fops = { .compat_ioctl = compat_ptr_ioctl, .fasync = random_fasync, .llseek = noop_llseek, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, }; @@ -1557,7 +1557,7 @@ const struct file_operations urandom_fops = { .compat_ioctl = compat_ptr_ioctl, .fasync = random_fasync, .llseek = noop_llseek, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, }; diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c index 3149114bf130..495678e9b95e 100644 --- a/drivers/tty/tty_io.c +++ b/drivers/tty/tty_io.c @@ -466,7 +466,7 @@ static const struct file_operations tty_fops = { .llseek = no_llseek, .read_iter = tty_read, .write_iter = tty_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .poll = tty_poll, .unlocked_ioctl = tty_ioctl, @@ -481,7 +481,7 @@ static const struct file_operations console_fops = { .llseek = no_llseek, .read_iter = tty_read, .write_iter = redirected_tty_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .poll = tty_poll, .unlocked_ioctl = tty_ioctl, diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c index e4a50e4ff0d2..9d23b8141db7 100644 --- a/fs/kernfs/file.c +++ b/fs/kernfs/file.c @@ -1011,7 +1011,7 @@ const struct file_operations kernfs_file_fops = { .release = kernfs_fop_release, .poll = kernfs_fop_poll, .fsync = noop_fsync, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, }; diff --git a/fs/proc/inode.c b/fs/proc/inode.c index f495fdb39151..711f12706469 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -591,7 +591,7 @@ static const struct file_operations proc_iter_file_ops = { .llseek = proc_reg_llseek, .read_iter = proc_reg_read_iter, .write = proc_reg_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .poll = proc_reg_poll, .unlocked_ioctl = proc_reg_unlocked_ioctl, .mmap = proc_reg_mmap, @@ -617,7 +617,7 @@ static const struct file_operations proc_reg_file_ops_compat = { static const struct file_operations proc_iter_file_ops_compat = { .llseek = proc_reg_llseek, .read_iter = proc_reg_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .write = proc_reg_write, .poll = proc_reg_poll, .unlocked_ioctl = proc_reg_unlocked_ioctl, diff --git a/fs/proc/proc_sysctl.c b/fs/proc/proc_sysctl.c index 48f2d60bd78a..92533bd0e67b 100644 --- a/fs/proc/proc_sysctl.c +++ b/fs/proc/proc_sysctl.c @@ -869,7 +869,7 @@ static const struct file_operations proc_sys_file_operations = { .poll = proc_sys_poll, .read_iter = proc_sys_read, .write_iter = proc_sys_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .llseek = default_llseek, }; diff --git a/fs/proc_namespace.c b/fs/proc_namespace.c index 846f9455ae22..492abbbeff5e 100644 --- a/fs/proc_namespace.c +++ b/fs/proc_namespace.c @@ -324,7 +324,7 @@ static int mountstats_open(struct inode *inode, struct file *file) const struct file_operations proc_mounts_operations = { .open = mounts_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, .poll = mounts_poll, @@ -333,7 +333,7 @@ const struct file_operations proc_mounts_operations = { const struct file_operations proc_mountinfo_operations = { .open = mountinfo_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, .poll = mounts_poll, @@ -342,7 +342,7 @@ const struct file_operations proc_mountinfo_operations = { const struct file_operations proc_mountstats_operations = { .open = mountstats_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, }; From patchwork Tue Feb 14 17:13:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61DB1C05027 for ; Tue, 14 Feb 2023 17:14:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E16E16B0081; Tue, 14 Feb 2023 12:14:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D9F9F6B0082; Tue, 14 Feb 2023 12:14:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C408A6B0083; Tue, 14 Feb 2023 12:14:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B17356B0081 for ; Tue, 14 Feb 2023 12:14:14 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 81BC940850 for ; Tue, 14 Feb 2023 17:14:14 +0000 (UTC) X-FDA: 80466545628.14.49F5F29 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 8E8F440019 for ; Tue, 14 Feb 2023 17:14:12 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gfL4Mr+k; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394852; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J4i4bwQTS34x6oFo2ZgYGZTE2NYetAQq5iydlM1fsxY=; b=P+JfYxAbuvAEJFKZnH3yLRHd4juneWdzrJJqWZWxj2VKjxxSbGqv85t1SmLtHQn05kuW2q mOP9gbmcsoevQ9me150+oLPoaeIKo/KnI7V+lO+bhoXeDTCZUzJXlzltMSd9IoDWGvgy5s SIHzDpJnZ6Gk5QxVN06pQ+dBF6fUrlM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gfL4Mr+k; spf=pass (imf11.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394852; a=rsa-sha256; cv=none; b=P7JIeAJqecoFUkWHhZrV6lk7s1BgEnWKXsTW0T84D6/T0dMFYRxdJLhbFcYZhMhT33BswC VDWUXhLzQxNy0C6na2wWv048KiIDs5LDM/XKUMHay2gV+Y2dXX+gyC5Dd7eXVnMZMFpFRe co3+EHE3JnAa9Q7jPTqNlytDGW1Xc5Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J4i4bwQTS34x6oFo2ZgYGZTE2NYetAQq5iydlM1fsxY=; b=gfL4Mr+k99US8+sHixVHCfJC1Oei+7T1Bkw2U+5bPSeJ8rExTbU5XKzSxx29QjhWMtVhR5 fNsaEn1SjAJWxSyULSSfDAyflfU8cphJYslDiyZ+U6kOuyYMTPNHf+uFlOInO59095RuJ7 YMaFuZrwIrAiCKAk1FHceqwU0AH9L6A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-52-ZogOeZ8jO0-C97KX6CrYIA-1; Tue, 14 Feb 2023 12:14:10 -0500 X-MC-Unique: ZogOeZ8jO0-C97KX6CrYIA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4219F100F907; Tue, 14 Feb 2023 17:14:08 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4767E1415108; Tue, 14 Feb 2023 17:14:06 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 08/17] splice: Do splice read from a file without using ITER_PIPE Date: Tue, 14 Feb 2023 17:13:21 +0000 Message-Id: <20230214171330.2722188-9-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: qzy186gofgnyip1utcdnsedw97bzob44 X-Rspamd-Queue-Id: 8E8F440019 X-HE-Tag: 1676394852-783761 X-HE-Meta: U2FsdGVkX1/fmTXUBjgmEniqt0Q79B11NN1erYsAqfvuIKhDLCoKtMY2KclvBJ+rTCQi1lAbWxSs5fU9a6523BV/u2yFhDXIDxCSlxcBSmrhX+uDdsLw7PPQyfejbiwqY5VbrzfBb56ksDz8xXqA59IUuecMy3jcRkm8FHztqPe+eXFOimwPaM1ETulOIOUkX3E8vvdzdqeunMAUFOzaU/7m5268UxGq1laIOalDY7Kh1TjJ/GJ4s/qdP0oW96HNIGSUyTj5l2jr7l0/M7NZWSXICtNEkuT+LO0Sm7GkIWycPEtoeZ6RtFaiQpEFFLD3CMo/f/JDd7YK7Lq6iKRg59qaD2TIkdfs9iF8cLzE++tgfaaqmk7n4SEEay7+Xt79IaHvvaWXp0DzzVXhvEIj5qffoAPNhpcpzLVFh/w0DVBsu01CD05/+g7YNxwSCHEcvtOSH9fEysxLxzw0YaisSnVDeWvuHTz9gLx5CtEGPEGNZNrMP0/9mcQIKIdE8yGY3PXTwPtjEllfw+qW3QBS6kTrl3vjrSdfBPYUXlwS87sCPusP+84jU/ST0egYXpwMQbB3RzZ0CRaBOWSgJgDf7EBQA+OpzemTS65e8/u9XZYsBtCVfy1ptNXBSqRFcf5vUdepLN0AA/QvcO8I/9UI+NdQDKNekBEW+9qsRAFjqARjoRZJdStk11EFln24S9n1vsFx90C0nlJbRPtwrNDkb46d3n55f4BzLy+ERXtGtWp3HGsvm49UN8GIl50O7c1WlbLTChp3+4J9O9ypoyiGav1wO3XxxRV5daF5vGe9eLWZEOyc/j3UGS7MAnIpUwtj/how6TIqrxCf4xf5zlwOg/OyV3qgaCfUDgY9mZanDKcwmoKamEOae/Vv493dm77w741L75Roe42RW4CdEpriOomfJSU19KQ8bPK8C013S6MtU7Vk2N6lZQT9e7h5qlp5/7oS5ztbb4PMHt0mYhr ZeexzO1W iQ3k8yvKVbusU9WfT7umgR5qjldCRxqhXLlvGIK3MDkfcoUBmrGvEQWK2UgQJIIbbxwe9tV2nA/EmS2T+/biVSmw45YvULc1/snDD5SSz7uaSiwAIa3RUrMvXAPFQ8yFskXP52rDZed4N1k1yaN1S4laesraarto0Wv4C2BdN4kmBtmtmxhiiUyMp7MyhPJYpdnpP79geYZnqvS1gQHkFu86QZn0vUv/vdlAFxahWmwGkO6iXlh2Pewnz1XDaFWsHcx42iDoWv5XgtcyJsNdR0HP99W0fJpcDhpNuY6HezXLuNkzMrjacCLZffPjZKXEfM5SBbSjGTEOA37d0bI/kL21XFKYMb8kb0+J4Zg5qW2Gka9w9cyhiNZbJDaemo1lYPhGHzzWQNFa5YOZLs1xvIb1AKhH4nnrz93w2d30fjAKbzgJiCpTsWf967SQE0sE8OuGZ0IZJiB4mh/Y36rBwXYAGvyeBqJ3Z6RrlBK89cL1nFeBYWlfCOkSoj+e+yrkTsf2N X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make generic_file_splice_read() use filemap_splice_read() and direct_splice_read() rather than using an ITER_PIPE and call_read_iter(). With this, ITER_PIPE is no longer used. Signed-off-by: David Howells cc: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org --- Notes: ver #14) - Split out filemap_splice_read() into a separate patch. fs/splice.c | 30 +++++++----------------------- 1 file changed, 7 insertions(+), 23 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 4c6332854b63..a93478338cec 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -391,29 +391,13 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) { - struct iov_iter to; - struct kiocb kiocb; - int ret; - - iov_iter_pipe(&to, ITER_DEST, pipe, len); - init_sync_kiocb(&kiocb, in); - kiocb.ki_pos = *ppos; - ret = call_read_iter(in, &kiocb, &to); - if (ret > 0) { - *ppos = kiocb.ki_pos; - file_accessed(in); - } else if (ret < 0) { - /* free what was emitted */ - pipe_discard_from(pipe, to.start_head); - /* - * callers of ->splice_read() expect -EAGAIN on - * "can't put anything in there", rather than -EFAULT. - */ - if (ret == -EFAULT) - ret = -EAGAIN; - } - - return ret; + if (unlikely(*ppos >= file_inode(in)->i_sb->s_maxbytes)) + return 0; + if (unlikely(!len)) + return 0; + if (in->f_flags & O_DIRECT) + return direct_splice_read(in, ppos, pipe, len, flags); + return filemap_splice_read(in, ppos, pipe, len, flags); } EXPORT_SYMBOL(generic_file_splice_read); From patchwork Tue Feb 14 17:13:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1104C6379F for ; Tue, 14 Feb 2023 17:14:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7FD066B0082; Tue, 14 Feb 2023 12:14:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 785CD6B0083; Tue, 14 Feb 2023 12:14:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58A0D6B0085; Tue, 14 Feb 2023 12:14:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3A5B56B0082 for ; Tue, 14 Feb 2023 12:14:16 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0CF841401F5 for ; Tue, 14 Feb 2023 17:14:16 +0000 (UTC) X-FDA: 80466545712.23.F813B9C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 33F51C0013 for ; Tue, 14 Feb 2023 17:14:13 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BfNcWIj+; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394854; a=rsa-sha256; cv=none; b=iuqKrEIBcS+Rp8hSWa7Ka3KOvSRHMTa9X9ywheH/xjby0b6VGiBc0bvFo0kKwxWMT8X6tx /Ex6OMp6sF2XGdZHReungpBFxm7wb0cCaDBGifjq0Fk/wA2jVEiIQG3poYumgnROLMApfP 1p7++QwGxVQQWyIIUb3ZDL0ZfeCWXmQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BfNcWIj+; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394854; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q/WvJQSOUXoHuE1z1Cu24KZfrF9JVSOUntkfW3f52zA=; b=KUXoA5JNERrIR/ArYV8khrbigvfFNm52Nes5WYpm39xM6wUf7QNE+ItmiwS3ZCUFZcsRwQ 0lh2UdHgBSiCcE6qXcaL2VcAVQQwWCoDR2iL0rCgLaE1vw7JRbDvPqoFVBTLUHqp1gBYpc OvzfFZV2adIqNCHWoStKERSR86iHuAI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q/WvJQSOUXoHuE1z1Cu24KZfrF9JVSOUntkfW3f52zA=; b=BfNcWIj+35TA+1F7WXk8XHnb43kB+/EQTnoCCNA15tT8ymQ+jzl8WjILg+Fy0ggX/oL8nO 9Kim804Agox4K0Xeg1Wmq0avpok6g9fE5kvhaDosymu/5Vj1g0RTy6zYLhNy2Qo7NZD+gN 5WtdfDYydpgeqo2qPfaycSosPukFN2U= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-298-LwGEXxIWPwy9ik3zHqW1GQ-1; Tue, 14 Feb 2023 12:14:11 -0500 X-MC-Unique: LwGEXxIWPwy9ik3zHqW1GQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E8B6380D0E0; Tue, 14 Feb 2023 17:14:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA42D1121318; Tue, 14 Feb 2023 17:14:08 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 09/17] iov_iter: Kill ITER_PIPE Date: Tue, 14 Feb 2023 17:13:22 +0000 Message-Id: <20230214171330.2722188-10-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Queue-Id: 33F51C0013 X-Rspamd-Server: rspam01 X-Stat-Signature: r55e5ckayjengo1pyixjc5j1ugqgz3d6 X-HE-Tag: 1676394853-34712 X-HE-Meta: U2FsdGVkX1+8J24MhqcYGCBoxC/rdSLcweJBMf1hr2jRr3Na8fSnPnpDW4lDEQKYE9Xpok/PUsRg9MAcF1hqBINgHiuUZ+A7aRGFBm9jcSjAP2CFUtyNsBD8r+qGIkhOhTb7obZERQvI0b1zWD51RDJhvXmM9MCphRzc03mfkI+PeUWJyt3aG389cVkkUgzl3u7rmXhzeQBbSYMyqxVltLgZ6nvsjnFpvTSyo+aXwkhuXzIS9DLn8Z6dHK+Y7XZWNRGq7XAAjIwHCMsz/eQSwTlQ74tirlsypi+jAkx3xmXFlrMveEBEMU4GieyqNEX115p3h1eHRommjdY6qup+8H9q5319OKmNFhyO9/OUA2leNVSAfqsmEREeYyxZEgBjgMRJ+b/yWRwVe8N/7cq5aHzPEj/NVkfNwiUfXahU4IgDnntehBLmA5nBvsYufxfcKucBHwzEv+zxV4ClKzbwmNV0zZq/XRsOo1R/vsUMCYeJ/5SzSeBDRFiSKt0i/OVdMWGYyTYfV7ny9nWqw/M3fs1PQ8l82XFAMKsyBZric7VM0HOGYpvbr4eDV+QUcyo39Uomd1Adj0ZPof9kn64eC1l2HH4QFd4e7j1gfAs4Z5VrPktjuO5AGG5onnNFXEPdApQVaR+D39qXR8+wFYRLzgq8iHy8cCZez0sCESlfa/99PWBE4WcFdtfiDNNZdlMyotFPt4CHIFBwxQcZgLkZJLkrEY6JGQUBWJyIOaS+ip+6ou6QRHLqUUkw3s8+TPpmpXXdc5NzWuURhLgO69rJZ12lm33f91cHTuVBNsz6zBEUOIIM/69MeCAaKEGdNY9t/XSPAc/pNL9gh/lYkCZjMGTfyvRRqlMrAJwRgiHhhC2R4pMDW0caVoTmi5JCCeKTVe5hOq6lu5OeGdtsO8MFEChwhVK9MLtL3S3CkQCpwUU5oM3+sUUNipHEjcyfco08fWlrWLAva40QV/7ZGyS w4Jj97nH ru5Zixsq4CcHOkuLl8eiMdvUHWAegW3RCEEPzrJfXMElqqc5JZMy4WWbrawtj6+1N+WGu1lvbmvECEhla1JABjE3IigTU0h54wQuDXb1p923c9KL4bzwmhQxSYR+U+CzejPpeAiKyq4yyRG+9offy1VMIVvIZ8EqeASEl2rALb0AtFOG4qi+3lmtGj79RbJz399UuPIfABq5AEag1EmpNZh+GOmYcusKHs/sYsniRb17pa+Saisc0+3u8hc1eBNj+3PqlBeQQ9GJWi4wzKIqTJgSAdy5SqPEDYAgV+9HF6Yj15NwYf1RD0uX46/7I6HrG7ES3lhW/jmzCBbbBMaHi526VK54zIivFRHBdLGK4ciOTJsz74aKBpdWvCu4EhWNziIUmRfFmhV56nGarl/qBWxzY7//KV0r/o9cCSgepiRtZCQkU8qbwjgzdWRrAyxuY+4rB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The ITER_PIPE-type iterator was only used for generic_file_splice_read(), but that has now been switched to either pull pages directly from the pagecache for buffered file splice-reads or to use ITER_BVEC instead for O_DIRECT file splice-reads. This leaves ITER_PIPE unused - so remove it. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org --- fs/cifs/file.c | 8 +- include/linux/uio.h | 14 -- lib/iov_iter.c | 429 +------------------------------------------- mm/filemap.c | 3 +- 4 files changed, 5 insertions(+), 449 deletions(-) diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 22dfc1f8b4f1..57ca4eea69dd 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -3806,13 +3806,7 @@ cifs_readdata_to_iov(struct cifs_readdata *rdata, struct iov_iter *iter) size_t copy = min_t(size_t, remaining, PAGE_SIZE); size_t written; - if (unlikely(iov_iter_is_pipe(iter))) { - void *addr = kmap_atomic(page); - - written = copy_to_iter(addr, copy, iter); - kunmap_atomic(addr); - } else - written = copy_page_to_iter(page, 0, copy, iter); + written = copy_page_to_iter(page, 0, copy, iter); remaining -= written; if (written < copy && iov_iter_count(iter) > 0) break; diff --git a/include/linux/uio.h b/include/linux/uio.h index 9f158238edba..dcc0ca5ef491 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -11,7 +11,6 @@ #include struct page; -struct pipe_inode_info; struct kvec { void *iov_base; /* and that should *never* hold a userland pointer */ @@ -23,7 +22,6 @@ enum iter_type { ITER_IOVEC, ITER_KVEC, ITER_BVEC, - ITER_PIPE, ITER_XARRAY, ITER_DISCARD, ITER_UBUF, @@ -53,15 +51,10 @@ struct iov_iter { const struct kvec *kvec; const struct bio_vec *bvec; struct xarray *xarray; - struct pipe_inode_info *pipe; void __user *ubuf; }; union { unsigned long nr_segs; - struct { - unsigned int head; - unsigned int start_head; - }; loff_t xarray_start; }; }; @@ -99,11 +92,6 @@ static inline bool iov_iter_is_bvec(const struct iov_iter *i) return iov_iter_type(i) == ITER_BVEC; } -static inline bool iov_iter_is_pipe(const struct iov_iter *i) -{ - return iov_iter_type(i) == ITER_PIPE; -} - static inline bool iov_iter_is_discard(const struct iov_iter *i) { return iov_iter_type(i) == ITER_DISCARD; @@ -245,8 +233,6 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec unsigned long nr_segs, size_t count); void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec, unsigned long nr_segs, size_t count); -void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inode_info *pipe, - size_t count); void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray, loff_t start, size_t count); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 47c484551c59..adc5e8aa8ae8 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -14,8 +14,6 @@ #include #include -#define PIPE_PARANOIA /* for now */ - /* covers ubuf and kbuf alike */ #define iterate_buf(i, n, base, len, off, __p, STEP) { \ size_t __maybe_unused off = 0; \ @@ -186,150 +184,6 @@ static int copyin(void *to, const void __user *from, size_t n) return res; } -#ifdef PIPE_PARANOIA -static bool sanity(const struct iov_iter *i) -{ - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_head = pipe->head; - unsigned int p_tail = pipe->tail; - unsigned int p_occupancy = pipe_occupancy(p_head, p_tail); - unsigned int i_head = i->head; - unsigned int idx; - - if (i->last_offset) { - struct pipe_buffer *p; - if (unlikely(p_occupancy == 0)) - goto Bad; // pipe must be non-empty - if (unlikely(i_head != p_head - 1)) - goto Bad; // must be at the last buffer... - - p = pipe_buf(pipe, i_head); - if (unlikely(p->offset + p->len != abs(i->last_offset))) - goto Bad; // ... at the end of segment - } else { - if (i_head != p_head) - goto Bad; // must be right after the last buffer - } - return true; -Bad: - printk(KERN_ERR "idx = %d, offset = %d\n", i_head, i->last_offset); - printk(KERN_ERR "head = %d, tail = %d, buffers = %d\n", - p_head, p_tail, pipe->ring_size); - for (idx = 0; idx < pipe->ring_size; idx++) - printk(KERN_ERR "[%p %p %d %d]\n", - pipe->bufs[idx].ops, - pipe->bufs[idx].page, - pipe->bufs[idx].offset, - pipe->bufs[idx].len); - WARN_ON(1); - return false; -} -#else -#define sanity(i) true -#endif - -static struct page *push_anon(struct pipe_inode_info *pipe, unsigned size) -{ - struct page *page = alloc_page(GFP_USER); - if (page) { - struct pipe_buffer *buf = pipe_buf(pipe, pipe->head++); - *buf = (struct pipe_buffer) { - .ops = &default_pipe_buf_ops, - .page = page, - .offset = 0, - .len = size - }; - } - return page; -} - -static void push_page(struct pipe_inode_info *pipe, struct page *page, - unsigned int offset, unsigned int size) -{ - struct pipe_buffer *buf = pipe_buf(pipe, pipe->head++); - *buf = (struct pipe_buffer) { - .ops = &page_cache_pipe_buf_ops, - .page = page, - .offset = offset, - .len = size - }; - get_page(page); -} - -static inline int last_offset(const struct pipe_buffer *buf) -{ - if (buf->ops == &default_pipe_buf_ops) - return buf->len; // buf->offset is 0 for those - else - return -(buf->offset + buf->len); -} - -static struct page *append_pipe(struct iov_iter *i, size_t size, - unsigned int *off) -{ - struct pipe_inode_info *pipe = i->pipe; - int offset = i->last_offset; - struct pipe_buffer *buf; - struct page *page; - - if (offset > 0 && offset < PAGE_SIZE) { - // some space in the last buffer; add to it - buf = pipe_buf(pipe, pipe->head - 1); - size = min_t(size_t, size, PAGE_SIZE - offset); - buf->len += size; - i->last_offset += size; - i->count -= size; - *off = offset; - return buf->page; - } - // OK, we need a new buffer - *off = 0; - size = min_t(size_t, size, PAGE_SIZE); - if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) - return NULL; - page = push_anon(pipe, size); - if (!page) - return NULL; - i->head = pipe->head - 1; - i->last_offset = size; - i->count -= size; - return page; -} - -static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t bytes, - struct iov_iter *i) -{ - struct pipe_inode_info *pipe = i->pipe; - unsigned int head = pipe->head; - - if (unlikely(bytes > i->count)) - bytes = i->count; - - if (unlikely(!bytes)) - return 0; - - if (!sanity(i)) - return 0; - - if (offset && i->last_offset == -offset) { // could we merge it? - struct pipe_buffer *buf = pipe_buf(pipe, head - 1); - if (buf->page == page) { - buf->len += bytes; - i->last_offset -= bytes; - i->count -= bytes; - return bytes; - } - } - if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) - return 0; - - push_page(pipe, page, offset, bytes); - i->last_offset = -(offset + bytes); - i->head = head; - i->count -= bytes; - return bytes; -} - /* * fault_in_iov_iter_readable - fault in iov iterator for reading * @i: iterator @@ -433,46 +287,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_init); -// returns the offset in partial buffer (if any) -static inline unsigned int pipe_npages(const struct iov_iter *i, int *npages) -{ - struct pipe_inode_info *pipe = i->pipe; - int used = pipe->head - pipe->tail; - int off = i->last_offset; - - *npages = max((int)pipe->max_usage - used, 0); - - if (off > 0 && off < PAGE_SIZE) { // anon and not full - (*npages)++; - return off; - } - return 0; -} - -static size_t copy_pipe_to_iter(const void *addr, size_t bytes, - struct iov_iter *i) -{ - unsigned int off, chunk; - - if (unlikely(bytes > i->count)) - bytes = i->count; - if (unlikely(!bytes)) - return 0; - - if (!sanity(i)) - return 0; - - for (size_t n = bytes; n; n -= chunk) { - struct page *page = append_pipe(i, n, &off); - chunk = min_t(size_t, n, PAGE_SIZE - off); - if (!page) - return bytes - n; - memcpy_to_page(page, off, addr, chunk); - addr += chunk; - } - return bytes; -} - static __wsum csum_and_memcpy(void *to, const void *from, size_t len, __wsum sum, size_t off) { @@ -480,44 +294,10 @@ static __wsum csum_and_memcpy(void *to, const void *from, size_t len, return csum_block_add(sum, next, off); } -static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, - struct iov_iter *i, __wsum *sump) -{ - __wsum sum = *sump; - size_t off = 0; - unsigned int chunk, r; - - if (unlikely(bytes > i->count)) - bytes = i->count; - if (unlikely(!bytes)) - return 0; - - if (!sanity(i)) - return 0; - - while (bytes) { - struct page *page = append_pipe(i, bytes, &r); - char *p; - - if (!page) - break; - chunk = min_t(size_t, bytes, PAGE_SIZE - r); - p = kmap_local_page(page); - sum = csum_and_memcpy(p + r, addr + off, chunk, sum, off); - kunmap_local(p); - off += chunk; - bytes -= chunk; - } - *sump = sum; - return off; -} - size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { if (WARN_ON_ONCE(i->data_source)) return 0; - if (unlikely(iov_iter_is_pipe(i))) - return copy_pipe_to_iter(addr, bytes, i); if (user_backed_iter(i)) might_fault(); iterate_and_advance(i, bytes, base, len, off, @@ -539,42 +319,6 @@ static int copyout_mc(void __user *to, const void *from, size_t n) return n; } -static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, - struct iov_iter *i) -{ - size_t xfer = 0; - unsigned int off, chunk; - - if (unlikely(bytes > i->count)) - bytes = i->count; - if (unlikely(!bytes)) - return 0; - - if (!sanity(i)) - return 0; - - while (bytes) { - struct page *page = append_pipe(i, bytes, &off); - unsigned long rem; - char *p; - - if (!page) - break; - chunk = min_t(size_t, bytes, PAGE_SIZE - off); - p = kmap_local_page(page); - rem = copy_mc_to_kernel(p + off, addr + xfer, chunk); - chunk -= rem; - kunmap_local(p); - xfer += chunk; - bytes -= chunk; - if (rem) { - iov_iter_revert(i, rem); - break; - } - } - return xfer; -} - /** * _copy_mc_to_iter - copy to iter with source memory error exception handling * @addr: source kernel address @@ -594,9 +338,8 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, * alignment and poison alignment assumptions to avoid re-triggering * hardware exceptions. * - * * ITER_KVEC, ITER_PIPE, and ITER_BVEC can return short copies. - * Compare to copy_to_iter() where only ITER_IOVEC attempts might return - * a short copy. + * * ITER_KVEC and ITER_BVEC can return short copies. Compare to + * copy_to_iter() where only ITER_IOVEC attempts might return a short copy. * * Return: number of bytes copied (may be %0) */ @@ -604,8 +347,6 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { if (WARN_ON_ONCE(i->data_source)) return 0; - if (unlikely(iov_iter_is_pipe(i))) - return copy_mc_pipe_to_iter(addr, bytes, i); if (user_backed_iter(i)) might_fault(); __iterate_and_advance(i, bytes, base, len, off, @@ -711,8 +452,6 @@ size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, return 0; if (WARN_ON_ONCE(i->data_source)) return 0; - if (unlikely(iov_iter_is_pipe(i))) - return copy_page_to_iter_pipe(page, offset, bytes, i); page += offset / PAGE_SIZE; // first subpage offset %= PAGE_SIZE; while (1) { @@ -761,36 +500,8 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, } EXPORT_SYMBOL(copy_page_from_iter); -static size_t pipe_zero(size_t bytes, struct iov_iter *i) -{ - unsigned int chunk, off; - - if (unlikely(bytes > i->count)) - bytes = i->count; - if (unlikely(!bytes)) - return 0; - - if (!sanity(i)) - return 0; - - for (size_t n = bytes; n; n -= chunk) { - struct page *page = append_pipe(i, n, &off); - char *p; - - if (!page) - return bytes - n; - chunk = min_t(size_t, n, PAGE_SIZE - off); - p = kmap_local_page(page); - memset(p + off, 0, chunk); - kunmap_local(p); - } - return bytes; -} - size_t iov_iter_zero(size_t bytes, struct iov_iter *i) { - if (unlikely(iov_iter_is_pipe(i))) - return pipe_zero(bytes, i); iterate_and_advance(i, bytes, base, len, count, clear_user(base, len), memset(base, 0, len) @@ -821,32 +532,6 @@ size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t byt } EXPORT_SYMBOL(copy_page_from_iter_atomic); -static void pipe_advance(struct iov_iter *i, size_t size) -{ - struct pipe_inode_info *pipe = i->pipe; - int off = i->last_offset; - - if (!off && !size) { - pipe_discard_from(pipe, i->start_head); // discard everything - return; - } - i->count -= size; - while (1) { - struct pipe_buffer *buf = pipe_buf(pipe, i->head); - if (off) /* make it relative to the beginning of buffer */ - size += abs(off) - buf->offset; - if (size <= buf->len) { - buf->len = size; - i->last_offset = last_offset(buf); - break; - } - size -= buf->len; - i->head++; - off = 0; - } - pipe_discard_from(pipe, i->head + 1); // discard everything past this one -} - static void iov_iter_bvec_advance(struct iov_iter *i, size_t size) { const struct bio_vec *bvec, *end; @@ -898,8 +583,6 @@ void iov_iter_advance(struct iov_iter *i, size_t size) iov_iter_iovec_advance(i, size); } else if (iov_iter_is_bvec(i)) { iov_iter_bvec_advance(i, size); - } else if (iov_iter_is_pipe(i)) { - pipe_advance(i, size); } else if (iov_iter_is_discard(i)) { i->count -= size; } @@ -913,26 +596,6 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) if (WARN_ON(unroll > MAX_RW_COUNT)) return; i->count += unroll; - if (unlikely(iov_iter_is_pipe(i))) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int head = pipe->head; - - while (head > i->start_head) { - struct pipe_buffer *b = pipe_buf(pipe, --head); - if (unroll < b->len) { - b->len -= unroll; - i->last_offset = last_offset(b); - i->head = head; - return; - } - unroll -= b->len; - pipe_buf_release(pipe, b); - pipe->head--; - } - i->last_offset = 0; - i->head = head; - return; - } if (unlikely(iov_iter_is_discard(i))) return; if (unroll <= i->iov_offset) { @@ -1020,24 +683,6 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_bvec); -void iov_iter_pipe(struct iov_iter *i, unsigned int direction, - struct pipe_inode_info *pipe, - size_t count) -{ - BUG_ON(direction != READ); - WARN_ON(pipe_full(pipe->head, pipe->tail, pipe->ring_size)); - *i = (struct iov_iter){ - .iter_type = ITER_PIPE, - .data_source = false, - .pipe = pipe, - .head = pipe->head, - .start_head = pipe->head, - .last_offset = 0, - .count = count - }; -} -EXPORT_SYMBOL(iov_iter_pipe); - /** * iov_iter_xarray - Initialise an I/O iterator to use the pages in an xarray * @i: The iterator to initialise. @@ -1162,19 +807,6 @@ bool iov_iter_is_aligned(const struct iov_iter *i, unsigned addr_mask, if (iov_iter_is_bvec(i)) return iov_iter_aligned_bvec(i, addr_mask, len_mask); - if (iov_iter_is_pipe(i)) { - size_t size = i->count; - - if (size & len_mask) - return false; - if (size && i->last_offset > 0) { - if (i->last_offset & addr_mask) - return false; - } - - return true; - } - if (iov_iter_is_xarray(i)) { if (i->count & len_mask) return false; @@ -1244,14 +876,6 @@ unsigned long iov_iter_alignment(const struct iov_iter *i) if (iov_iter_is_bvec(i)) return iov_iter_alignment_bvec(i); - if (iov_iter_is_pipe(i)) { - size_t size = i->count; - - if (size && i->last_offset > 0) - return size | i->last_offset; - return size; - } - if (iov_iter_is_xarray(i)) return (i->xarray_start + i->iov_offset) | i->count; @@ -1303,36 +927,6 @@ static int want_pages_array(struct page ***res, size_t size, return count; } -static ssize_t pipe_get_pages(struct iov_iter *i, - struct page ***pages, size_t maxsize, unsigned maxpages, - size_t *start) -{ - unsigned int npages, count, off, chunk; - struct page **p; - size_t left; - - if (!sanity(i)) - return -EFAULT; - - *start = off = pipe_npages(i, &npages); - if (!npages) - return -EFAULT; - count = want_pages_array(pages, maxsize, off, min(npages, maxpages)); - if (!count) - return -ENOMEM; - p = *pages; - for (npages = 0, left = maxsize ; npages < count; npages++, left -= chunk) { - struct page *page = append_pipe(i, left, &off); - if (!page) - break; - chunk = min_t(size_t, left, PAGE_SIZE - off); - get_page(*p++ = page); - } - if (!npages) - return -EFAULT; - return maxsize - left; -} - static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, pgoff_t index, unsigned int nr_pages) { @@ -1480,8 +1074,6 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, } return maxsize; } - if (iov_iter_is_pipe(i)) - return pipe_get_pages(i, pages, maxsize, maxpages, start); if (iov_iter_is_xarray(i)) return iter_xarray_get_pages(i, pages, maxsize, maxpages, start); return -EFAULT; @@ -1571,9 +1163,7 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, } sum = csum_shift(csstate->csum, csstate->off); - if (unlikely(iov_iter_is_pipe(i))) - bytes = csum_and_copy_to_pipe_iter(addr, bytes, i, &sum); - else iterate_and_advance(i, bytes, base, len, off, ({ + iterate_and_advance(i, bytes, base, len, off, ({ next = csum_and_copy_to_user(addr + off, base, len); sum = csum_block_add(sum, next, off); next ? 0 : len; @@ -1658,15 +1248,6 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages) return iov_npages(i, maxpages); if (iov_iter_is_bvec(i)) return bvec_npages(i, maxpages); - if (iov_iter_is_pipe(i)) { - int npages; - - if (!sanity(i)) - return 0; - - pipe_npages(i, &npages); - return min(npages, maxpages); - } if (iov_iter_is_xarray(i)) { unsigned offset = (i->xarray_start + i->iov_offset) % PAGE_SIZE; int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE); @@ -1679,10 +1260,6 @@ EXPORT_SYMBOL(iov_iter_npages); const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) { *new = *old; - if (unlikely(iov_iter_is_pipe(new))) { - WARN_ON(1); - return NULL; - } if (iov_iter_is_bvec(new)) return new->bvec = kmemdup(new->bvec, new->nr_segs * sizeof(struct bio_vec), diff --git a/mm/filemap.c b/mm/filemap.c index 8c7b135c8e23..c01bbcb9fa92 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2693,8 +2693,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, if (unlikely(iocb->ki_pos >= i_size_read(inode))) break; - error = filemap_get_pages(iocb, iter->count, &fbatch, - iov_iter_is_pipe(iter)); + error = filemap_get_pages(iocb, iter->count, &fbatch, false); if (error < 0) break; From patchwork Tue Feb 14 17:13:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D32C2C61DA4 for ; Tue, 14 Feb 2023 17:14:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27E3A6B007D; Tue, 14 Feb 2023 12:14:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 208B86B0085; Tue, 14 Feb 2023 12:14:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E64546B0087; Tue, 14 Feb 2023 12:14:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D0EAD6B007D for ; Tue, 14 Feb 2023 12:14:20 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A33BAA08E1 for ; Tue, 14 Feb 2023 17:14:20 +0000 (UTC) X-FDA: 80466545880.11.61EF9F0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id CBA2710000D for ; Tue, 14 Feb 2023 17:14:18 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IQQ+hBzN; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q5bJAUy0Yf/xIkHbL0gYjIax658ix+8tiANnbR9foj4=; b=PXAyBSxtzuwQjdA3q7mdJKoZMq9M9GoZ1Q/Okz0XjGaoQPiSv/Eyx7VIz+sv9uyQf2JKz6 R7Feq6wboa8IH04Mr8DkJ6cV9vEt7MY72bTmLBCrqmV3OT+Y7l2jbZNE/7msf9k8YWdSUg j90z8CJ1JT4g2UTRbc06nQjHtJUTXO8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IQQ+hBzN; spf=pass (imf14.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394858; a=rsa-sha256; cv=none; b=thYo+IS80NOlbsG/6rj1k5iqkDQy72BrKSL1e74/b7NB8U6n7j12rdS4gp47tIhdZJlsDa hDpDfYvDsVMI6iwHemjA5j23Ys+cKzOWE4hprw3RzG3H49xNH7nAJyNf3lkwo6aamYvxz5 WcOMM7dZKg7SVqrsvIuEv/tqxppTPZA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q5bJAUy0Yf/xIkHbL0gYjIax658ix+8tiANnbR9foj4=; b=IQQ+hBzNOl4JP9Asqelq9lACXQGwpHaq/Qr0HoiBsfj7UDUFwxAQDYFoiOU6U6U2my8Ll8 KYcSAa8CCwJcPlV3trBDZEPrFOEzRJLnMX0Q5aUHrcW6YU46LIv5MnZwc4MrqdRqrWAlFm x1Ufxchon1nRI6JAbG18BvMCY1PoMA0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-411-XEGHYl5VN7igzLHfz7WV0g-1; Tue, 14 Feb 2023 12:14:14 -0500 X-MC-Unique: XEGHYl5VN7igzLHfz7WV0g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9EDA5800050; Tue, 14 Feb 2023 17:14:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id A9D902026D4B; Tue, 14 Feb 2023 17:14:11 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 10/17] iov_iter: Define flags to qualify page extraction. Date: Tue, 14 Feb 2023 17:13:23 +0000 Message-Id: <20230214171330.2722188-11-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 3d43bz3tka8m8ntd9ujac43sbxgxrxjp X-Rspamd-Queue-Id: CBA2710000D X-HE-Tag: 1676394858-818545 X-HE-Meta: U2FsdGVkX1+Jr3uGy1URBCmPnWtWTDx0b0CMytxtVxO6Ewnn1hvJTUYB/W6xwRHZI/oAR9J1D1aqliJz0mfEl6pX39XUWIsjwxPO3zJBSOM6lLPvguOv4NP7CvacoRXN//RjD7j7MIojJtRLKlAwqBKH+cpETq5DYjTyvC6lSqrfWy7GGqWBWoH2bJ2htCinlR0hYyWnAFU6I6CiuomfqZY2QXner9tkawn0Q3nKFKtZAGV7IkO3imOxez04or9PBgZpWCDgBj7ZQuqNTYu2AohtSav+0Kp/8lQUiQ7a/aHyS7s+l6/RxE5GsAeg1pzIntV0vineGHTOnCQ+00P94CIXQga8Isfldv54hcqfGJJJKbVwUOcA+DSVX9fdFanJdOo7/P7UCoatrwNp+ypNFvgAMh60uWJCS96hNgDOvEvyKXdU0x2lR2noJ3S7XTyTZofhr+fFYMEWMYAGvXf+e/PGLGAOeH7YA2lwzG4xTVsYIpcxeAH62ZEMGQHJFqc3W2IBpFraQDWTykdNnskZeJySxgTubTZ5n/8Nln2jZPFOy99RQ5uKJh2eK5/0Eogs9ih8c8y7OMlQiOH58jceTmZcGLOb2CHRFML5MnGBCdXK9EsM0RpQdmIOJrzTSCsv0VXQ6JG2jBBZyLRjshTEv0XACky0OPNsEsl/KXIULgyCgsHXFWsGynQZlEQg44lCwwf6MDa2SEEkINSb5MfEkPMnOEgAgjmJJKGx3iU/CbuDLeGkRRVKqCamWftmGVkgRfSlmBayrFq9bZUc2Nm0VRma3oR31tFdUKx7Jq6ovi5SLL2rAG41riClUIv2WqJP4xglutVNkHJwgMvBoHLcg9sQ8dOpQGU7YelYRyq8p3MfUpT5R2GzUHMNFMd6DHrA/dMNDjaN21ytmbIjpQCV6mwNXPwp8tIZx685FsUYyYk5xnsQKpIv6WWZo5mqXAoa78P3xlhp+Jsb91j16dl UvTuX/zH MUnIZ0VBeLItQQP9kcqGlmt6fN8QGveUaWeTfTeieituTst4ZyBm0d77CONiPuvLMeLYnAHdlBoBgjOOep/smccX1/IzS5cDaIS3KbSg6dUeS9J1k0gPsdvNBxhDMGHVhEKXhyceYDbFD0prFB4FHxRX/dlSnBSbIw8uds5CmDmb9prf0Dq/FKabPvEGXEW3PI7VhPGiwUgMvg5N/VVA3hkDUGQDcisIm/4tyQqPuPUo1fbjRdNX7lwDN8jGGOs77W6GgbLX0VYHGvJ32TOhYp/x8YTcpelLLbMBgHC9GYkK0teeC4jDcuhCm+7SBrKz8cDYw2zpGUmffuo4LVO28R4CZ/40yvrjA8ZvnesJXXFQr6NB6QoKIhSrp+R/PSRxUfz8qZWns3VQT+GQ+3OBu45MFhYbrdvyNGNUOjfqO5p2freBkgTu6ciNvmAbee0qKQE+R X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags. For now only a flag to allow peer-to-peer DMA is supported. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Logan Gunthorpe cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org --- Notes: ver #12) - Use __bitwise for the extraction flags typedef. ver #11) - Use __bitwise for the extraction flags. ver #9) - Change extract_flags to extraction_flags. ver #7) - Don't use FOLL_* as a parameter, but rather define constants specifically to use with iov_iter_*_pages*(). - Drop the I/O direction constants for now. block/bio.c | 6 +++--- block/blk-map.c | 8 ++++---- include/linux/uio.h | 10 ++++++++-- lib/iov_iter.c | 14 ++++++++------ 4 files changed, 23 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index ab59a491a883..b97f3991c904 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1245,11 +1245,11 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { + iov_iter_extraction_t extraction_flags = 0; unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt; unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; - unsigned int gup_flags = 0; ssize_t size, left; unsigned len, i = 0; size_t offset, trim; @@ -1264,7 +1264,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) - gup_flags |= FOLL_PCI_P2PDMA; + extraction_flags |= ITER_ALLOW_P2PDMA; /* * Each segment in the iov is required to be a block size multiple. @@ -1275,7 +1275,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) */ size = iov_iter_get_pages(iter, pages, UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + nr_pages, &offset, extraction_flags); if (unlikely(size <= 0)) return size ? size : -EFAULT; diff --git a/block/blk-map.c b/block/blk-map.c index 19940c978c73..080dd60485be 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -265,9 +265,9 @@ static struct bio *blk_rq_map_bio_alloc(struct request *rq, static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, gfp_t gfp_mask) { + iov_iter_extraction_t extraction_flags = 0; unsigned int max_sectors = queue_max_hw_sectors(rq->q); unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); - unsigned int gup_flags = 0; struct bio *bio; int ret; int j; @@ -280,7 +280,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, return -ENOMEM; if (blk_queue_pci_p2pdma(rq->q)) - gup_flags |= FOLL_PCI_P2PDMA; + extraction_flags |= ITER_ALLOW_P2PDMA; while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; @@ -291,10 +291,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (nr_vecs <= ARRAY_SIZE(stack_pages)) { pages = stack_pages; bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, gup_flags); + nr_vecs, &offs, extraction_flags); } else { bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, gup_flags); + LONG_MAX, &offs, extraction_flags); } if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; diff --git a/include/linux/uio.h b/include/linux/uio.h index dcc0ca5ef491..af70e4c9ea27 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -12,6 +12,8 @@ struct page; +typedef unsigned int __bitwise iov_iter_extraction_t; + struct kvec { void *iov_base; /* and that should *never* hold a userland pointer */ size_t iov_len; @@ -238,12 +240,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count); ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start, - unsigned gup_flags); + iov_iter_extraction_t extraction_flags); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start, - unsigned gup_flags); + iov_iter_extraction_t extraction_flags); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); @@ -346,4 +348,8 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +/* Allow P2PDMA on the extracted pages */ +#define ITER_ALLOW_P2PDMA ((__force iov_iter_extraction_t)0x01) + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index adc5e8aa8ae8..34ee3764d0fa 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1020,9 +1020,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i, static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start, - unsigned int gup_flags) + iov_iter_extraction_t extraction_flags) { - unsigned int n; + unsigned int n, gup_flags = 0; if (maxsize > i->count) maxsize = i->count; @@ -1030,6 +1030,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return 0; if (maxsize > MAX_RW_COUNT) maxsize = MAX_RW_COUNT; + if (extraction_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; if (likely(user_backed_iter(i))) { unsigned long addr; @@ -1081,14 +1083,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, - size_t *start, unsigned gup_flags) + size_t *start, iov_iter_extraction_t extraction_flags) { if (!maxpages) return 0; BUG_ON(!pages); return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, - start, gup_flags); + start, extraction_flags); } EXPORT_SYMBOL_GPL(iov_iter_get_pages); @@ -1101,14 +1103,14 @@ EXPORT_SYMBOL(iov_iter_get_pages2); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, - size_t *start, unsigned gup_flags) + size_t *start, iov_iter_extraction_t extraction_flags) { ssize_t len; *pages = NULL; len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, - gup_flags); + extraction_flags); if (len <= 0) { kvfree(*pages); *pages = NULL; From patchwork Tue Feb 14 17:13:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF614C6379F for ; Tue, 14 Feb 2023 17:14:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B31D16B0085; Tue, 14 Feb 2023 12:14:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ABA486B0087; Tue, 14 Feb 2023 12:14:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86F686B0088; Tue, 14 Feb 2023 12:14:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 663EC6B0085 for ; Tue, 14 Feb 2023 12:14:21 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3A6C280EDA for ; Tue, 14 Feb 2023 17:14:21 +0000 (UTC) X-FDA: 80466545922.05.F5A2401 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 5ECD71C0014 for ; Tue, 14 Feb 2023 17:14:19 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fs3T2YSV; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394859; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G1Az3EBV9SOKUryI5J/Szq/Ryazp7FGIfAGTSAZ/Hdw=; b=qugMWMIW+R0zKwwhGV1oHpqhFaprLc6cQ6wjIxDuoYYCPRqrjvNUe/5+uBwnAQV+tv7V8j KI5FRXy7Kb60xj2is9qTPe3u5KFWC1mK/qBdCs+36pSLQM5DCA1FHdyF0GomWmaTAOEbEa wAVCJPcmKOhFtMncp4tyFPEy4Pci248= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fs3T2YSV; spf=pass (imf20.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394859; a=rsa-sha256; cv=none; b=2LyaRG47HyRQ26aigP2hazHz0tlMmPncGnxBO+WWJV56frXQMog0OBuAgOXOTJs3RtunSu bGNCX1f/hKuYGkFXRaDzfjmIhU+2IJcPBVxdOKw9TbDQoNZqGsoWoQvjoejsr3cYwvkoWn ac22q6uCkKA2HBd+tfP1Rdk0aPi0lPg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G1Az3EBV9SOKUryI5J/Szq/Ryazp7FGIfAGTSAZ/Hdw=; b=Fs3T2YSVnnFkvGoHAzqasA2nOXHlrnoVvBUowtdJXgp063QXqXcVFzEzRbREw4eu7abTIA QipKaOevdlLIFdYYe1ffw+0hulX7/ZJ8KQLlhlwrEVIvXd7R+ZtdzRS7/YOhHcu82mD0et F6Wr350NCoC+zWO7cBFWpn+YlkYKWVE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-564-UFOQF0M0P1K7pE4T4FEjsg-1; Tue, 14 Feb 2023 12:14:17 -0500 X-MC-Unique: UFOQF0M0P1K7pE4T4FEjsg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 601132817236; Tue, 14 Feb 2023 17:14:16 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 42A961121318; Tue, 14 Feb 2023 17:14:14 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 11/17] iov_iter: Add a function to extract a page list from an iterator Date: Tue, 14 Feb 2023 17:13:24 +0000 Message-Id: <20230214171330.2722188-12-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5ECD71C0014 X-Stat-Signature: keqh1eyaoh3dxfqwgf5ttt8r9jhtm7rn X-HE-Tag: 1676394859-558911 X-HE-Meta: U2FsdGVkX1/D7b3pCeA63H37n/UEZSBP0p1rPU06W7YlvD+juHBVhFdWcahIvtuY4X+xBPHRIZNlkPvmJWy7xpdwwFwC4BbzHVkl/Lwg0BtyISL7riIem8QCOSqnAayyecKpP5sM53LqwJmVel8o7K49rIASH0sKyBF3/eUuBuvdjF+BEoc1wG1oFdHkfXe1Aa5qvdEJVKJDRAJkVL2aqFMTOsikXH9AfacbpAe4XoQULhI4YyckJUcQxMjEb3FvImg5mDLeTMsf8P6Oows3L+fw3PiJmbLfxTBIrfQhgpA9ggUFYceeuFu6hcpEx2G4xpv6l3mnsf53PUgy1HWK2WC/LUM0sjqzgoQgSsLMHTcWSWp9GFHcSBKUho0l+ZudlcF8YCAXut/G4OL+6q+/V1WX/nXPtMBG3OGBPHHQwUFvuuhw9F2a29dIMW+FTI+Z5OpuCzecO/zlvSCSNN2HZPwX5hqFXFYUUubYDzJsKFFDTrtLXihDpmS96aGMh24cpcigUcB4WqrZgIAHR6f9No3aof/qqDa6IE5QCIv3sOlqMVUxVtm5VjfENid9RGDbTQG+hjb2gDDi1LqdVXBU1K9HBE2mH1jDz4DR4h8orUpTUaQdAgUCrG4VdPuxaKmi3g0uax2XBBz9hmxCDKvXlhalKj2fwwqeXUtG1YgBTvuvJ57V8QymMU+4UFScqWsZN9nvZDnJKcRU3OTlOeF50QbtVse0S6SXGrWUfOOWBnmSaPSYcihL5USNFtWGZkmJ4FiUsw21BnZGnvTivjVJuxi1d+mS3i6k8ot3URZ2b5DyUPVzqrAue/QclzlT1qCE0uwfU40tTaYwUUNsgg6dTYdjuRwDf6W3I92TT3YMpK1dioPICUrpkSg6ggN/Lgb0jH/YZktMPZpFFXkTW1tFIinWD0ep0uywzQS+FdNvnbGZSTBfbYmcbk6qvu+6JLILZzjjKaQeqlLL0zcvwIB E/hNdVCe Q6ZEz8/YoO1Uf4f3JZKIWcU+CUtBgoW14h1e29VW00hkm2kVNQfQE3epltfT9pU8Skz/5Ee6W9RWW5dvI6VQwJaAjqVNRsCjX3AFVxcmAqPwNMCAHuUMZG9lDBOI8xKQ1UcJ99H374jjHQ+H7Z/1SxM5FUpTzei1l+xKJXX2zSQzMokotFIIMpRHjoLeDXutv8K1UyoKTZWN5sV1vJ8mZxmI8vVlhs1QLd9sdrQonNXiSCfQycaQTjAP7lDAaibJTZ/TlueOyiYDeWUo1M4I0ajdh24GZPLQ3o/MUqf+fZLIsM1Rge65bMIR0Wj+3x3zrpkbBW6mAZGji/v50+KZus3pDYXZoStmQpn3j9vppu1Ct4aPcYJU5jomZAotbK3OMXjl8d+z2eMyJNUiEujO+tHAFiaO3PehZfOHd2ft3KDdQMdOdmJecLV5DrW7pwTz6b7LnmZtTX/iFsYwNgPBRc/MKLtMkcZdewV8o4XRIzu3IHSj3VKA9Y2iLp55CBzENNJOe9hBDlKT2nS/EQI4LppjEGFAfhpIjrHHayBpfVDJmFvgTnfnIQ4YxwA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a function, iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a pin added or nothing, depending on the type of iterator. Add a second function, iov_iter_extract_will_pin(), to determine how the cleanup should be done. There are two cases: (1) ITER_IOVEC or ITER_UBUF iterator. Extracted pages will have pins (FOLL_PIN) obtained on them so that a concurrent fork() will forcibly copy the page so that DMA is done to/from the parent's buffer and is unavailable to/unaffected by the child process. iov_iter_extract_will_pin() will return true for this case. The caller should use something like unpin_user_page() to dispose of the page. (2) Any other sort of iterator. No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention. iov_iter_extract_will_pin() will return false. The pages don't need additional disposal. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- Notes: ver #12) - ITER_PIPE is gone, so drop related bits. - Don't specify FOLL_PIN as that's implied by pin_user_pages_fast(). ver #11) - Fix iov_iter_extract_kvec_pages() to include the offset into the page in the returned starting offset. - Use __bitwise for the extraction flags ver #10) - Fix use of i->kvec in iov_iter_extract_bvec_pages() to be i->bvec. ver #9) - Rename iov_iter_extract_mode() to iov_iter_extract_will_pin() and make it return true/false not FOLL_PIN/0 as FOLL_PIN is going to be made private to mm/. - Change extract_flags to extraction_flags. ver #8) - It seems that all DIO is supposed to be done under FOLL_PIN now, and not FOLL_GET, so switch to only using pin_user_pages() for user-backed iters. - Wrap an argument in brackets in the iov_iter_extract_mode() macro. - Drop the extract_flags argument to iov_iter_extract_mode() for now [hch]. ver #7) - Switch to passing in iter-specific flags rather than FOLL_* flags. - Drop the direction flags for now. - Use ITER_ALLOW_P2PDMA to request FOLL_PCI_P2PDMA. - Disallow use of ITER_ALLOW_P2PDMA with non-user-backed iter. - Add support for extraction from KVEC-type iters. - Use iov_iter_advance() rather than open-coding it. - Make BVEC- and KVEC-type skip over initial empty vectors. ver #6) - Add back the function to indicate the cleanup mode. - Drop the cleanup_mode return arg to iov_iter_extract_pages(). - Pass FOLL_SOURCE/DEST_BUF in gup_flags. Check this against the iter data_source. ver #4) - Use ITER_SOURCE/DEST instead of WRITE/READ. - Allow additional FOLL_* flags, such as FOLL_PCI_P2PDMA to be passed in. ver #3) - Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1]. include/linux/uio.h | 27 ++++- lib/iov_iter.c | 264 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 290 insertions(+), 1 deletion(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index af70e4c9ea27..cf6658066736 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -347,9 +347,34 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, .count = count }; } - /* Flags for iov_iter_get/extract_pages*() */ /* Allow P2PDMA on the extracted pages */ #define ITER_ALLOW_P2PDMA ((__force iov_iter_extraction_t)0x01) +ssize_t iov_iter_extract_pages(struct iov_iter *i, struct page ***pages, + size_t maxsize, unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0); + +/** + * iov_iter_extract_will_pin - Indicate how pages from the iterator will be retained + * @iter: The iterator + * + * Examine the iterator and indicate by returning true or false as to how, if + * at all, pages extracted from the iterator will be retained by the extraction + * function. + * + * %true indicates that the pages will have a pin placed in them that the + * caller must unpin. This is must be done for DMA/async DIO to force fork() + * to forcibly copy a page for the child (the parent must retain the original + * page). + * + * %false indicates that no measures are taken and that it's up to the caller + * to retain the pages. + */ +static inline bool iov_iter_extract_will_pin(const struct iov_iter *iter) +{ + return user_backed_iter(iter); +} + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 34ee3764d0fa..8d34b6552179 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1487,3 +1487,267 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) i->iov -= state->nr_segs - i->nr_segs; i->nr_segs = state->nr_segs; } + +/* + * Extract a list of contiguous pages from an ITER_XARRAY iterator. This does not + * get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_xarray_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + struct page *page, **p; + unsigned int nr = 0, offset; + loff_t pos = i->xarray_start + i->iov_offset; + pgoff_t index = pos >> PAGE_SHIFT; + XA_STATE(xas, i->xarray, index); + + offset = pos & ~PAGE_MASK; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + rcu_read_lock(); + for (page = xas_load(&xas); page; page = xas_next(&xas)) { + if (xas_retry(&xas, page)) + continue; + + /* Has the page moved or been split? */ + if (unlikely(page != xas_reload(&xas))) { + xas_reset(&xas); + continue; + } + + p[nr++] = find_subpage(page, xas.xa_index); + if (nr == maxpages) + break; + } + rcu_read_unlock(); + + maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from an ITER_BVEC iterator. This does + * not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + struct page **p, *page; + size_t skip = i->iov_offset, offset; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->bvec->bv_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->bvec++; + skip = 0; + } + + skip += i->bvec->bv_offset; + page = i->bvec->bv_page + skip / PAGE_SIZE; + offset = skip % PAGE_SIZE; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + for (k = 0; k < maxpages; k++) + p[k] = page + k; + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of virtually contiguous pages from an ITER_KVEC iterator. + * This does not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + struct page **p, *page; + const void *kaddr; + size_t skip = i->iov_offset, offset, len; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->kvec->iov_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->kvec++; + skip = 0; + } + + kaddr = i->kvec->iov_base + skip; + offset = (unsigned long)kaddr & ~PAGE_MASK; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + kaddr -= offset; + len = offset + maxsize; + for (k = 0; k < maxpages; k++) { + size_t seg = min_t(size_t, len, PAGE_SIZE); + + if (is_vmalloc_or_module_addr(kaddr)) + page = vmalloc_to_page(kaddr); + else + page = virt_to_page(kaddr); + + p[k] = page; + len -= seg; + kaddr += PAGE_SIZE; + } + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from a user iterator and get a pin on + * each of them. This should only be used if the iterator is user-backed + * (IOBUF/UBUF). + * + * It does not get refs on the pages, but the pages must be unpinned by the + * caller once the transfer is complete. + * + * This is safe to be used where background IO/DMA *is* going to be modifying + * the buffer; using a pin rather than a ref makes forces fork() to give the + * child a copy of the page. + */ +static ssize_t iov_iter_extract_user_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + unsigned long addr; + unsigned int gup_flags = 0; + size_t offset; + int res; + + if (i->data_source == ITER_DEST) + gup_flags |= FOLL_WRITE; + if (extraction_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; + if (i->nofault) + gup_flags |= FOLL_NOFAULT; + + addr = first_iovec_segment(i, &maxsize); + *offset0 = offset = addr % PAGE_SIZE; + addr &= PAGE_MASK; + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + res = pin_user_pages_fast(addr, maxpages, gup_flags, *pages); + if (unlikely(res <= 0)) + return res; + maxsize = min_t(size_t, maxsize, res * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/** + * iov_iter_extract_pages - Extract a list of contiguous pages from an iterator + * @i: The iterator to extract from + * @pages: Where to return the list of pages + * @maxsize: The maximum amount of iterator to extract + * @maxpages: The maximum size of the list of pages + * @extraction_flags: Flags to qualify request + * @offset0: Where to return the starting offset into (*@pages)[0] + * + * Extract a list of contiguous pages from the current point of the iterator, + * advancing the iterator. The maximum number of pages and the maximum amount + * of page contents can be set. + * + * If *@pages is NULL, a page list will be allocated to the required size and + * *@pages will be set to its base. If *@pages is not NULL, it will be assumed + * that the caller allocated a page list at least @maxpages in size and this + * will be filled in. + * + * @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA + * be allowed on the pages extracted. + * + * The iov_iter_extract_will_pin() function can be used to query how cleanup + * should be performed. + * + * Extra refs or pins on the pages may be obtained as follows: + * + * (*) If the iterator is user-backed (ITER_IOVEC/ITER_UBUF), pins will be + * added to the pages, but refs will not be taken. + * iov_iter_extract_will_pin() will return true. + * + * (*) If the iterator is ITER_KVEC, ITER_BVEC or ITER_XARRAY, the pages are + * merely listed; no extra refs or pins are obtained. + * iov_iter_extract_will_pin() will return 0. + * + * Note also: + * + * (*) Use with ITER_DISCARD is not supported as that has no content. + * + * On success, the function sets *@pages to the new pagelist, if allocated, and + * sets *offset0 to the offset into the first page. + * + * It may also return -ENOMEM and -EFAULT. + */ +ssize_t iov_iter_extract_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + iov_iter_extraction_t extraction_flags, + size_t *offset0) +{ + maxsize = min_t(size_t, min_t(size_t, maxsize, i->count), MAX_RW_COUNT); + if (!maxsize) + return 0; + + if (likely(user_backed_iter(i))) + return iov_iter_extract_user_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_kvec(i)) + return iov_iter_extract_kvec_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_bvec(i)) + return iov_iter_extract_bvec_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_xarray(i)) + return iov_iter_extract_xarray_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + return -EFAULT; +} +EXPORT_SYMBOL_GPL(iov_iter_extract_pages); From patchwork Tue Feb 14 17:13:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D6F9C61DA4 for ; Tue, 14 Feb 2023 17:14:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F29266B0088; Tue, 14 Feb 2023 12:14:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EB0C76B0089; Tue, 14 Feb 2023 12:14:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2B3D6B008A; Tue, 14 Feb 2023 12:14:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BE41C6B0088 for ; Tue, 14 Feb 2023 12:14:27 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 67246AB66B for ; Tue, 14 Feb 2023 17:14:27 +0000 (UTC) X-FDA: 80466546174.05.87CCD80 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 95381140015 for ; Tue, 14 Feb 2023 17:14:25 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BCSrFSm9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394865; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AA3SDxqAw6tV+jeEhwhhXCCPQx7iw6NiGu/cAPT0aQk=; b=gsgCqmxRwJNDbM1rKLGcbOdo3wICfCQoKkk2cVl8iJj/PMsMuXEcvLJGy8cPos/xhEkNMI is82512/gU9s/Pk5+6CjUXMsEgQSDcxzkiXJ7hRNLu2t4g0IxpO84hZcnbp37WhOwVb1u9 /uSf3laes3z6bgqFDkO7OVpCzse0Gco= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BCSrFSm9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394865; a=rsa-sha256; cv=none; b=AG8Hyu7Zdnq+THgY7xWa3JcQpcuUL1gMkw2v6GFCmDVYkhHcrORiaHN0m6OJhrM3xBYBxp T6g702bNGKB93Zoo+EnO4XkiazpfC1DYHVMbMXzFoafYjc3cpIAxn+5zLbzhg4ScmTIICS ychFrmaecExknVgRjRXiBLJQ7PmZ60U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AA3SDxqAw6tV+jeEhwhhXCCPQx7iw6NiGu/cAPT0aQk=; b=BCSrFSm9m4vCFf6Z47aB/J5gIS1hhLR9Jora5YI3jbQWFJxTF6Dk9i7xJS2VU7WmjNZILq HHtLpHmi6534FdaMK1rjGeqt2Yd+EDp+u5sbZehD45CDaUGyULi/wh6owDjNebyE37s8a4 WeQMySuKgnsoqnNVSu4XO97LQGV267A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-515-DQOy9zFmOZSICpQlPKxSDw-1; Tue, 14 Feb 2023 12:14:20 -0500 X-MC-Unique: DQOy9zFmOZSICpQlPKxSDw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EE7998027EB; Tue, 14 Feb 2023 17:14:18 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 179BC2026D4B; Tue, 14 Feb 2023 17:14:16 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Hubbard Subject: [PATCH v14 12/17] iomap: Don't get an reference on ZERO_PAGE for direct I/O block zeroing Date: Tue, 14 Feb 2023 17:13:25 +0000 Message-Id: <20230214171330.2722188-13-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 95381140015 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: dnj9u39x6kzf6m8ux6c4ydh8fbrcq51i X-HE-Tag: 1676394865-288265 X-HE-Meta: U2FsdGVkX1/yoUfjz5Zs+xCZ7Vwa0HJ29cTpP2chQ49LcL2dtvTSEnuZH9ICbfvT5hgd8vqLMeh8oqe+fLKstmoXdey+BFeKfNWcveCkccuDXZrHvn+XXyhfjKy4yh55+oWRcVJNppOQXaWHHWYjdjOZ+tIl1voWk7GmjuzAhGPuYL2gt58pF20oOd4WdZb+KC9T+x1qQOMGCSsHjcifJhtkRFo3iDAwj0aO+u5Jp8+9rWOPn+oH8zTLGwenkmyIl4VY7RpHgahDTpFh6xoEwvjj2GyJ0oB7AqxFeB5YBipW6jUfvifXR4wkD+r4kqYaH6q1TJifnmNavirtdO+lhgE7Ze7xDZ+mEG5vkcSjwHgWhbU+WuOe0FYS4LBcRcf/ZV6nJauTdOySFd3magcaH6dhlt8RqqfwYbhgnYuMJqi7FHPlZv5ZMMSLMxDm+obRL3N2SwoJIjaw28IUPRHS0JASA27n1FJlgJpWQGaaRVUg+yewAZvHEA4HQOj2rhhVCGW35L0YP0Z6NUCpfXIE0Otoa2qJHSxUJx1nHcAxDPTA6d5WIlKVPlfX3lEqiROKLIFCcEnUnMqkWtUJclIeDvH2LdpgmXXyMivptmZNniLdaicrGPLBp3l9LN538ff0+BCMGOsFPQOQYISqHxOFEEYGKga/Qw0O7l4UuYbN9AZHttoJMaSfZdP0isi6BsfliW64MJRedlnHc0S1rYz9VyUw5oCdPJww/nvjuMkFc0/0VwjqzymlEwWnaw2dVMbO2IRBydBVH9v+kQy36rb7mqg4oPfXz3YDRUolTfv9O95Lk3VU+O9jIwhYHGnma0RLSkjvEZOb0uC0M0fBscvOKdajRDH7zpaOH7OrXlmPcyjlBRaklFmI87ITGnEYn82Rand/SRo9Q4NOcSlTko0UtWy0A8DjfWQ9jOqKc2yAAYxfyjwIuEwdi9IfvUViwozqomTXi4bKlQ144ZRo2re OrixbcQR qrx9tDIsmImPJ7khD3XD8JwGBTpBfGGl0ISVRIBQqkmmtecksVNN5sZmJrtTsNJYmC0UqbQhc7/Zx66aIZa5eA0Ikmyx0HgWf5V0PvXrYg2Z35BxAnd5avOO0p0NzDTsywK3r6/r89yqOgPL5F4w1OQwvmqamWV/zdot7nalJRtWiTq2H4VA/Djn+DfsSU6G8/BuWtrC7ZtH7zbuSfGhVr3wjLFDTcagW4gtb8E55FK9hVY5tWJ5KG0RtphV3JLqFXYsdEPN9wnirJ2MHUAEkrw9x9tuj/mQXd8wORtzB4N3MaJ3OrIenHejL6VJK+exqR0EXpZTsK3TgINdXV3Q12VvIxRF4t8lht2IQklAQShvouQBP0aMLhEPJsF20YwPI51bz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ZERO_PAGE can't go away, no need to hold an extra reference. Signed-off-by: David Howells Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard cc: Al Viro cc: David Hildenbrand cc: linux-fsdevel@vger.kernel.org --- fs/iomap/direct-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 9804714b1751..47db4ead1e74 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - get_page(page); + bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } From patchwork Tue Feb 14 17:13:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45BA5C6379F for ; Tue, 14 Feb 2023 17:14:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D30196B0082; Tue, 14 Feb 2023 12:14:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CB6D66B0089; Tue, 14 Feb 2023 12:14:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE1776B008A; Tue, 14 Feb 2023 12:14:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8EA546B0082 for ; Tue, 14 Feb 2023 12:14:32 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1DD5712024A for ; Tue, 14 Feb 2023 17:14:32 +0000 (UTC) X-FDA: 80466546384.22.787290E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 39F741C0017 for ; Tue, 14 Feb 2023 17:14:30 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cFjtMRy+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394870; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9PNg3WPaFWzxoLjSof2qU8psox3R+qpqOMDOMyjiLi8=; b=AyuiWmFghFkCO2CDna70j80ucUfORn9KeARVkcX1abM+dHgAeivCNbmCuF7noX689gf5fK YRXxrIvnyjc0RMdptZpVz2e2TlpAPapHgv/p/oD9BKOTNW0j6xYpStut8iJV6iIBIm+pdA F4AkcaJvBfgZGJ2RKX0tEQDr74aA3SE= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cFjtMRy+; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394870; a=rsa-sha256; cv=none; b=7EDRwA524X/t8vS+d+o7GnqMVuhl/exPkT+xjYzRAM+PWbewG2WXt3YbRGrruqyepIdw3p HwowFtjHzP+/Ukz6WxbEXruPioZT91YOmZv2on9aTIO4bUBvl8Ja1IP6wqRt55Bx1cgAXU l4PKvfsZAGvJw7esUn/gJL4hdyDtnXY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394869; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9PNg3WPaFWzxoLjSof2qU8psox3R+qpqOMDOMyjiLi8=; b=cFjtMRy+FpHk/63IaSJEpFZgOmHqIkVgvgee5SWknVKflChtvzVrdw9D0batV7nlLdaTPr 2GKtCRrkzMWHcAs4zR/ODoeS2OETQGCmdTxPJwgo8MRGVh7BYMzFsk1efyYqkjiR4+1PLw S8pmOZ8OSGtEqBQda2pwv7eIQTJaHk8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-88-Gp-NasbONamSrWK0LXoPrQ-1; Tue, 14 Feb 2023 12:14:22 -0500 X-MC-Unique: Gp-NasbONamSrWK0LXoPrQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8698B1C08798; Tue, 14 Feb 2023 17:14:21 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9817DC15BAD; Tue, 14 Feb 2023 17:14:19 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 13/17] block: Fix bio_flagged() so that gcc can better optimise it Date: Tue, 14 Feb 2023 17:13:26 +0000 Message-Id: <20230214171330.2722188-14-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 39F741C0017 X-Stat-Signature: c8wqzfpqzf8b51ndbutqe94fp75p1ftr X-HE-Tag: 1676394870-91500 X-HE-Meta: U2FsdGVkX18JZnENfd1OiM7yyKXp4pXh7exi0T+8jbF8zRguke/e+woTU/epbPQk+oJ4hOtvYKktg0lWJ8eeQ5IbFkVfDnCRLDAfECCwqIJE+InPI/iSWwhM/hZpwJdYIHJF/3d14/0LXg67795VCEtw3n3zDmNtjp5yLJLowcWJurCXuNNSl0AdoygiFgx9dcmMk9s7F13tzdEwDx9HEdMV7wrbojQu5+Zf/9AiJ4kxK12N4j1kp24bto2uvJu014RuFPeNwJlOSeMxUiAvjxijBnFzIvJ8TxxHQmKgdQ324M3nSFmgghCcNfzVo0gwZZ5vlN8M4EcLSWHfVDckoczrhngKe6mg+s1Za/EVOd5KaTW8Eag0pTD3HjRMi4f1uLV6jPKzyt7luL0X4qvmJDdJKF1TtcI1MTwWSOdot8/F66AyiR6Z18SVTm0ztbBEk/q18zx7juWj3Az6/KXtIHcDghqbf33Q5nyn7FQsFEWnXZZ5vFga3iYPPMAX1vTkT1Hnbe6TB8iBXvNMMz3im4a2626hKzdN4ju0DiuNwhPI42dSVgXqYn6KUT+0rUJbjsO4IBYkgCwO1OuHW9/XZI1X45BA/NDsig+/mhKb2C48WQ/QTtjfPSR48umXQNV6UmS8IepVUxgrTZgQs/bo/TELT6cgBCLkylSYnSquWv/F3JnVNdy3NV+TobqrxDg/HQVUICutoZPBwB6LhJTfePl3mtEjp7iHd3cLgsKbrnw27Mw2ksqEwj5xN4IbUK9tXIXfKQCZXNXfnC4H5olxS1AEeZqzCOwlpGY+py4qoauha3pOm/NFdquXVjdA41vozQ1YItA6YzMPfDPnsIIion1C3T5elxCprxKWUiChYypUkCCjLbCMDJdPGvtDap1fEgH/cCStUfVEvqQmDkekwxZYivpLrQJlC17Y6ig0Xo8o5CTWJZxRdYF1TlAmW0JpsYZTIJfpkrA6zXTcSwA umJBLZk4 OhzNk1isyvb1VYqcRNJ8D8auQQEHD1QIApLdEYC/lYMNMfXCRBULMTe8TvYcY8MBPkl0N0PVbe9jJFwI3Q+1FbcIOy/+OqCqbMis/dn7yMR7hiHlZhZmAF2XuVicNHtuFiXccdH33m/CBxL9CSLeSHvhCtq/W3F2ewoUlhPr0QOoRnSnqphYYAeYePypp+iupYOLCFJgpcOizioZa8v78b5n456zW9ejIAZVb9XW+ePp0ByvK5vY/c8dG+QqtqzEVJz7KFAAeRoYUaY6BCrAwgu4X07vs4fA/gRRdbCaKHJAbJwKXcOuUtBRt8/XUYbj70qsMeNspCXQwz/dxP65ShUpxWdEBGXnYSkj1OlCzTYBPSdHKw7JM/S6TGRC2e1kXv/yNohxKJ96cpR3YZGjhIJ/7J+hWcrXk8F2kCh/A1Nqn8PApn5Bg3cejCLHaCfYlt3DS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix bio_flagged() so that multiple instances of it, such as: if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED)) can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]). The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like: <+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk> However, the test is superfluous as the return type is bool. Removing it results in: <+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk> instead. Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Jens Axboe cc: linux-block@vger.kernel.org Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108370 [1] Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108371 [2] Link: https://lore.kernel.org/r/167391056756.2311931.356007731815807265.stgit@warthog.procyon.org.uk/ # v6 --- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/bio.h b/include/linux/bio.h index c1da63f6c808..10366b8bdb13 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -227,7 +227,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count) static inline bool bio_flagged(struct bio *bio, unsigned int bit) { - return (bio->bi_flags & (1U << bit)) != 0; + return bio->bi_flags & (1U << bit); } static inline void bio_set_flag(struct bio *bio, unsigned int bit) From patchwork Tue Feb 14 17:13:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE6B7C64ED6 for ; Tue, 14 Feb 2023 17:14:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8238A6B007E; Tue, 14 Feb 2023 12:14:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7ACC56B0082; Tue, 14 Feb 2023 12:14:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FF8B6B0089; Tue, 14 Feb 2023 12:14:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4828B6B007E for ; Tue, 14 Feb 2023 12:14:31 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 223B91606AC for ; Tue, 14 Feb 2023 17:14:31 +0000 (UTC) X-FDA: 80466546342.28.B196E5F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 7115BC0016 for ; Tue, 14 Feb 2023 17:14:29 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LlnHaPK0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394869; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dqniJi1fcm9DuVb2jGUzwy7ATEE/wnT//KjRosSsKEA=; b=xNdVf0+RXwGwvaEWynga8qEhV+LKw7FFn/ulVgvC+CVStXMRMnBorGIjMPHHuY2PEyX0Wh lKKeAGljNPcRUt7nbxYiWYWDFfjLjNh9512RwFMHpoFkfQyEPltg2N4hJ7YY+IJ9rOatPU 7mCk01WeKqVWOmZCyRVBdP+JDiusosc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=LlnHaPK0; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394869; a=rsa-sha256; cv=none; b=35XpmEyTduIb0tL8IfCTBZXn/2L7BdzChlmC3fTN6Y26RWMJxgVTTiNhzBC6HZyGI3/4XQ Hnd2JKULbFfdBBsPjrEy1xIry4+DcOkKtLDHqV9w3OU76ftmhs5l411VtLQj7syTKg48vX dwmpRxJjHHIfT8NWhv67otbSrh1Yu68= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394868; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dqniJi1fcm9DuVb2jGUzwy7ATEE/wnT//KjRosSsKEA=; b=LlnHaPK0NvrQVvkv3Nv1y3lmLSlNb5eVRmfB+/SvEj+zIep3SM6WP+tT5TE1FCGQi0RVEW 4moWoM8qvSXjPtvhBeCEHTeCqSCgWSoET5rdBMDl+rXFdhE8QbkZ590M9ietY8M888RgKp 0RkoH6Hi/YhDjyRlMPncgtf+yf7mLYc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-376-CS34Hu8FOKSxyZVTcNhLCw-1; Tue, 14 Feb 2023 12:14:25 -0500 X-MC-Unique: CS34Hu8FOKSxyZVTcNhLCw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4250A85A588; Tue, 14 Feb 2023 17:14:24 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4367C2166B26; Tue, 14 Feb 2023 17:14:22 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 14/17] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic Date: Tue, 14 Feb 2023 17:13:27 +0000 Message-Id: <20230214171330.2722188-15-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Queue-Id: 7115BC0016 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: rgbpxcfgjoy5e6cxdshrr7h87rwr4yqr X-HE-Tag: 1676394869-813365 X-HE-Meta: U2FsdGVkX1+Zy1MCEOudM2gkN50WQ/BJriTexQfErXcEbco3NO7Vj7zKXA/K7N6bjkvVpfHGOB0EyIVaI1pGGdJBGE09XF8Uh1rroSsL232bP5MCTXT7UrcUq+mgxFTTzbWXskh4+5b7Mh0AwGT+ZX76hqMqnnKHKiCvGgYfvAuuyytMe5HJckVOyN2VvsG1eVMnOvVU6+A43tCrUeHSVw+MTXKxupzIHaBcT3h88RQKwlL5d81BRQDj8Ctl6gPVdRi+ryJXaxcvpQdvVo2PFFmRk/LQwrrlFr2xIHTjVDlp/W3eomRs5fRYMvG2Yw5UfdQn3cjtFikcVTNxqJOjmJQ40x/9T8syYJRYi8sbHy9cZqFOfcbYYU36P5WPmDlpSdg2bawnii9zAkl7YNaPxaI+tC1rTBkuVZtefLAILry4Tpig++As3zNK2/oPhFkTrbfqwF3Y9h9heIvCu2w1qQuplP7UgrnCIPHqASfIGZHuxG5/6ifnzuzLoRw+LyQsxgylLIDvCQwCjkyR2Q1BFNCAcTshLJtIg978RxQ1tDpKkUhWGdPTqFbCA4RWqeynMI5tAImBc5g96jpi5lBAP3yaRqoL3bXDno8LDG8AUWWlgU8C82qrtdVVx2hlMLN+xXy77tnSwRLst8OhcZ0We/l7tCCyJq8Z10fGR6a5RW5cQRu8N5q1nKyhYMkI/wqOqCewO8TbwRwOlTxgajuKhngys1y26B8G6NTQN5+soxRwm9FWI9cbtqDTljy4dVFYhaecdVe6jQC/rYzoE5GCSfZP6IEFw6lRYHSTt4ZD9PZ8tUGwnd99wp65otmRuJHgrpPF3BSzZnYhNhV1gqof4tqsZxZFrH5BZhSsmmCU6+2i2y+4Fn3Vj9uBNhd6caYAZCKA+bt5tsB5gPK4RD2JDzhRtWmLXpayKgvmeG1v6ZPhWUulwA4uFZ2vXCn3tJ6SUh7HkMjDIk8gBxvuC/Z /EBdRe9M 287FCRIUj2MuUqh7v4Ia3U0MR9EYVnWFnZkF77zTDvqs+WBmCfItU9lYYla1pMN/f2F4Nia92ydPCpbo3YZZB5Q7+8QTMMUNk9NP4niB7EcKchmQK5dNCuTPOoiQl7y/NpUC7I8cZ0Lg1ZShgVDinFZLJ7SM1Cg1kofA/uqVutapNj97kI9I+S5+ti2iimqovBRjpFf15h1B+Uj5y6bD0RZHlmZi/3xWgmQNz3G9ieSBOGsa39iO7JJTWmxY9YpwDxoeoHjem28bbbKViZJrxSkEMjix6yXcSGk69Iy/9b4BWukPZXV/F55z1jykKIfkyMsLHeWUfoxwpUK+UYekCWU3lU5ilYP0qp5FC8YcdcZaAmlw3qO12VNlV9X6fHSu9YUPYXExbRQ3vYOhLi5VIUDMwr9L49P1Y9IiYXhgXmuJFfFEj3F6ywTniIzBqhaXUj33fAn52GP9iWk8a5LdxpDo32zflpHZ6fMKBDl3Ib4hpxDf631UAQh+uGw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Christoph Hellwig Replace BIO_NO_PAGE_REF with a BIO_PAGE_REFFED flag that has the inverted meaning is only set when a page reference has been acquired that needs to be released by bio_release_pages(). Signed-off-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #8) - Split out from another patch [hch]. - Don't default to BIO_PAGE_REFFED [hch]. ver #5) - Split from patch that uses iov_iter_extract_pages(). block/bio.c | 2 +- block/blk-map.c | 1 + fs/direct-io.c | 2 ++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 6 files changed, 6 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index b97f3991c904..bf9bf53232be 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1198,7 +1198,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; - bio_set_flag(bio, BIO_NO_PAGE_REF); bio_set_flag(bio, BIO_CLONED); } @@ -1343,6 +1342,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } + bio_set_flag(bio, BIO_PAGE_REFFED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); diff --git a/block/blk-map.c b/block/blk-map.c index 080dd60485be..f1f70b50388d 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -282,6 +282,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extraction_flags |= ITER_ALLOW_P2PDMA; + bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; ssize_t bytes; diff --git a/fs/direct-io.c b/fs/direct-io.c index 03d381377ae1..07810465fc9d 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -403,6 +403,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_end_io = dio_bio_end_aio; else bio->bi_end_io = dio_bio_end_io; + /* for now require references for all pages */ + bio_set_flag(bio, BIO_PAGE_REFFED); sdio->bio = bio; sdio->logical_offset_in_bio = sdio->cur_page_fs_offset; } diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 47db4ead1e74..c0e75900e754 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } diff --git a/include/linux/bio.h b/include/linux/bio.h index 10366b8bdb13..805957c99147 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,7 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (!bio_flagged(bio, BIO_NO_PAGE_REF)) + if (bio_flagged(bio, BIO_PAGE_REFFED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 99be590f952f..7daa261f4f98 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,7 +318,7 @@ struct bio { * bio flags */ enum { - BIO_NO_PAGE_REF, /* don't put release vec pages */ + BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ BIO_QUIET, /* Make BIO Quiet */ From patchwork Tue Feb 14 17:13:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E63EC64ED8 for ; Tue, 14 Feb 2023 17:14:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E34C6B0093; Tue, 14 Feb 2023 12:14:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 121576B0092; Tue, 14 Feb 2023 12:14:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB1406B008C; Tue, 14 Feb 2023 12:14:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D4F316B0089 for ; Tue, 14 Feb 2023 12:14:36 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 983E6C0911 for ; Tue, 14 Feb 2023 17:14:36 +0000 (UTC) X-FDA: 80466546552.30.9E8E702 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id B8ABA2000F for ; Tue, 14 Feb 2023 17:14:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hzNH40xw; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394874; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GZNnQ4PFOAio/M36ha2lSqPmGocJT5Xn1bbdC3s23DY=; b=XFypwnPWQzTY2MsLmElSiVWtC3ZDFEr0QDb2NzbdS5n4VqCeFEVq/5yszqlN+GNiYB0zUx jHEosBOdJ6A9v14wJ+LC7ceWjH4nIvoMoPEW/1y92nSU1+SEOnOhHjDWoR8/KKRXa6llaJ /0V1FfiXOFxQU4oExJVXmFNkqh7dY5Q= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hzNH40xw; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394874; a=rsa-sha256; cv=none; b=lRptP1cbEzTD8S4QUzVD6UgGoMTPO8Ea4TmLGmQbdwR/vuqplU2bPYSuN/MVuC+cJ4yACa hGB4lfrORphgl72gFNj/jl5p11z9EdvNLtURjS8TXURFrGKz6TCzzqvx5jat2B6ulpTgsj g+YwHTFAYn6Kk08pyVPzXtAO0woMH2A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GZNnQ4PFOAio/M36ha2lSqPmGocJT5Xn1bbdC3s23DY=; b=hzNH40xwGJW3IzDzwGkclmKDkHyfPA5aBBlhzL95GGJFvXUDXY8ELsSDrJdE9YJ77ROFJX w4AW0qGBxyYYUh6j2e3YZK6vYPwguzAbbsC0pObLT+Pg1Mhf5e7R7Z6WeN9eA9H/NYJI0u 87Yr5o8mA6wHRrF11p97THHZlirWVfc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-466-lS0UXfJPMyu2AbeIcCgt5A-1; Tue, 14 Feb 2023 12:14:28 -0500 X-MC-Unique: lS0UXfJPMyu2AbeIcCgt5A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5E66A38123AE; Tue, 14 Feb 2023 17:14:27 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id D969340C1106; Tue, 14 Feb 2023 17:14:24 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 15/17] block: Add BIO_PAGE_PINNED and associated infrastructure Date: Tue, 14 Feb 2023 17:13:28 +0000 Message-Id: <20230214171330.2722188-16-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Stat-Signature: jz99k6q43gt3m7i83m1qxmdjw1aw3jaz X-Rspam-User: X-Rspamd-Queue-Id: B8ABA2000F X-Rspamd-Server: rspam06 X-HE-Tag: 1676394874-220190 X-HE-Meta: U2FsdGVkX19VnjuVgVdSL2EcY+P2JRVgd8T0zLalEttp8o5poBB7UYMamYLAIkVbDvlj1Y6c25m3EZBbfeeQ9BSDttVQ2vhFEjjJMDgv+PpTWGSuT1LOmmwCnDsUO6e3c5ViRw3IxV7S/wL0ENB5g0bkssI4ATPqiOK1Y9dzXPwvnW+rZeMg3Qjf7RZCbl29p2GLP45pb5T293o2MxJARNLArijDXTpoL/j9lb7lpdry2Ml4LYK8ngBOdzQijO+G9KZkt1PBCkfDohLNfyW5AKVAvM+Zu2hcA1Hd64c/5biGigLsPk/pWT5YvtROMrOYLs23VoDqV4wU+gdgaGVjcP7jYP1Al0xtsnhoeFKM/+mqRH8zG3aC4lfQDI+yEFHcDizYx4C4dJ1bAGiGtL9c1vz2X6X1l5jHQc3qQ/ZUInEpuQ2eTd4MqzGoqVoddw16hBMu5VKy522pVBVLHF4Oe/G5IyL96mmmIeCMV3HUWcjapou9Tkdke7t/RO07cLVT0k3JDUQ0WopUT7htYAVrmW2XIUX6ZuNnUIIjn4FHNqfDKJwqBPg6JFDjBJeZwPVr6grw1ehOg2uxAq+U1H7QyIklsNO+0PpA1JMSrCiHAIXy4sfs1Uvp3cHy1Z8Wm8RDE+qPlIN2TKvZhF2wfecalFiFtLtrKX08J5u5Kd2MZIJyOyZ6qD9WpTHMNmkCl4rtmLP/lMaAYS4asF5Uv6BOo7IfHhzOHC1nWTt4V+3SHIh9XRhmEVGdA1kSbQk8+TtYvBb1R/dFMLHcxgenNi0e1ou//pTIs1fVY9ZscMJoNXMzNuuS7oUKfk0Cx7oqDwnsn0haJcbWn4qWZXISNBckdqvK3AeRIPTNgTqiGGL3NEKyLdiNYgX258nJMSJX9K+k+L8Fytb9n6k0KaqfS9+ImWFBu2Ybu/cPQgirqxo21R1M0B/Yi5btaGx9NhDewfG97QlYDcntvTn76LeRECw eTT141e1 E2ONQuha/q3geUcJOgo+pHHAO20syEekx3eFxRX2Kbt2/7At7FJxMC1vIoQmpqPQrkghSvopJwjoDereQ5UJbU3dvujXYIYyx+jPD6zn5e03P+Uv6g2QdzbVEED5/rMmMV7zDuNkZOD+jn71rQQ/O4Jc8nmqY2jwwKNpvuJS/rN+g19D70cQww6Uq3c0wJzWk8+fyWOPfmH1GUzLfZWRaxxkTaFgQMOFS1SsrEp4fZFKAPAWibVPGiqylOr8SlvmW9W+/Ift6AejmUdHcZi/W7chdpLDOmIgQMYhsRwut5YJA51t/C9DVlw07UXnJubFAspFz3NYgcKQ9cwzE/S7XCZ87f+4YTbYZGV/k6Jbh1AxxkDxqeBFQIWFZgOot4XXXXLSdAGxi6PuqxdhxdVGm3hSf52trOKnYo5NK73wgShvyJpU2zLd1YMn4BbK2hUcSqQ8ECrgF/w939/SkTXajyngS5wjOjfoLSEPeU83DYOXCtliROyJwmTnnbw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add BIO_PAGE_PINNED to indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #9) - Only consider pinning in bio_set_cleanup_mode(). Ref'ing pages in struct bio is going away. - page_put_unpin() is removed; call unpin_user_page() and put_page() directly. - Use bio_release_page() in __bio_release_pages(). - BIO_PAGE_PINNED and BIO_PAGE_REFFED can't both be set, so use if-else when testing both of them. ver #8) - Move the infrastructure to clean up pinned pages to this patch [hch]. - Put BIO_PAGE_PINNED before BIO_PAGE_REFFED as the latter should probably be removed at some point. FOLL_PIN can then be renumbered first. block/bio.c | 6 +++--- block/blk.h | 12 ++++++++++++ include/linux/bio.h | 3 ++- include/linux/blk_types.h | 1 + 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index bf9bf53232be..547e38883934 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1176,7 +1176,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + bio_release_page(bio, bvec->bv_page); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1496,8 +1496,8 @@ void bio_set_pages_dirty(struct bio *bio) * the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from - * here on. It will run one put_page() against each page and will run one - * bio_put() against the BIO. + * here on. It will unpin each page and will run one bio_put() against the + * BIO. */ static void bio_dirty_fn(struct work_struct *work); diff --git a/block/blk.h b/block/blk.h index 4c3b3325219a..f02381405311 100644 --- a/block/blk.h +++ b/block/blk.h @@ -425,6 +425,18 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* + * Clean up a page appropriately, where the page may be pinned, may have a + * ref taken on it or neither. + */ +static inline void bio_release_page(struct bio *bio, struct page *page) +{ + if (bio_flagged(bio, BIO_PAGE_PINNED)) + unpin_user_page(page); + else if (bio_flagged(bio, BIO_PAGE_REFFED)) + put_page(page); +} + struct request_queue *blk_alloc_queue(int node_id); int disk_scan_partitions(struct gendisk *disk, fmode_t mode, void *owner); diff --git a/include/linux/bio.h b/include/linux/bio.h index 805957c99147..b2c09997d79c 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,8 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (bio_flagged(bio, BIO_PAGE_REFFED)) + if (bio_flagged(bio, BIO_PAGE_REFFED) || + bio_flagged(bio, BIO_PAGE_PINNED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 7daa261f4f98..a0e339ff3d09 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,6 +318,7 @@ struct bio { * bio flags */ enum { + BIO_PAGE_PINNED, /* Unpin pages in bio_release_pages() */ BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ From patchwork Tue Feb 14 17:13:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31FF5C05027 for ; Tue, 14 Feb 2023 17:14:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C5016B0089; Tue, 14 Feb 2023 12:14:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B0176B008C; Tue, 14 Feb 2023 12:14:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E0056B008A; Tue, 14 Feb 2023 12:14:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 08C206B0089 for ; Tue, 14 Feb 2023 12:14:37 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B7DF2120F6C for ; Tue, 14 Feb 2023 17:14:36 +0000 (UTC) X-FDA: 80466546552.02.A7ED240 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id EC79780024 for ; Tue, 14 Feb 2023 17:14:34 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ImBqUJuk; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394875; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YX3eOMnEnHmNO2fsUm7QGJ9rpDZ86GbDPShQEUjns7k=; b=c1r+rV0t69ItYUpsfDXw7qeyJioYu8QzP0wHUDBId+ifO62LX9t4kWkUWqjsoRY4Zbm6Wn fecxW6ZJ9r7OtHPCmij/i64ogwCCAHsc6UBBdErsVJ5R2fob27eJrvm3aJJFevJaumxYAj aOeV5aDQMC9uR3BCnIa5FdqlxVC3YLM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ImBqUJuk; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394875; a=rsa-sha256; cv=none; b=byBiGkwG+ZVpUwctabweAgRhJF619FFEixPTblXBil1csXfGzTwRsXq1buX50Pj6iE6OD1 uzcBOr9NKYqc8Dw+oXNcRLITLKvAj4NgSZCMsCDDVG2iJLZkv1C03E3MXJjLeJTSuK4MbA ZyP69nNqxgUqH2GG8DuvxnWnhbMKElo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YX3eOMnEnHmNO2fsUm7QGJ9rpDZ86GbDPShQEUjns7k=; b=ImBqUJukWeThHSQmxMHgS4sSGeUN5KnH/CigC1J4nO67/OURq3gUODU0B3imlFS9JI4H2O XL1yA0RK8Stu1PbfK0OtTaHv5dj3wTs+IbgfMS5Pfb1MKm7ftPREi7H3ahd11OjRnT0IMu JBiGl8HUsRU7PwR8YWZ9ozpXLomfoRg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-52-XUarQZNaNTu_e2ZfEJgYPg-1; Tue, 14 Feb 2023 12:14:31 -0500 X-MC-Unique: XUarQZNaNTu_e2ZfEJgYPg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3BA24100F90C; Tue, 14 Feb 2023 17:14:30 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14FD41121318; Tue, 14 Feb 2023 17:14:27 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 16/17] block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages Date: Tue, 14 Feb 2023 17:13:29 +0000 Message-Id: <20230214171330.2722188-17-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EC79780024 X-Stat-Signature: 17bt7k5fd8mu7cejckku97zkiw9d8fow X-Rspam-User: X-HE-Tag: 1676394874-665641 X-HE-Meta: U2FsdGVkX18lSi/wUhdQu+yAxCz0XLqrAx14JRtTqqylNrPsGGw6ZASWehw6Dz8yPiE5aHSLqht3fgIGIRPVfrtutBmK9UDsuhyVmuqfuLabSGP7Qj2eCj/edaV9D1IIDgcjEQnu17d+KCfyMrehedcigw7XKLmCxUfvFhl3jbcgRiQotx6aCNgZvboGrd/VbG48NHbvnfH5HT7Dog57eU8+whgf87apKMUCzCZND1TD2sL8PIAF2UYyUhVgXTsNOwekuEyIvh5/rAyRD9Dqrv1hXWLcNQYHxboNhhS1iLEyXRk7fchkpstaQD2to8nOP+HqJyWhHG2IUHkaX9+ttpNsUEpBXYFyJynHBAyux5yfOenqkL+3NGHIrvoT0i6jojykNPoOiRUuNxmnl13wHxru8dOFwF8Sp58rEwc0nv1i7UgmQb5OpjMCbGvFY3QwYvwSZC6kUiPXuCxivaRNxuD7pRKyuAmgVBzxeDf0PloIPevXv/4e/+GhhHVrDb+u3Yd+JoRHMYN7MCOQ1aAKMLwiF+tyOSWo3LwlF7259LiuCpdgcAAt89LBOze1PXKQ03iPFrZbxjeOL8GxjAXCPfiDxSriYSShaW6P/VWT4xaIn+7bJFveGvFoRj9zF9WEnucm+wSzqhOL6CqxN4Tw4Y46IWJ3+Lj0QYE5gImHjn51l32WQZNbj3BY+uQ3cbIx5XduHdHjKGN0+pAFt7DoCuOmvjUobvla2bWenxY48z/G8mQ6kT8EYNwHfkAbZ5aqUhErKDGmQ8I6VDB9F/GvuNIKG64Ck8QY7cwqsGZ7YR/JBRsoot1B0w0CgL53/xxHgy+dPI5yu+46t2fuHUut0rCNFPQLyeddhv9hzqk6ypPVcZXA9IHQhT0tdT0GSiL0y/HarlgDOJH+bFbdChNCC0A42SCnfpsmiGxNxwTdOiZaWxmMbtxwQ+oJomS6j2sjut0PkEjb7uDvY81freM cyHY+FxJ /fhS8DdFKitnwC1gNLYEXsFrmz/g1i+zI8U0W5fIZ2lF7G1JhB6SS36PWnNu0cuTA3sy89hW6HZWdMSou+sVYqA+IG6Fegx2vZoTAozjDTEfhLJC1agbSMF5/SGRZJ3blQsjN/sFP1okpepM2rHpxGfkFFRVZH2iAI5VezO1qNu2Wske4lT5ZRN5hJePE6ofT0MZy/85TwrOVcK9ldEExxTcWWiYz+Soqe3FANQhZ8kZkfGzvDbtKSTrQbFgEXAOwuctErFUhV2r7En3M10biwAnRuhudGzuVBFEW3F0eDLnpO+E2LxAhx+L6Vd988Yzhj6E0umcKNpi9UDkGx9MDM1Uzt5AlHw0AVe/l7Q+C/udBRH7wYuJxvrSvmO6UXymW0aW+GBKe6OmS+y2g+eArzknRGMaKaVQhljaePqfTJYo7RqX9Xd6q1uHMB2n0hxSlo/aM4XPwkenGzUcJ7FyAEk9od0ia4XwXpwOz8WUZSE3ZUxrdtcSuiQ3ppw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being affected by/visible to the child process). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/bio.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/block/bio.c b/block/bio.c index 547e38883934..fc57f0aa098e 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1212,7 +1212,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, } if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1226,7 +1226,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len) return -EINVAL; if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1237,10 +1237,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, * @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. - * For multi-segment *iter, this function only adds pages from the - * next non-empty segment of the iov iterator. + * Extracts pages from *iter and appends them to @bio's bvec array. The pages + * will have to be cleaned up in the way indicated by the BIO_PAGE_PINNED flag. + * For a multi-segment *iter, this function only adds pages from the next + * non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1272,9 +1272,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages(iter, pages, - UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extraction_flags); + size = iov_iter_extract_pages(iter, &pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extraction_flags, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1307,7 +1307,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) iov_iter_revert(iter, left); out: while (i < nr_pages) - put_page(pages[i++]); + bio_release_page(bio, pages[i++]); return ret; } @@ -1342,7 +1342,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } - bio_set_flag(bio, BIO_PAGE_REFFED); + if (iov_iter_extract_will_pin(iter)) + bio_set_flag(bio, BIO_PAGE_PINNED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); From patchwork Tue Feb 14 17:13:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13140641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C07BCC64ED6 for ; Tue, 14 Feb 2023 17:14:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D91C6B007B; Tue, 14 Feb 2023 12:14:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 562986B008A; Tue, 14 Feb 2023 12:14:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DCFA6B008C; Tue, 14 Feb 2023 12:14:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 252A86B007B for ; Tue, 14 Feb 2023 12:14:43 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DD025160F18 for ; Tue, 14 Feb 2023 17:14:42 +0000 (UTC) X-FDA: 80466546804.26.94CB1AD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id A58EF4001C for ; Tue, 14 Feb 2023 17:14:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="S/cSpxgH"; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676394880; a=rsa-sha256; cv=none; b=sDbdYSHhUqu1H5q3aUgkqztSy5iKC94Ls7dheKeGlR4bgxc35/0vKLgQbocPcL4Z+xvSvN sVxCfPEfa/okA2wLvBFFmKi1Zwk4NL4JHTPtv8nS5xROL89yH4x1KHY6rLr3Wye7p4kxzx 7j7zi4g5rCBvSttxKAOHw+vGBVs4ODQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="S/cSpxgH"; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676394880; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7RYncx3d2QCFno76YxQ26A3gARIW2WsBk/g5n5z32ew=; b=Z/55hJ3Rfe1jYe1A7jU/s8o7SvsU0p/h8jUALxoy8vmTzU/q9M5m9NYELqbJQN4KwZF1mK QDIck6j7EEm6z/kR24SI72Yn4Tj5yVveRWH1wlyPwxsAXKJdyyP38UiYUVzpFy1qQyfjAU OaDTEiQRzxEWKc8QJeuPT9Nl4G6P2N8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676394880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7RYncx3d2QCFno76YxQ26A3gARIW2WsBk/g5n5z32ew=; b=S/cSpxgHjfB4rctxmfGIDb8xvgyfbwaIdZvnPCq4zDHwdA1f9EB0/V/zkHUUlVSkQUVVeN NoBoH/HlXzdBl1obC0PMuTRnve4jrijTLsMFvI/d66WCVHZzVlGkw5pr3oS0t+pdpp0B8A ckHlebraSyLlfMQV+lfSyYXIzMnE4Gw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-528-53yWEtX-Pm2chQYNZopvZA-1; Tue, 14 Feb 2023 12:14:33 -0500 X-MC-Unique: 53yWEtX-Pm2chQYNZopvZA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C77ED38123AE; Tue, 14 Feb 2023 17:14:32 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id D47AE18EC2; Tue, 14 Feb 2023 17:14:30 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v14 17/17] block: convert bio_map_user_iov to use iov_iter_extract_pages Date: Tue, 14 Feb 2023 17:13:30 +0000 Message-Id: <20230214171330.2722188-18-dhowells@redhat.com> In-Reply-To: <20230214171330.2722188-1-dhowells@redhat.com> References: <20230214171330.2722188-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Queue-Id: A58EF4001C X-Rspamd-Server: rspam01 X-Stat-Signature: wxjdbe6qbgdzre91ryxiuief45dwbwk3 X-HE-Tag: 1676394880-75606 X-HE-Meta: U2FsdGVkX189iSSdoLVn7b4hiErklYCRZ/Psh4xz+HKThieLWoqT+wXgwllgZtZ65vHkMEDy/hrN2keXcSnTcoct+9EipMhEmrwvfz8tgNYXA/hFrEEQxMKCDzwySHqPbjbRvAHHJoE9GRSz69pOMxcQ5G0d+MkdUAymszzbUXxiurE0SlK1+lXPcouCIpsO8sP/vUwdZ4xNUsUqd3vlyDyS1d7C2IDsd6GyYl4nIiObzRStM6pvZxtsrbh+uiIFNIwG8ffhmSy308oO11L6zqrdPxnowFXywkOVDKcA7pXSfvqzxU48V0ztjE5VxB+5pzrfbFQ9e0aLug7y9KCakzv6uL2t6Uh+N/0o1G5h4B10dPoODlziCzqRmNDd9s6IW2+grMSY6ceInkOqiL2FF7qgqGolN+fjZFRNt74RUdycRFUfyfL7RRetamU3voyxumTkD8Zsman15g6v2Kz65hTkHFxqQiV9KPsDg6A7iuiCDwuKOKGTLbHDo6qp+3I+A72Sf0D1lyGdNSDXDD5sSgyb0knWgsMZk+ACtCaFFEfcpp6swA/3aeWgOf0mfFcSCY+zR5GGo74Gga9gieBJjAxvz7E6s/PsmREReiNiQCg7fCyJxo/5pTl8qNuWzsWfFcu+x+C6PX6XgcM7CbyGeXe/eyC8o/WNyYpUBELRpkiSK/PlmRkyMVFcAK5lpkjDAmvSVpSzqvOnydcKK7NUsc/3ixYrbl0knlkMiH7LMpeJsnYk3Yaa8EdZ/tpm8jIUOolUE2ptCo8DguuB4Q+ihYilkqOA3N4YRFAs+Ml6f/dAstn5RdCVMswvIF1FBoDfNhs//TGSVf0IWQjx7PM1QUcydhoXW4uheXyO1KFY6QP5Git6xUtJpGZygbTLs8r+/ZH6XtSuFPtoi+7JMng/V1V0clZ6ZehP3qA/wyZwZtncJu3PdG/SrZ604lJTUoY6cwGq3wh5/UzI2yXS8jX BcFLwy5z IqG81anQtsZo9i1epRcSVUBbrY0I/JHrt1vSMJVVsRj/WE3LRV6/R11rs14ShJKrKWd4sfeN6JDr21yf7R+T7I9z+28Sp8sFQbJQmYO+bY2EV9/ISBAA+RpbL6YhGn3KbYN6dvfdw+k/hEL37XZZwVumtN8PeduUKCNHTPzqs0uURSd/cD8c66YnDe5qkABxwGsuj2q80WHJZhTeaZcsBUFOtG5WCfj7SgbV7SseJZZHr+CqFIkjy7eutxUhSVAXRjdAq3HBjAXUMDvq9u9qIq8b2l4BxsGMpn2NELAGgqxj1wTtyNMLUa2HXUWJzCO6ru0yQZeoV9rSIl4LIOTf1uuD/t4p87y57zTElkIZu0wtaIAHMu6vC9BFDt9vmenz44dskpfluTJvTIRHEHcXfYiXsGokD5Du3aH7zu5PckxeXnmC4RIn1IBuKyyUB05l7MQkRoOWwtx75Q8D95NZiXVlb/2d68qeNotk1a3l1uyl3/DdyoBoEhJodbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being visible to/affected by the child process). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/blk-map.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index f1f70b50388d..0f1593e144da 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -281,22 +281,21 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extraction_flags |= ITER_ALLOW_P2PDMA; + if (iov_iter_extract_will_pin(iter)) + bio_set_flag(bio, BIO_PAGE_PINNED); - bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { - struct page **pages, *stack_pages[UIO_FASTIOV]; + struct page *stack_pages[UIO_FASTIOV]; + struct page **pages = stack_pages; ssize_t bytes; size_t offs; int npages; - if (nr_vecs <= ARRAY_SIZE(stack_pages)) { - pages = stack_pages; - bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, extraction_flags); - } else { - bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, extraction_flags); - } + if (nr_vecs > ARRAY_SIZE(stack_pages)) + pages = NULL; + + bytes = iov_iter_extract_pages(iter, &pages, LONG_MAX, + nr_vecs, extraction_flags, &offs); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -318,7 +317,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (!bio_add_hw_page(rq->q, bio, page, n, offs, max_sectors, &same_page)) { if (same_page) - put_page(page); + bio_release_page(bio, page); break; } @@ -330,7 +329,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, * release the pages we didn't map into the bio, if any */ while (j < npages) - put_page(pages[j++]); + bio_release_page(bio, pages[j++]); if (pages != stack_pages) kvfree(pages); /* couldn't stuff something into bio? */