From patchwork Wed Jan 25 21:06:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27543C54E94 for ; Wed, 25 Jan 2023 21:07:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B08226B0075; Wed, 25 Jan 2023 16:07:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB8C56B0078; Wed, 25 Jan 2023 16:07:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 932756B007B; Wed, 25 Jan 2023 16:07:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 836BD6B0075 for ; Wed, 25 Jan 2023 16:07:13 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 62573140EC9 for ; Wed, 25 Jan 2023 21:07:13 +0000 (UTC) X-FDA: 80394556746.19.206F422 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id AF9D4C0007 for ; Wed, 25 Jan 2023 21:07:10 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nn3bnUqH; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kMCsX6uIACqQJdOvi49zJL2jZZzhVnqUm9NN1YXqPiU=; b=tNUZqYuremA8n37Tg/p9mZMdcu6dnZxGOtpnDVvNugJhRuLopHae/Hz/iUUcMzqanv7Bp0 PjOdgNLMU9XFGrNnzGI9gFzYB4FjqI0Jy7oZ+GrOmI29uSMqenI49zDzGD+6ghTVfVNQJX cyrWTKMkZKApeJG7z9XGEtkdBBYxJS8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Nn3bnUqH; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680830; a=rsa-sha256; cv=none; b=zKJMG4IACz5jKnwneN0e8IQUgyQoT4zyZ50GLBbG7ZrH4wzd1l1ZPVPkUxtZp5GFX8bFOB n+NVH9HLJj7lVeQ6xQXJ7Tp/u97GCdUluYIGX3nYRW64krOcUR/mIRJhQBS0wklYdr+ZEP iPBRtlLUTBw4RpR+HMp8ZWij4TWorxE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kMCsX6uIACqQJdOvi49zJL2jZZzhVnqUm9NN1YXqPiU=; b=Nn3bnUqHspyG1yvDQB7JRObXTAKKpZXs8yFqfYE80S312p2P6Yh2jJS7qsng2yFFAxyYnn +3VGOrGXvDqH3AzobH50f68L8cOu2svXaBHFCOstNQZXAR//MFRrhqhGukkq1LosgqF5Vo y4g+n7TOdqNezI+a3NGHslMHgjPgJCI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-370-GKZzfjEHMeirJsV37BahGA-1; Wed, 25 Jan 2023 16:07:06 -0500 X-MC-Unique: GKZzfjEHMeirJsV37BahGA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 069E2857F41; Wed, 25 Jan 2023 21:07:06 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3C1F492B01; Wed, 25 Jan 2023 21:07:02 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 1/8] iov_iter: Define flags to qualify page extraction. Date: Wed, 25 Jan 2023 21:06:50 +0000 Message-Id: <20230125210657.2335748-2-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: owa1hqifs364zsosryrot64u17da6mfe X-Rspamd-Queue-Id: AF9D4C0007 X-HE-Tag: 1674680830-753456 X-HE-Meta: U2FsdGVkX1+QdGCgh56AaLenjbla1R8eNIKEp09loqYLDhVNGnqBS0oI0U5o27MpobiusZs35IvUCuo7M6xzc1SSyGZ+IBQkSjkVKEyur8a4QX84j9r2g1uxvI6xxIGXCz33kLlxGfeyAjmfSniOSBWTM/uaiqFxoFos6XbJfZRAGz+x8U0T8gyl065ylj4O0cMLd0vhWtL5pqHqXvEC6huta/50XIEs4TqxnPCZM+QVPx5lesVZFqwoXx0Gc9T7m6YbsW2JtDyao6PjXDGqsJIpF5scU4ouVGA0QxtN6e1/o9vLaYwQj9NNLryDgTSxJveQWAifzmsDM2/g4YOxOrTAP6tHtivisXmwJnZ2t8ih/S+9TwNgyW4EJWXxaDAhP2LtD9FcvrLJS4846qs0OW8EanNVVBdConbozEkSs8z9/mmSTEvMASLDmgUhYOgUpdTeje94GKeHj2pq9jF5GFMxNSo6q81y+lFmb8fwVFzY2yfNh5Q7kkWEcT+kcdKs5QcW/j/l+jfq1I1tK7hv1fQKpmlxIqqryCDBa8zsn3drwIR+xjxoMSEgxVba2o1DLHaR/Y2BaKryb8i6cpqAVfH0qm/SXwcMhAgVb496XxXb9pGPA6OeQgwY78HyUQ2pY8uILo8KyuAjLAZg53HI65osNQASqgblC3NOIRLEAXJT5iTP9E5JKpVhbY9bGtg7quK8SUNJmDuQpv9cNPRrHVpMggxhcGj5XEsUNYXjhAeLKlawXGUczozTM3X2hrGpPuQ34qRLzb2jROzNolwP6zugGOPZNLwX5P2tu0xTXZtE0/gzg01Q23dkG5tHFOLgGhNZCJXi1scEMyC8UTuyhfOVL6fXoLl0hXIoCkjTSRBLWd2xYdpGutj9ISePzV2RnWkKNDyJmb4jwk8tdGyu82qY0WWb8R9zMoXhfxj2WFe+oaPbesZi0hGVp92SJAFY6NteSLHVNlxm4Ztw05U tW838L3b DtLJr4ni68qDeexNyQj4x8OcpkJyXRmhow+buWRSf3Df5JWPJvwUjcWSUQp6LHX5Rur93gd2UESSxrs80tJYj4U8sfKl2jp+saLNNsdFzRgHQmm8FztwmDzhe0iAr7HbAWNaOhraCv/g4uNAyykzzT99MDTO6kb2QwRMSGtWIMS7S5Pge6jOurFB2wdJjIk2oolwRqm8+jOAgGG1iXZ6dX7mrxxq3oPjYHj8OlIZ/MqMyBQXI6A+QAPdShFXHkdsStDsc0lKuB1TcbbHgeiCvnFd2EWIwcDPDWoUnjBrr9/XLeGUGKLBMmaS573ZUGQ5lJGJNaHBsH4Sua81fXiDzrEk/bRlcYYBOKJjTN7EBMXnREqAkHkIt33mHfVY6FYrdJgEkL66O9zj+KPbHt5fadZP8NYOVXJL+rHGpPjAozuMAGrUAs9qZFOL31oNqYDA++6r+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags. For now only a flag to allow peer-to-peer DMA is supported. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Logan Gunthorpe cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org --- Notes: ver #9) - Change extract_flags to extraction_flags. ver #7) - Don't use FOLL_* as a parameter, but rather define constants specifically to use with iov_iter_*_pages*(). - Drop the I/O direction constants for now. block/bio.c | 6 +++--- block/blk-map.c | 8 ++++---- include/linux/uio.h | 7 +++++-- lib/iov_iter.c | 14 ++++++++------ 4 files changed, 20 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index ab59a491a883..683444e6b711 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1249,7 +1249,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; - unsigned int gup_flags = 0; + unsigned int extraction_flags = 0; ssize_t size, left; unsigned len, i = 0; size_t offset, trim; @@ -1264,7 +1264,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) - gup_flags |= FOLL_PCI_P2PDMA; + extraction_flags |= ITER_ALLOW_P2PDMA; /* * Each segment in the iov is required to be a block size multiple. @@ -1275,7 +1275,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) */ size = iov_iter_get_pages(iter, pages, UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + nr_pages, &offset, extraction_flags); if (unlikely(size <= 0)) return size ? size : -EFAULT; diff --git a/block/blk-map.c b/block/blk-map.c index 19940c978c73..7db52ad5b2d0 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -267,7 +267,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, { unsigned int max_sectors = queue_max_hw_sectors(rq->q); unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); - unsigned int gup_flags = 0; + unsigned int extraction_flags = 0; struct bio *bio; int ret; int j; @@ -280,7 +280,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, return -ENOMEM; if (blk_queue_pci_p2pdma(rq->q)) - gup_flags |= FOLL_PCI_P2PDMA; + extraction_flags |= ITER_ALLOW_P2PDMA; while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; @@ -291,10 +291,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (nr_vecs <= ARRAY_SIZE(stack_pages)) { pages = stack_pages; bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, gup_flags); + nr_vecs, &offs, extraction_flags); } else { bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, gup_flags); + LONG_MAX, &offs, extraction_flags); } if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; diff --git a/include/linux/uio.h b/include/linux/uio.h index 9f158238edba..58fda77f6847 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -252,12 +252,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count); ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start, - unsigned gup_flags); + unsigned extraction_flags); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start, - unsigned gup_flags); + unsigned extraction_flags); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); @@ -360,4 +360,7 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +#define ITER_ALLOW_P2PDMA 0x01 /* Allow P2PDMA on the extracted pages */ + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index f9a3ff37ecd1..da7db39075c6 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1432,9 +1432,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i, static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start, - unsigned int gup_flags) + unsigned int extraction_flags) { - unsigned int n; + unsigned int n, gup_flags = 0; if (maxsize > i->count) maxsize = i->count; @@ -1442,6 +1442,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return 0; if (maxsize > MAX_RW_COUNT) maxsize = MAX_RW_COUNT; + if (extraction_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; if (likely(user_backed_iter(i))) { unsigned long addr; @@ -1495,14 +1497,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, - size_t *start, unsigned gup_flags) + size_t *start, unsigned extraction_flags) { if (!maxpages) return 0; BUG_ON(!pages); return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, - start, gup_flags); + start, extraction_flags); } EXPORT_SYMBOL_GPL(iov_iter_get_pages); @@ -1515,14 +1517,14 @@ EXPORT_SYMBOL(iov_iter_get_pages2); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, - size_t *start, unsigned gup_flags) + size_t *start, unsigned extraction_flags) { ssize_t len; *pages = NULL; len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, - gup_flags); + extraction_flags); if (len <= 0) { kvfree(*pages); *pages = NULL; From patchwork Wed Jan 25 21:06:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0E02C27C76 for ; Wed, 25 Jan 2023 21:07:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BEEE6B007B; Wed, 25 Jan 2023 16:07:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 86F226B007D; Wed, 25 Jan 2023 16:07:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 625356B007E; Wed, 25 Jan 2023 16:07:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 516FA6B007B for ; Wed, 25 Jan 2023 16:07:20 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 132D4120E6D for ; Wed, 25 Jan 2023 21:07:20 +0000 (UTC) X-FDA: 80394557040.30.7701910 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 457B840003 for ; Wed, 25 Jan 2023 21:07:18 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=D2A1rDSI; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680838; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UepUyW3/cmlMjdWLU4KtSRnF5dNIEk+IVroeIJZYwbo=; b=DrZwrYay4RBYxPLFh4MCMyFXW32MpKv231GlpNhj8j0L99nko8TnCBfCSxbUo4XyjhO5YL 2bitGQ44tUaljebwCCvOmVDArwBzQrJ7P7JVN91gXbxCNI/uBTljU5bc+NsWBRW6wO3YPp 9LBJG2befBfIKdJu+1p7iEYPoAR3vtc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=D2A1rDSI; spf=pass (imf27.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680838; a=rsa-sha256; cv=none; b=WIGn+/XHA0VzAq3mfRRKR5QhoGcrpxORx5c9A0BYIBcKnufvy/oRUzeCLVvcD8AEpRbXiA 30Nvq9nF+N5w2kWXh49AYbCoojYswa8qKynP4aJc5mWc1v4cHBSnmDugchRY9c9vY9Rt6p EH9FbUqW5vvyNCysjItvBMg/JLxS4Zs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UepUyW3/cmlMjdWLU4KtSRnF5dNIEk+IVroeIJZYwbo=; b=D2A1rDSIUewIUo3tJgM3/Q0VAOF9wJqmDKsQ7VUWT6636b1/XE5kbj69hxg8jN4o3iFy6U ah2L5eeGr9LzT2Wuj3xcXB97iFlT5e5AsRJJEg6ZgR7Z5KpYBAeh25D/jb9firgRWImzt1 nGHfY80Z6NPQxnh1zIx5aYnxDCliTyY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-322-9AeSPl0iOImDajpr6Cq5Yw-1; Wed, 25 Jan 2023 16:07:14 -0500 X-MC-Unique: 9AeSPl0iOImDajpr6Cq5Yw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EEF2885CCE0; Wed, 25 Jan 2023 21:07:08 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id CA6FBC15BA0; Wed, 25 Jan 2023 21:07:06 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 2/8] iov_iter: Add a function to extract a page list from an iterator Date: Wed, 25 Jan 2023 21:06:51 +0000 Message-Id: <20230125210657.2335748-3-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 457B840003 X-Stat-Signature: snh7xqnrq7rstcasac3tcp5nrcm1wwig X-Rspam-User: X-HE-Tag: 1674680838-474613 X-HE-Meta: U2FsdGVkX1+AxtPezOJOTQDFuyT6daWVmmtzTGhB5ZoB82iG73a4Qr8aLhWOBhSXvwdKMpzVg2CSkOTB3zXED9qte8sIC2vqL1qMfcPZWWRxnQh4A4kYDOCWOmPJ8EMhV5+0hZcGkaK65ZLtOMs3ozJ3MKeQ/5gVJAIbDqp1CumTlBetyT8jGLG8k+okKkmrfpFBbN5dgrhDMBli/1YjuOfGD0YKQTb7D4gYNLj8BxALgC9i0XsDvLE3X257R+a5S61EZXED9EVrDvtEOqoxw+SLIfaHQPtTx1oMedLBUwoM4RgjcdNwDtjL2wxLQLAYyKoARsaUpoleUhE8YibMbjszuqHC0g1RJ/bvnvcFuIB5nXHYFV53LaF1CjMJsMcz24h/bXi+oJUHWo1J4PxJkm1ZMLQeoPwXjbeoS6NTKnNbP23iE/Iw/e7GEYLlrCXzDvfSWdwikRZGbtEbr4vSgKVmfi2mqVT1Lh/Kz2fBFM/cDh52HIZbTu2xfWMg+nigVovocLpWVkeeiu5WZvKZSNiP+wmwrvw76/0PaVzjvcxz0Tdi1BfW+5AZ3lSlveBb44v5M7jznYczt2VHEQXT0vEqIaaV5ekgZEQrFxOjoAux7p6dCKgUy49LK8sJ9vM1ekzNzATLEo0AmufSZwxF/sao0f4SUpUSUISw7HBIpHxrPYbt/MGN50/ISdC1tpXIIZYYAaSzncxBV7YREbzt3nm93pwagfFydFcwhi3dmZh3uEWWpvcPT88vUpY6KkpTkLclg3K6VEXCJjY/mYnbb6x+kTzo/++GNd5AZ28bQtbeFFMhr7gu1NivVpMQxwSHB0q+KvzTvotQJYAoXt1UwwTo3DznH+SMF8OfDt7JoAX/nWy7GIrAKbG5DzOoznwLm2OmNaryIRdYE6JoUR+5AqdLGWJjDK20J7YHSbHEpbB+FJ9IGOp7CPLxTHOU0MQSDm8mTg07peLI1fJdz6k 7FytYgkv zpyQ8rdtAuOqFHsv43R2g5wQUqkf6HdH+2n6coRk1W5EOiPt5Xrt8SSBI3qFptLu3nsmGvH/rJzkNtr1LQrpt9phOhXLAvGMcOwjdVjOGhBj/hA8PR69kduMAKPzX9FsaUCKO69UBmXnPGKzCnfFsugqz8iQtAsrnJXR0EoD1oodEN1t71WXcr1iw7XsWwROMq2tZ54ndhrPkELGVmGkpH/c1zuazI5OUwSbcEOQqPA5HLfCgxQ58gsGbxImZpIo4bF0xwv0rnUSLmp+ZxBmdSJJmhQZLwBt5W9Vec7ZLYzHEDmXlN1rm5hr5ItICdK8CpmObTsDoyi/FI23zkcFwqTx7leZP3iJpBUFoTktSELWm4r7DdRWJbkyfB73m1/ATRjVWPiH5UlahCJ9DIjpKY0p7rdl7WDetwq4Xd3H8vy8Mk4VZGa4Hlgq3d0zDiSu5UXfPI0oGzG+kN0z+1F+hxEHndK6iq4rIbM1ox7/djk891fXpSROLjx6dRIsHVmEiiMMcAYmXjlIFnruq1fNd3/uS46FeYi925xfp649IBvhMX2zH1AKIOFUbug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a function, iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a pin added or nothing, depending on the type of iterator. Add a second function, iov_iter_extract_will_pin(), to determine how the cleanup should be done. There are two cases: (1) ITER_IOVEC or ITER_UBUF iterator. Extracted pages will have pins (FOLL_PIN) obtained on them so that a concurrent fork() will forcibly copy the page so that DMA is done to/from the parent's buffer and is unavailable to/unaffected by the child process. iov_iter_extract_will_pin() will return true for this case. The caller should use something like unpin_user_page() to dispose of the page. (2) Any other sort of iterator. No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention. iov_iter_extract_will_pin() will return false. The pages don't need additional disposal. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- Notes: ver #10) - Fix use of i->kvec in iov_iter_extract_bvec_pages() to be i->bvec. ver #9) - Rename iov_iter_extract_mode() to iov_iter_extract_will_pin() and make it return true/false not FOLL_PIN/0 as FOLL_PIN is going to be made private to mm/. - Change extract_flags to extraction_flags. ver #8) - It seems that all DIO is supposed to be done under FOLL_PIN now, and not FOLL_GET, so switch to only using pin_user_pages() for user-backed iters. - Wrap an argument in brackets in the iov_iter_extract_mode() macro. - Drop the extract_flags argument to iov_iter_extract_mode() for now [hch]. ver #7) - Switch to passing in iter-specific flags rather than FOLL_* flags. - Drop the direction flags for now. - Use ITER_ALLOW_P2PDMA to request FOLL_PCI_P2PDMA. - Disallow use of ITER_ALLOW_P2PDMA with non-user-backed iter. - Add support for extraction from KVEC-type iters. - Use iov_iter_advance() rather than open-coding it. - Make BVEC- and KVEC-type skip over initial empty vectors. ver #6) - Add back the function to indicate the cleanup mode. - Drop the cleanup_mode return arg to iov_iter_extract_pages(). - Pass FOLL_SOURCE/DEST_BUF in gup_flags. Check this against the iter data_source. ver #4) - Use ITER_SOURCE/DEST instead of WRITE/READ. - Allow additional FOLL_* flags, such as FOLL_PCI_P2PDMA to be passed in. ver #3) - Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1]. include/linux/uio.h | 25 ++++ lib/iov_iter.c | 321 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 346 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 58fda77f6847..47ebb59a0202 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -363,4 +363,29 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, /* Flags for iov_iter_get/extract_pages*() */ #define ITER_ALLOW_P2PDMA 0x01 /* Allow P2PDMA on the extracted pages */ +ssize_t iov_iter_extract_pages(struct iov_iter *i, struct page ***pages, + size_t maxsize, unsigned int maxpages, + unsigned int extraction_flags, size_t *offset0); + +/** + * iov_iter_extract_will_pin - Indicate how pages from the iterator will be retained + * @iter: The iterator + * + * Examine the iterator and indicate by returning true or false as to how, if + * at all, pages extracted from the iterator will be retained by the extraction + * function. + * + * %true indicates that the pages will have a pin placed in them that the + * caller must unpin. This is must be done for DMA/async DIO to force fork() + * to forcibly copy a page for the child (the parent must retain the original + * page). + * + * %false indicates that no measures are taken and that it's up to the caller + * to retain the pages. + */ +static inline bool iov_iter_extract_will_pin(const struct iov_iter *iter) +{ + return user_backed_iter(iter); +} + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index da7db39075c6..7f597d530569 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1916,3 +1916,324 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) i->iov -= state->nr_segs - i->nr_segs; i->nr_segs = state->nr_segs; } + +/* + * Extract a list of contiguous pages from an ITER_PIPE iterator. This does + * not get references of its own on the pages, nor does it get a pin on them. + * If there's a partial page, it adds that first and will then allocate and add + * pages into the pipe to make up the buffer space to the amount required. + * + * The caller must hold the pipe locked and only transferring into a pipe is + * supported. + */ +static ssize_t iov_iter_extract_pipe_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extraction_flags, + size_t *offset0) +{ + unsigned int nr, offset, chunk, j; + struct page **p; + size_t left; + + if (!sanity(i)) + return -EFAULT; + + offset = pipe_npages(i, &nr); + if (!nr) + return -EFAULT; + *offset0 = offset; + + maxpages = min_t(size_t, nr, maxpages); + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + left = maxsize; + for (j = 0; j < maxpages; j++) { + struct page *page = append_pipe(i, left, &offset); + if (!page) + break; + chunk = min_t(size_t, left, PAGE_SIZE - offset); + left -= chunk; + *p++ = page; + } + if (!j) + return -EFAULT; + return maxsize - left; +} + +/* + * Extract a list of contiguous pages from an ITER_XARRAY iterator. This does not + * get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_xarray_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extraction_flags, + size_t *offset0) +{ + struct page *page, **p; + unsigned int nr = 0, offset; + loff_t pos = i->xarray_start + i->iov_offset; + pgoff_t index = pos >> PAGE_SHIFT; + XA_STATE(xas, i->xarray, index); + + offset = pos & ~PAGE_MASK; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + rcu_read_lock(); + for (page = xas_load(&xas); page; page = xas_next(&xas)) { + if (xas_retry(&xas, page)) + continue; + + /* Has the page moved or been split? */ + if (unlikely(page != xas_reload(&xas))) { + xas_reset(&xas); + continue; + } + + p[nr++] = find_subpage(page, xas.xa_index); + if (nr == maxpages) + break; + } + rcu_read_unlock(); + + maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from an ITER_BVEC iterator. This does + * not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extraction_flags, + size_t *offset0) +{ + struct page **p, *page; + size_t skip = i->iov_offset, offset; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->bvec->bv_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->bvec++; + skip = 0; + } + + skip += i->bvec->bv_offset; + page = i->bvec->bv_page + skip / PAGE_SIZE; + offset = skip % PAGE_SIZE; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + for (k = 0; k < maxpages; k++) + p[k] = page + k; + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of virtually contiguous pages from an ITER_KVEC iterator. + * This does not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extraction_flags, + size_t *offset0) +{ + struct page **p, *page; + const void *kaddr; + size_t skip = i->iov_offset, offset, len; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->kvec->iov_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->kvec++; + skip = 0; + } + + offset = skip % PAGE_SIZE; + *offset0 = offset; + kaddr = i->kvec->iov_base; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + kaddr -= offset; + len = offset + maxsize; + for (k = 0; k < maxpages; k++) { + size_t seg = min_t(size_t, len, PAGE_SIZE); + + if (is_vmalloc_or_module_addr(kaddr)) + page = vmalloc_to_page(kaddr); + else + page = virt_to_page(kaddr); + + p[k] = page; + len -= seg; + kaddr += PAGE_SIZE; + } + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from a user iterator and get a pin on + * each of them. This should only be used if the iterator is user-backed + * (IOBUF/UBUF). + * + * It does not get refs on the pages, but the pages must be unpinned by the + * caller once the transfer is complete. + * + * This is safe to be used where background IO/DMA *is* going to be modifying + * the buffer; using a pin rather than a ref makes forces fork() to give the + * child a copy of the page. + */ +static ssize_t iov_iter_extract_user_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + unsigned int extraction_flags, + size_t *offset0) +{ + unsigned long addr; + unsigned int gup_flags = FOLL_PIN; + size_t offset; + int res; + + if (i->data_source == ITER_DEST) + gup_flags |= FOLL_WRITE; + if (extraction_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; + if (i->nofault) + gup_flags |= FOLL_NOFAULT; + + addr = first_iovec_segment(i, &maxsize); + *offset0 = offset = addr % PAGE_SIZE; + addr &= PAGE_MASK; + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + res = pin_user_pages_fast(addr, maxpages, gup_flags, *pages); + if (unlikely(res <= 0)) + return res; + maxsize = min_t(size_t, maxsize, res * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/** + * iov_iter_extract_pages - Extract a list of contiguous pages from an iterator + * @i: The iterator to extract from + * @pages: Where to return the list of pages + * @maxsize: The maximum amount of iterator to extract + * @maxpages: The maximum size of the list of pages + * @extraction_flags: Flags to qualify request + * @offset0: Where to return the starting offset into (*@pages)[0] + * + * Extract a list of contiguous pages from the current point of the iterator, + * advancing the iterator. The maximum number of pages and the maximum amount + * of page contents can be set. + * + * If *@pages is NULL, a page list will be allocated to the required size and + * *@pages will be set to its base. If *@pages is not NULL, it will be assumed + * that the caller allocated a page list at least @maxpages in size and this + * will be filled in. + * + * @extraction_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA + * be allowed on the pages extracted. + * + * The iov_iter_extract_will_pin() function can be used to query how cleanup + * should be performed. + * + * Extra refs or pins on the pages may be obtained as follows: + * + * (*) If the iterator is user-backed (ITER_IOVEC/ITER_UBUF), pins will be + * added to the pages, but refs will not be taken. + * iov_iter_extract_will_pin() will return true. + * + * (*) If the iterator is ITER_PIPE, this must describe a destination for the + * data. Additional pages may be allocated and added to the pipe (which + * will hold the refs), but pins will not be obtained for the caller. The + * caller must hold the pipe lock. iov_iter_extract_will_pin() will + * return false. + * + * (*) If the iterator is ITER_KVEC, ITER_BVEC or ITER_XARRAY, the pages are + * merely listed; no extra refs or pins are obtained. + * iov_iter_extract_will_pin() will return 0. + * + * Note also: + * + * (*) Use with ITER_DISCARD is not supported as that has no content. + * + * On success, the function sets *@pages to the new pagelist, if allocated, and + * sets *offset0 to the offset into the first page. + * + * It may also return -ENOMEM and -EFAULT. + */ +ssize_t iov_iter_extract_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + unsigned int extraction_flags, + size_t *offset0) +{ + maxsize = min_t(size_t, min_t(size_t, maxsize, i->count), MAX_RW_COUNT); + if (!maxsize) + return 0; + + if (likely(user_backed_iter(i))) + return iov_iter_extract_user_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_kvec(i)) + return iov_iter_extract_kvec_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_bvec(i)) + return iov_iter_extract_bvec_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_pipe(i)) + return iov_iter_extract_pipe_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + if (iov_iter_is_xarray(i)) + return iov_iter_extract_xarray_pages(i, pages, maxsize, + maxpages, extraction_flags, + offset0); + return -EFAULT; +} +EXPORT_SYMBOL_GPL(iov_iter_extract_pages); From patchwork Wed Jan 25 21:06:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D762C54E94 for ; Wed, 25 Jan 2023 21:07:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABAA86B0078; Wed, 25 Jan 2023 16:07:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A43836B007B; Wed, 25 Jan 2023 16:07:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90B446B007D; Wed, 25 Jan 2023 16:07:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 80E466B0078 for ; Wed, 25 Jan 2023 16:07:18 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 68576140D91 for ; Wed, 25 Jan 2023 21:07:18 +0000 (UTC) X-FDA: 80394556956.19.37A033A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id A4B28140015 for ; Wed, 25 Jan 2023 21:07:16 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cMSKG0h9; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680836; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AA3SDxqAw6tV+jeEhwhhXCCPQx7iw6NiGu/cAPT0aQk=; b=2wg1A2Twpa2QFZOlKYbzKTdMhdEUOwzGJZJM83PXoeWE3ZxhWpm+tlE5KWwMK/HdcH/xAt AGjVC/VjoVD4LXaOv6FsVEVnvZQlCaGhyBOJ0Q5f0ikyjdhUygye1cfQb59X9H9ZigqTNR PnAz5zlbSIjfuYtDHdduRUyeidKUgqE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cMSKG0h9; spf=pass (imf09.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680836; a=rsa-sha256; cv=none; b=Z9ROwAFO42h7s+LM+xYh7ExfHzaLTYf7DRJsbYxXCXnzTmQH6NwYAFU9rRw0/ZSWFUkhR9 JKA9+lUC0RzuwDql+nhvUJdjdT/IHnNJHaDnhRwemJUKqGPC0oQnllagqdK/q0Q0RnaQj4 gir+STPHiu5VNQUEaLfZ3eKs4cy8aUY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AA3SDxqAw6tV+jeEhwhhXCCPQx7iw6NiGu/cAPT0aQk=; b=cMSKG0h9z3G3eDAxpAVzMWGmIW4MkKSVoc8vwjwEaWSfCJxzx5qS1jBabQlGSAtNaddpiN IXUveQPgToZeb94S3pFh7G//hDHil6dfITyu3LEzrB2GJXSFytPzVHZFyQpmNWqRyesZbt YtuAlxm7EWOM80RX+y1RAgJDCoNVFNQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-62-bzFi5apAOA-xyIVD8HJxAA-1; Wed, 25 Jan 2023 16:07:12 -0500 X-MC-Unique: bzFi5apAOA-xyIVD8HJxAA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9333B1871CDB; Wed, 25 Jan 2023 21:07:11 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id A052C42203; Wed, 25 Jan 2023 21:07:09 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Hubbard Subject: [PATCH v10 3/8] iomap: Don't get an reference on ZERO_PAGE for direct I/O block zeroing Date: Wed, 25 Jan 2023 21:06:52 +0000 Message-Id: <20230125210657.2335748-4-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A4B28140015 X-Stat-Signature: 8e71fk7dg86yi58ak4uzfrewzpf7b44p X-Rspam-User: X-HE-Tag: 1674680836-846829 X-HE-Meta: U2FsdGVkX1/Sta9l6Ntpo28hVP5AsWh7vVEpX7qoXOVAzpJbANNB/8w0uKR6SgVx1kXcT7JPGPDtz7qWLZvwNAUSy39fsdL9IA7g2weUcvG0xn7acxC1G5cDSFbUmZD6PqVF5FWzme8DX93Y2O+t+rlnDR4lkcM1cXbhlaDJcOm/j1I07tLkgv161Sr7BxpxOXaZQqCpSk+uQqRaSUJomJoPh9/yTtW1k5O/PyHowjDFrHFo069ZQ1FD90fgeheYf+AF1rhAfTNksIWYVJXn6wT8sSqzoWS2XNo4GagfOhNCIoWA22j55TVbjx1TX7fc1WWn4WTUKwV86fZSSHRG7we1N8mrFpJACgkl0cQfWSRK6UUvuunMf1xAeNobjUOTMUrdkqhf+9QYAXUGSZVCD2qCioXvP492jjreZoSAPDFIoFVAZlqfhTdxIue8DHyr1tXzxgHlZigR/ODLKNh9NrajKPld2iXGiEXoyGPLvEpCoDNVWqwMaU33ZHdl+FO696AsRTr9AAZT38NtVyH9WEWflsumGTB1aNRXWnlFou9Toqpmxq3PKcpgzv9ibkaEhHaq9ylblwkO9kdeACo/uR7ra8ScIFhYutAZ3xZGnw9FIlPPeyHeCyIEAea/jJykDN1X6D8S1fNse0QrlPEuI9dZu41JMS1MiXJSa6kl+8tYowjOZS2lGWW3S0vV7ZsAcnhnUL66cSGZeauQvVGfiSFT6Zu5gZJLUgjDMQOoM7ltvXgau0eZGmVR/WmlIaa25z3qP/k1BGp9s0Pw+61WynYc89I78NCxJekxJB7WUh5ZKQiA+oZKOyY0jWVyq390mSDxGowUsQhWmfGYHDDTTzWCd9geJvL/hYFNb8E9PAWX/5I3eGfCVDJXUE3QzqkuavWiJp/qgv1d55/SgnK5zBMY4tE+DZaifPBrhLvd7pr0sfMwFMeKriglGK59l72EESrmwoles2yTILAA+FR jlqmAvsY 5AsmDrl5PLaw5yYVc1Cgnv5He+xvsY9f/OVjV0bde6Xxd7my37G73Rt8bC5PzRWmajlMMR9R7hn1of2WDuIMM0otGvcnEH6JKjh6/6NQ0Fyya9e0n9izL75F/+LP2vo+H4RLbL0Irkk50fHCjIr3csdPaS7HHWBGZ0J9OR/SlZFWRchM9/7JK7Zp9mUrppjimgsXs7yIHxOVLh0Q+tKGt20qt7qmWJXq/D3CaGubTIavtBAXxf5DNpasWMynOiaY/ojtlpVRx/bvtdYtV2Ap1vw/ryFVB3sR5bqJ9e0v8WeZFOvhkoMWv8gbVjNV/C6636JUfjhVoeIuys4DTlTKs1X5XgJ39KlmsX9vfb06ie+TPEXOAKdf9S9pnboFmbfnv/Ty3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ZERO_PAGE can't go away, no need to hold an extra reference. Signed-off-by: David Howells Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard cc: Al Viro cc: David Hildenbrand cc: linux-fsdevel@vger.kernel.org --- fs/iomap/direct-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 9804714b1751..47db4ead1e74 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - get_page(page); + bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } From patchwork Wed Jan 25 21:06:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89F9AC54EED for ; Wed, 25 Jan 2023 21:07:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 256936B007D; Wed, 25 Jan 2023 16:07:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 206376B007E; Wed, 25 Jan 2023 16:07:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A93F6B0080; Wed, 25 Jan 2023 16:07:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F0DC56B007D for ; Wed, 25 Jan 2023 16:07:21 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C7AC01C66D5 for ; Wed, 25 Jan 2023 21:07:21 +0000 (UTC) X-FDA: 80394557082.16.BAABEC8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 19E6180023 for ; Wed, 25 Jan 2023 21:07:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cfN0jHQ0; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680840; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9PNg3WPaFWzxoLjSof2qU8psox3R+qpqOMDOMyjiLi8=; b=RvFeW8uvaqmwIucarZTgPbQ9tlye7zkEk0f/Jx+FlDrTUobt0ILheFxz9c85cPKEjyq0/P ghDHPIYW2xJUwhAsrm8rJFNRnqzPj8tculbPgVXrI/dgk3gE2a9kE9nz/giRGdICpjD7U+ 52O8rpkYVft+BFNRRU1dda/GTnigMJI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cfN0jHQ0; spf=pass (imf02.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680840; a=rsa-sha256; cv=none; b=PK7RQczQ9csqhk4Hkt3bs4ulseDGMBG0TjL9LBKKOa58fotBJDNAzikOhy0pLRycKmXHDy hB0K+vVBOTJV+Vb4FJVvoj+K0XF2otyMLPmwpmReLZM0y+f3nalibMzgNhUSozhdM9qrZc SdboFJOIcnlh+tJHTO9TFVj6YbNmygY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9PNg3WPaFWzxoLjSof2qU8psox3R+qpqOMDOMyjiLi8=; b=cfN0jHQ0zrb6l1JIzQNI+SzrfqRXsDwYizeD1ryfsLegNywyEJcuMHA1ia3sirnlhvDBFj HE5e67O65lpwk2AlOeH9TRA0MgAUlfTfHvgn9FspaG3joKEuDp+XenFQNqv5sAFsfMY5mD T/h/7uAAkc3hgw9d3toCntGeE0e9nF4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-570-EUx7EZNBM3WJWvkpN3fCVw-1; Wed, 25 Jan 2023 16:07:15 -0500 X-MC-Unique: EUx7EZNBM3WJWvkpN3fCVw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6EBF73C22744; Wed, 25 Jan 2023 21:07:14 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 67530492C14; Wed, 25 Jan 2023 21:07:12 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 4/8] block: Fix bio_flagged() so that gcc can better optimise it Date: Wed, 25 Jan 2023 21:06:53 +0000 Message-Id: <20230125210657.2335748-5-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 19E6180023 X-Rspam-User: X-Stat-Signature: hkjj5a4ch4aikced46qxnsmgkkxpp6qo X-HE-Tag: 1674680839-667641 X-HE-Meta: U2FsdGVkX1/w4icv+cszkfb6n7PCTwKMASMxacfPgyBKc17bK5JWMmgPf78vh+ipxD6yqWLLpCcKANf5JQYkZBtBIjYv0ZyJH+XAnMiMBy7wK85usLkZu6cbo/4gDR38YoZsKlFmVl5B97Usc9J5KaPSquOmuPym+j8YRJDNSZKyg3pvDVMiB0ub0cIWRKYa+MV/B9ghISPUpNNsGKxWc90o6CsTANXhzch38b5QX01PIhYidtUd0ccegpwN7N/kG6Yj2/V6xcsOW0vT6VzsW3QQ8SSsFZNlAShEohNnQ65jQ2OnIvreVD+NeXUT9KiIJWALSJpI+miWVhm3pZ8vW0pkaFNX7dqF6Iz1e7JvONFIjS1sjttgcr70a6bSP/cIyCCqB5IFFQKwN4iRgNm1ubf5/rrXuPvrcV5jrOe2nf5hvm1vvorvwxQZFkntXnkRpe6k2pSgn85jAvf/plXDsNAt8WoVnT79HKOTq+uWitc1PRZ8YjBIHkp8BNEGt9B/DhX/7heKx2NjGyu80jh94vPMUvNMLpJQqRZEx5qK0ryGK004rTv/TNffKHb3qDZixAEj8nh7bsr9LxdRNFJYmC6XNADeSl5n0hFZZcItgzWdJjHlCaGPAXgGk50N72+bOqYaWyyc3FUL6VHFmDcj+kpx/3rpbsFPWQk3MIwKVxvS19qu4HpEbT9NxnEK4WXLbgOQBg/Xhk20gY4q+yXYPLNjTYAfFlAXzIhS2UPUeYH6CLomXRRhW27eMLQiXV4HknHGkl2wF4a9cBcBrZnnuPIc1JqJ5YyXxpcydKGQjki7WoThxEyiNdS7v9VqrE+wzhjlqg3Eb72NbVvwLbPVtLeZ7NIwTfmldvhHOoXf2piIgUt5ALmJsYbpGhkQeovlrpPm68ahGo5lUxeh7xn4uNkSYpbgrabinKC4+nHORaWwrTkC/2zoMxQ7AcuS0E50tR/1ruXCGVPIPoLrmKf hFd1Vi/7 feIdlbr3DZK8IEmuFQlHX08hwiRD7sqIBlCkNk6hv93s0FfWUS/8tKJOJNIOXgzYkDeYFrVo446p5vysmkjuSDaj8ge9xkfNuPU+d1i8x2NEXgKbi39MpO0urCO++UE8UHag1MjHu1RZzxBq1TMvs9TrbOrh9jAenB6p7kwZahqVKSrp96lNxb4mNYGT5OQHwD76wu6Zoy6TuIJa75pdOMuKPSrv/N+elS0qiSHwOOFrmrwQ5VdupmMZ1k/Hp+KLWvoeIVrm3K4nmE+hsFS/lEAD+TlqEy1P5zGU+oYz/Er2CRrTTB5f5A0JRrl9AFOrLHX15oarpm3HXkbi/hGqAs7XPMr5vRuF8K0PqRorev2PWjvtBXmr0+iifUXtL7pcmhrMlGKh/Z8dKzv12hfvFKRFWO3DW1WrZJWbFLAYZ0ytnC8pzX1l5ECJ1bgELLHtU7WkA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fix bio_flagged() so that multiple instances of it, such as: if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED)) can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]). The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like: <+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk> However, the test is superfluous as the return type is bool. Removing it results in: <+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk> instead. Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Jens Axboe cc: linux-block@vger.kernel.org Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108370 [1] Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108371 [2] Link: https://lore.kernel.org/r/167391056756.2311931.356007731815807265.stgit@warthog.procyon.org.uk/ # v6 --- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/bio.h b/include/linux/bio.h index c1da63f6c808..10366b8bdb13 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -227,7 +227,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count) static inline bool bio_flagged(struct bio *bio, unsigned int bit) { - return (bio->bi_flags & (1U << bit)) != 0; + return bio->bi_flags & (1U << bit); } static inline void bio_set_flag(struct bio *bio, unsigned int bit) From patchwork Wed Jan 25 21:06:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1460C27C76 for ; Wed, 25 Jan 2023 21:07:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47A2E6B007E; Wed, 25 Jan 2023 16:07:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4036B6B0080; Wed, 25 Jan 2023 16:07:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A3386B0081; Wed, 25 Jan 2023 16:07:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1CBC06B007E for ; Wed, 25 Jan 2023 16:07:24 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B043C1A0E6F for ; Wed, 25 Jan 2023 21:07:23 +0000 (UTC) X-FDA: 80394557166.30.DD45F48 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id E001410000E for ; Wed, 25 Jan 2023 21:07:21 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dWSPeEO6; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680841; a=rsa-sha256; cv=none; b=BlqqoOADBm+JEMAxIdKpWDus5hI7i5HLGD0EfvRfLgxglvael3KHSRjBQjWV2+c3RBIoZY qzLIEJ9NCQTHRMEF59tsoSPM9R4Dc+P5jLKYPgVW2HtDC0+8O1w3i8xFney8LHEQ18pIVq ZADZ7/LxqAFeuCapE2tYDgeBwTjp0RU= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dWSPeEO6; spf=pass (imf05.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NpRga8VxfxN6lE75EkYWgLaBl9tiTC43SRcsCFIIcl8=; b=1DCJYFuJXbAzNlnMVnQf8bfZfjmi1YV9oaTPd/NWaRhjMW2y+SpRNBYU28Upq9YR4nIMRd 4l9eNIeit9WAgLsQ+YRp4Z6Tii6njSlFAlS30XK5byHVSgEmQ9FyFmpTJwE/NaaXLqYnzM 0K3VdSgkrwTDwI4I9EFyXq1MH4rJVsg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NpRga8VxfxN6lE75EkYWgLaBl9tiTC43SRcsCFIIcl8=; b=dWSPeEO6BWUi4bZUpbKDr3Eba1wjKKYrC07/V6HlVao1jjpFknGjjfUi4VZO7mte07K0GO E5aPd58N7Us/tO/f7NYELfhwgjvUtyGUORbnSgPzmttkqlxcNOZUj2WjyKztKbu63s/8Nn p5xvMIkcU2+lv5qe9P1SB3S16wBjDAU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-AkuViztfMFmk_DNAx--MGQ-1; Wed, 25 Jan 2023 16:07:18 -0500 X-MC-Unique: AkuViztfMFmk_DNAx--MGQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 29AE6858F09; Wed, 25 Jan 2023 21:07:17 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 22ED4492B02; Wed, 25 Jan 2023 21:07:15 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 5/8] block: Replace BIO_NO_PAGE_REF with BIO_PAGE_REFFED with inverted logic Date: Wed, 25 Jan 2023 21:06:54 +0000 Message-Id: <20230125210657.2335748-6-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Queue-Id: E001410000E X-Rspamd-Server: rspam01 X-Stat-Signature: nh9tbr7cgjgmyx84sg8sk9n7z5yqr63i X-HE-Tag: 1674680841-687856 X-HE-Meta: U2FsdGVkX18RoWrPfACiuezEaiSRu+vgXzi04NMKlsTGAuYfesXt/V6qRyl5gONpYVwOS2HNlbPQeseiUyXpbNxKqLoeePbPzWXqvtA7FFw2jbJ5QjVUzeNYBx1Q9RwIxq9Z8OhStq+pGuq3ORxTy0BA0iYbHwJq+eqZJvn8m37lSw4iIzw09fmSwidkcG4u4o05H9r737EydANoI6JsWvOyT+4Vp7ov5lP3ir2qJs9Ya9GRoULwiMPI1DcWWugmdkYCGsq480aqxj+xjaKcn0cexKZsik8RnEeLNU6qtbVGvIiH0lelJziD+nZw3MfQTsTCjcrq1luF294Ig3pQ1PKZYlLvzD8XlGfnf65ELUT6YA0p48mxhwT/BwCMJ84Nkq4+bHfWMZH+cU1Uwmn454YwN08vLF3i6brK/4jAgimgcsNIU9JSLuQ3yjz/Y6X91Gj+7yQxneWM0/c2XI39lPl058EXPwtK5oNO73TBD7iKO4RaiOZ4+INm93HUdU32SukVmfS4IpFo4NMnZh99iC2jrFSyonQKkRmb8GzxLApgHGdjA/8xW9vZGZTKpXt/n4tMgvzNOpzBN/53eSP6B8YSXjHbku66nzp+kXx+yOS8Hz2K4v9Wmdxe9yddhwRXm3D3+lPNAnb5mKseVECug4rvcckE80Ot1ZUkh3AE4DyKutxUx5Q5K0N4xnim9b9u+hv7+h8q05gRsLH/Df/yetgyrnEel2zE4kKP0tivhMF/hPgNFnXIXfc3YTdLknkqbbQUkSpwBIjF67CCle3OpO3jMM706fxKaUeFygpf7zvBE+XgNvqBC10AU5DXhiI5dgf0S8rM27NTwD8ZTct+QX7TPBd2Wmrj/CiaiBm84XygM0v6XmTQ3/6N+Uvzab4rMXRPOjtZj4dqzDn8CtVSAlBQkdiQ2vyhxi/1Q5UfB2o5BcOZDXxsBkrblSr5HZL3yqeEsPccw5MoFFwKyEF 7ZjQ7tcz L/c77XT0DHWY+gOTxrt/dxTaoqgNx1SzSUnyjCnstNBDXeulgLrTxwMmZJo0qDX3vUGayufKVDwbS04TL9zGrq/1R1C9c0PxkQ0/aMAtHiw1gkNOBCkKbUYP0DMc/xDNXpUJRbyf1+FbUZvcUS1TLHwMI782oP9o+kAXBSHMtD3X45jUBChDDGk0mebejIUa24nKX6FD6JYnTVdHKSAnRQx2T22qsfWRKnFWpn5VuUsi10qNJ1/3YSpnnRBHECN6eLjg2GxKb3paEKkkFgTkha95ze7m1ptpCR+kRRoNp8rOmMNQHsXsDGp4/+xeYUsPhT9AclMRWa+LUjetlO+JKMNOO2IlnNSxpvWFqfGcdPiQzxYQTLcX3xpEUneqoWzgYPx5wuOZHFBfoXc/Sh08yyq5jaK6J4A+gLhdxoSPdCCMAlrA5WU2XzgUrCrZBphLY+F1kR1Fy4FOmz8uMuf7cExFiq4HhZZqvn8HzvFJh8Z5MP8mh64pPgFQVVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Christoph Hellwig Replace BIO_NO_PAGE_REF with a BIO_PAGE_REFFED flag that has the inverted meaning is only set when a page reference has been acquired that needs to be released by bio_release_pages(). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org Reviewed-by: David Hildenbrand --- Notes: ver #8) - Don't default to BIO_PAGE_REFFED [hch]. ver #5) - Split from patch that uses iov_iter_extract_pages(). block/bio.c | 2 +- block/blk-map.c | 1 + fs/direct-io.c | 2 ++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 6 files changed, 6 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index 683444e6b711..851c23641a0d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1198,7 +1198,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; - bio_set_flag(bio, BIO_NO_PAGE_REF); bio_set_flag(bio, BIO_CLONED); } @@ -1343,6 +1342,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } + bio_set_flag(bio, BIO_PAGE_REFFED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); diff --git a/block/blk-map.c b/block/blk-map.c index 7db52ad5b2d0..0e2b0a861ba3 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -282,6 +282,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extraction_flags |= ITER_ALLOW_P2PDMA; + bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; ssize_t bytes; diff --git a/fs/direct-io.c b/fs/direct-io.c index 03d381377ae1..07810465fc9d 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -403,6 +403,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_end_io = dio_bio_end_aio; else bio->bi_end_io = dio_bio_end_io; + /* for now require references for all pages */ + bio_set_flag(bio, BIO_PAGE_REFFED); sdio->bio = bio; sdio->logical_offset_in_bio = sdio->cur_page_fs_offset; } diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 47db4ead1e74..c0e75900e754 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } diff --git a/include/linux/bio.h b/include/linux/bio.h index 10366b8bdb13..805957c99147 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,7 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (!bio_flagged(bio, BIO_NO_PAGE_REF)) + if (bio_flagged(bio, BIO_PAGE_REFFED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 99be590f952f..7daa261f4f98 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,7 +318,7 @@ struct bio { * bio flags */ enum { - BIO_NO_PAGE_REF, /* don't put release vec pages */ + BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ BIO_QUIET, /* Make BIO Quiet */ From patchwork Wed Jan 25 21:06:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCC14C27C76 for ; Wed, 25 Jan 2023 21:07:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58A276B0080; Wed, 25 Jan 2023 16:07:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53A8E6B0081; Wed, 25 Jan 2023 16:07:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B43D6B0082; Wed, 25 Jan 2023 16:07:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2C5F26B0080 for ; Wed, 25 Jan 2023 16:07:29 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DCEB3120E6D for ; Wed, 25 Jan 2023 21:07:28 +0000 (UTC) X-FDA: 80394557376.07.0E491BA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 39E441A0025 for ; Wed, 25 Jan 2023 21:07:26 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Ffx/JJ1O"; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680847; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e+R2Fab+zsp5/4JKdTrc4PXZk8RqADa0qecJwQvHEwc=; b=fu7q7U0pGhIxbw5QghgcJuKWD2mTV+MKd/wDy0MOGsCHunOQQRGoGltfRfujZ5uQiQv4hr hlTPPtpUvnmPaOC/eAjDkK+i20hzlyjzTmhUEMjnZ0IU2Fn5qLxL/GP69eySwEvWPMjWrg HvDqCbWF6o6vzR0cdt+jN4odZqp9Iao= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Ffx/JJ1O"; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680847; a=rsa-sha256; cv=none; b=saIAG9j8UQa3YyNXLEj77dCmCVNTa1NPUfNeONt/xTHEy8OhJxwAiJhF0/MCY8y4bc75QQ kzpezGv+PSfcXw3qLF6sBeNTn79vux9Oz01EbCkXQ6YmN4PlNDzPqTBHfXok0dt55aOrKq c5Nqvletn8NTpx1RRGNUiQbjNj7+QOo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e+R2Fab+zsp5/4JKdTrc4PXZk8RqADa0qecJwQvHEwc=; b=Ffx/JJ1OT0lCzNtlpRmsdAVQi9qCZKcRfB9vV85rcCWhxj7xoKLz+n8BOEdLgWeSq5O3Jg HmJ0thqpznMpX9fESNGsxk9lRnbEM36z1b+eEr7KmOQK93yn4JcUaiVBm1l/K9WHJPQL6O soBtPKxaBUhHH8LDhOX32KCoO5OgAYc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-47-63ymXbHLOMShaBfr1SrLhg-1; Wed, 25 Jan 2023 16:07:21 -0500 X-MC-Unique: 63ymXbHLOMShaBfr1SrLhg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0770285CCE0; Wed, 25 Jan 2023 21:07:20 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF1B12026D4B; Wed, 25 Jan 2023 21:07:17 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 6/8] block: Add BIO_PAGE_PINNED and associated infrastructure Date: Wed, 25 Jan 2023 21:06:55 +0000 Message-Id: <20230125210657.2335748-7-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 39E441A0025 X-Stat-Signature: mpqf91qddt585r9rndhmxr1n5645h7ry X-HE-Tag: 1674680846-968516 X-HE-Meta: U2FsdGVkX19lVAZff47SNENI2lMubad2JAvbY0j2uTYquZryu9Y7GbT4y+zHlmYqZkjtZoNPL1SmqsvJMfBuGcGEHrSBVobp/GRR5ejdS6yr8cghDqRo5qdJRURzgToi3VFzfV16kLg+njI/U93KNmb1VGs0FaJX+qB1QpqJCKVYvXdYLMdDj+NceQ+++DbNkPeq8hlj9Td2c0Trr9Yqq403tFmmfiBs7hX77KVodT+dtWz12cxeqTKKaQfq8GeukSVTygwGWjStW+WQaDXmwP52IpvjwhbQGtwid+sIWRkU/0RpnSpWMQ/houKyWGQnhpKsglT8Pr6tgN0tsePxqJ8H0pRdPoETP8be0GPW/D1Xdap1bs5y53IfjkSNMwdD4olSStAyb6Lq8fBeF8N0W/2H/0iZkbYg5fPVS3gG6Ol89UUn+9WJt7j3js5V1cBiih3GyuCPcw/va8YDQPOkkRnFpWZ5hIx7SONnG4RuV6VSvHmKDhDmxKBr54XQEKiGKKRKfP1DZGSe8lS12AYYQLg1dy4Ty4nRNcRaGVeuNAGpEa9S9SdukTG06Q6qMsrHl1m9huo1OWLMfY0Po/wArltM884r6e9PqUTSv8Q/3OnClftEvQ3pm+zgJF1Ynw4dWTKGcC4kWgOM4vg0XizzoX1YWuiv2bbnVsJNRAGTe0NzUCnB/ePDi27f3uHiCVn/fZZ0S9l8zHXAr6o6C5UlyrnJnfeQ0xezCKxYLEXAd0hs/5fQ77jr8JE3NDos2A66OAvBLZhIE8sIMjdSXU767zrd68WNYDXqCD3XCWmFl2W5eRiI230P4RnsG3rRU9JxxctqvvkRJXi5yiiYqyRVJpDKJLTJWUFTaKpULKyVHt3qyEK+z5hBxpzhKwUt6c0hftluWnnPXL5i095Uf7Plj8TpP4zOahQnQYZt44bYCCEuA1bwuoLoukARbXDvVe/n+A52FWo3VDLth83oPSG IEmnHJWq gweFLOVCFGJ5qX5DJ7QM/tN73qoeNiiCcFg/fPQzWdUAeCJUDMyyoMwvv/o6/9UNOuhGA/xrLxhOKFRuxuf95h89oOIws7bn0CbZIa6C+0Wch7rRZyzxAq2jIyFVc2HUbYWfIf6t39bXWdAZTgDV3/VPY0OGArvlSzzmUJNGhsnAUHNjtxiFogEnEdQ5jQtvwylgUhDpR32fYU0TqcaOyd//PmQLA2lOBjddwxXotoCTqNKeKFdAheVBRGm+EMP2SUM6NgEJWHn3rh4sGIeh2U0+mFppKJsnh4x0X5h1o2APySSt6a77isHLhHy+Sd3AgwDszuuW5WRoM7RsshWvrWVYBYYU+RV10DvC1MzrHh3xFXGeJvU3sCXdBgIo7j40zSIoqm/vrsKfhI2igX0UMjijrV3u2VnW+MGkl8GD5qX6pD4Dts6lN5vZpOmlZGp3UtQncM32xhuiVFVIFGk+MyvHI0GGaYlqCW0wXO8VXxGl3vNtfBu/GywZUBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add BIO_PAGE_PINNED to indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org Reviewed-by: David Hildenbrand --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #9) - Only consider pinning in bio_set_cleanup_mode(). Ref'ing pages in struct bio is going away. - page_put_unpin() is removed; call unpin_user_page() and put_page() directly. - Use bio_release_page() in __bio_release_pages(). - BIO_PAGE_PINNED and BIO_PAGE_REFFED can't both be set, so use if-else when testing both of them. ver #8) - Move the infrastructure to clean up pinned pages to this patch [hch]. - Put BIO_PAGE_PINNED before BIO_PAGE_REFFED as the latter should probably be removed at some point. FOLL_PIN can then be renumbered first. block/bio.c | 6 +++--- block/blk.h | 12 ++++++++++++ include/linux/bio.h | 3 ++- include/linux/blk_types.h | 1 + 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index 851c23641a0d..fc45aaa97696 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1176,7 +1176,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + bio_release_page(bio, bvec->bv_page); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1496,8 +1496,8 @@ void bio_set_pages_dirty(struct bio *bio) * the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from - * here on. It will run one put_page() against each page and will run one - * bio_put() against the BIO. + * here on. It will unpin each page and will run one bio_put() against the + * BIO. */ static void bio_dirty_fn(struct work_struct *work); diff --git a/block/blk.h b/block/blk.h index 4c3b3325219a..f02381405311 100644 --- a/block/blk.h +++ b/block/blk.h @@ -425,6 +425,18 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* + * Clean up a page appropriately, where the page may be pinned, may have a + * ref taken on it or neither. + */ +static inline void bio_release_page(struct bio *bio, struct page *page) +{ + if (bio_flagged(bio, BIO_PAGE_PINNED)) + unpin_user_page(page); + else if (bio_flagged(bio, BIO_PAGE_REFFED)) + put_page(page); +} + struct request_queue *blk_alloc_queue(int node_id); int disk_scan_partitions(struct gendisk *disk, fmode_t mode, void *owner); diff --git a/include/linux/bio.h b/include/linux/bio.h index 805957c99147..b2c09997d79c 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,8 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (bio_flagged(bio, BIO_PAGE_REFFED)) + if (bio_flagged(bio, BIO_PAGE_REFFED) || + bio_flagged(bio, BIO_PAGE_PINNED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 7daa261f4f98..a0e339ff3d09 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,6 +318,7 @@ struct bio { * bio flags */ enum { + BIO_PAGE_PINNED, /* Unpin pages in bio_release_pages() */ BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ From patchwork Wed Jan 25 21:06:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BDBFC54E94 for ; Wed, 25 Jan 2023 21:07:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B314F6B0081; Wed, 25 Jan 2023 16:07:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB9F96B0082; Wed, 25 Jan 2023 16:07:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90BA86B0083; Wed, 25 Jan 2023 16:07:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7D79F6B0081 for ; Wed, 25 Jan 2023 16:07:30 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5DC408085A for ; Wed, 25 Jan 2023 21:07:30 +0000 (UTC) X-FDA: 80394557460.19.0EC178B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id AD39A14000F for ; Wed, 25 Jan 2023 21:07:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZbRgbNvV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680848; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eEzlfYU/tqpy6qDlnj3bxdUXICRLHkKU5QK5uzYqBQA=; b=78w/NnMqdZSZtAoPtc6sRV6ohf8VvOk/ld5grOjx4CNehF1FndynYj1Q7oIPbxHU6Wxawj pbJo2hToeyEIpE23c71UoUenwYWEZTd66PpD8Iah1qy7ystrS+RB6HQfvNNDMqqQfCxFqw WJAvz5LjJyu82R1JFfS558CIQRhMgn0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZbRgbNvV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680848; a=rsa-sha256; cv=none; b=dDqQiukRDPiu8H65c/YWdOsas+WSyzhsk1F5qm0JhEWZMCZcBlogEdpfDEN5Bxx3nfKE3e 4uiEyU1zKXYYIKX3ZZes2Ujsz+QdFaXbLdKSXY8uIg6LbiJYPzORtHdNhjxaeGje3HmQZm ok/XyFn7GENTs5MXXtycQT2QygM/w1Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680848; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eEzlfYU/tqpy6qDlnj3bxdUXICRLHkKU5QK5uzYqBQA=; b=ZbRgbNvVzS2Mfb7v+vI2BR+p5Ej6SR88sIt8NCQShJd2hi1wWVPe8bWA+KfWbkdTKEHoDj WsWvnKSgIQFonYZVJc3U05yLZT+iwpopS2ZMgMRuWdr5x16u2iYr3wpbO1SuX+Mh/4/c/X tjhvnncvFJ23Hlm8eICekQnooDDQq9s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-331-RfvTkJ_FN4KVukCYXckWLw-1; Wed, 25 Jan 2023 16:07:23 -0500 X-MC-Unique: RfvTkJ_FN4KVukCYXckWLw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BFDFF85CCE2; Wed, 25 Jan 2023 21:07:22 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id AFAC414171BB; Wed, 25 Jan 2023 21:07:20 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 7/8] block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages Date: Wed, 25 Jan 2023 21:06:56 +0000 Message-Id: <20230125210657.2335748-8-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: AD39A14000F X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 7xdozune69ongd41kchy8ib48osu3535 X-HE-Tag: 1674680848-895909 X-HE-Meta: U2FsdGVkX1/4Ylr13Sj2j6E/46VZtJaEB+HkjjVjMVctfhPNSdd/j0joYQRk7ioAygcoeCoGaMw6gE0Z+M6Umar2iSXIFR1O74NgzPLMbOE782Wi3QjPkv2J0GUfR3H9cGUYSFyPlRTtok1X5NvGAeII2lAAUhRCDSRtOstm6PkKSW67l4INykozs+hrrddwE6JqiJV6Jh6mlZYQLwmpXYwIdbNFvuso77GZpn4s3DgkecPNGy6SaVQZKJuqenq1RgotdLOBtnV/MHszQ2HniEokxl1CgLUfcDfgNzUzXDrgrntGcX/KSr4e716g0+6Cnl5CmJkxbB5bJTzFA3Of6p8cC7d5lMYGRtGQRHuKry/kJPHumOde8CA6tZTDqKsmkPmeRvHiD9c1B45kHuqAkl57Bs4mLm7lguZenjH13Zc4yAKGbt9OZKsAMBTbdU0LNuXr5dWgfZyFDsvWwNkbbUvjJAPkJHxrma39U4csma7ClWUH69a7jtAVYiIyTxJbugyGNqs3FeyLMRRp5MLRUua9BpPbOb4fvBkOXHFuCR5wAo2fXjUI0MHenimWqBlsltSAzCsd7CCMbtIrq7JtWMk21g/2XZflbzjwad8aS2fQIU+iuZqA2UEEGjb8m1dVRwA0PkxEZRyciqQDnX4zbDFS2KSLQT5GuSp46iu4ZWEblvoXf4efUxlqiRZOoTnXhsqUz31NdE30Ax9shMb3N4bygiRSS39ewKZPuTaDTcqV+gV1mLn8kFBNV70zp0zcadYaEK43wL5be1lqF+mycq4ufaonAcXowCCTKm9WuagMPKkvgfpktWQbptcyrALsJ4rmm0HqB6bOB9keDAYlUl/ZXEzEyd4sRD0U648d/plARuGOjv4HlRDvkZv5rZUqRPAxwC4RgcIg3cuvaIN4ruRdtTMEspz1d2kIaHJcZoIpe+YC89iEHLinJINtam+aL9kFvrG3Xa8Qirr+k0Q IYFOsOpw snWgDim9VlDboo5FeWPiI7+m4ux/KzawNef2zEY98/D7hlzo0UA4Je67mHnas5WXFEzYsVv30tV4lX5XJf3Q4u3ts43dOevP9rvAi7kr/Ty6bZXDYjIApBOTfWE8yEuzWnw3t/ZAv2DcO7Bl5c3KlCEmaSNAPlfp+HwTLwuAkXK5ZLP/hNrWj/tTmAfwlXhsKHDmVFUboulbZffpKIl+cBGcVe2S9/RVGzbmd9cRh2yeSX4XKn2oCdTWT8Ugm1qZLsne5hNY9UKUhuQjLoEkEyz5ku3KptFpOklxdeRJ5LgG5faJYuA46X5jVsyWQF/MzXbulc7mBdup/8G4lvaCgrgBmAHsLZNOi+DLEqGvSOdjw4gELFoa3omvQl9ZyhSY98QH7JRG6Kx1XHj7YleD1VPzcARAgR6JIo3m23/6Ve/G7ZQC2DcXrhST3CvO4F6V3t1ydjYdGE4rZCeWz0qVf9sPR7wEXTMUfBbIrPtUrYqRNbF5NC5MI9Fq/Fw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being affected by/visible to the child process). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/bio.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/block/bio.c b/block/bio.c index fc45aaa97696..466956779d2c 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1212,7 +1212,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, } if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1226,7 +1226,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len) return -EINVAL; if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1237,10 +1237,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, * @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. - * For multi-segment *iter, this function only adds pages from the - * next non-empty segment of the iov iterator. + * Extracts pages from *iter and appends them to @bio's bvec array. The pages + * will have to be cleaned up in the way indicated by the BIO_PAGE_PINNED flag. + * For a multi-segment *iter, this function only adds pages from the next + * non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1272,9 +1272,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages(iter, pages, - UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extraction_flags); + size = iov_iter_extract_pages(iter, &pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extraction_flags, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1307,7 +1307,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) iov_iter_revert(iter, left); out: while (i < nr_pages) - put_page(pages[i++]); + bio_release_page(bio, pages[i++]); return ret; } @@ -1342,7 +1342,8 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } - bio_set_flag(bio, BIO_PAGE_REFFED); + if (iov_iter_extract_will_pin(iter)) + bio_set_flag(bio, BIO_PAGE_PINNED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); From patchwork Wed Jan 25 21:06:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13116178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC28BC27C76 for ; Wed, 25 Jan 2023 21:07:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 789AC6B0082; Wed, 25 Jan 2023 16:07:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 70FC66B0083; Wed, 25 Jan 2023 16:07:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D76F6B0085; Wed, 25 Jan 2023 16:07:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4F28E6B0082 for ; Wed, 25 Jan 2023 16:07:32 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 334031201E6 for ; Wed, 25 Jan 2023 21:07:32 +0000 (UTC) X-FDA: 80394557544.02.7816378 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 882C61A0010 for ; Wed, 25 Jan 2023 21:07:30 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OQ+HCzL6; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674680850; a=rsa-sha256; cv=none; b=1vomzzQv0iIasKJrF/PPabY0ob+mOuog+hYUCSGIEwaHUaaZ3LYSDY0XfzkT3QR6u8+tea 2iahn2E044R8j5OKmLCE8zjY/xrB/MeSaaT+OQFWBkVjbyihV8igx/y3/iU4N8hMhXWzOu uNdN/7a5OLm/++tNEawEUa9sfFb1OiY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OQ+HCzL6; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674680850; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HxbGCr5tQJM018IcUZUsOd5DaRqruIG+kHm6Wg5+DfQ=; b=t0ODYJnNOS59r9W5Aj/h+IvbxxaqdtZCHsG1Aiueu6aCJYKyuFe2Jyj9KtmMtUWR06nf3W D72AjwqBhbKlPJ0RO5x0EhJDus4LS4okhIPnFyYa8mEJnkW67ZcUbxh2V/gPGc27p++Cg6 o5gVD36trqoO1rcpM+Dv+Bi3vmM/Onk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674680849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HxbGCr5tQJM018IcUZUsOd5DaRqruIG+kHm6Wg5+DfQ=; b=OQ+HCzL64ecp3a6hCgYP6aH5k5ghWbg2ru5XDiO9wPvU5w5TsDAwyyJ9mPUGezGdJ9TxEf Ns6o55YDEiQ2VRawmMWfbEy89R1blKYsupybFqxRKBvs8Tg+6VH0gu0DcnE8ozQYFHMsDq hBb/zSrStipg8Tsys4YGM3lIcBNjfoU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-453-R8pex_C2NICT8VgrohPLJQ-1; Wed, 25 Jan 2023 16:07:26 -0500 X-MC-Unique: R8pex_C2NICT8VgrohPLJQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9A1591C0A58E; Wed, 25 Jan 2023 21:07:25 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9317A14171BB; Wed, 25 Jan 2023 21:07:23 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v10 8/8] block: convert bio_map_user_iov to use iov_iter_extract_pages Date: Wed, 25 Jan 2023 21:06:57 +0000 Message-Id: <20230125210657.2335748-9-dhowells@redhat.com> In-Reply-To: <20230125210657.2335748-1-dhowells@redhat.com> References: <20230125210657.2335748-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Queue-Id: 882C61A0010 X-Rspamd-Server: rspam01 X-Stat-Signature: bqsoaumi3g4kzupnb6fjjy4syzm13mxj X-HE-Tag: 1674680850-146377 X-HE-Meta: U2FsdGVkX18evOmNVUym6UoIwv+kUj4e+zedLzRgkfQhNm5m/bZdwWkUgUFrWK77xoO9Z3y9SQCO+Vy1ZXQkCpj8S7Smxgn3atZAabsePA7OUA9xzspkvA2pOq1tXa0URDpA5S702DeXxgiW1ZzbaR77gbMUw6WW2ccEb64X7RXDRo2P6rd0Ml6dqQHX41BU1u8FWwHtQD/dGHL902Y42IxYI+sVZSIBAO+sT4rVMdYoNvA0YOPoVvSjhwsDqcPCdY8GLGqpmOs3mZd+rgKrSB6rVz0CgfRV9BzOQEzeZUcouxSxSQ8bIpqqlMDoav8xYQwCqO8UosPuJsHN5ZQ7TonZLZ7lpKb6zYASbD4SdvAp/+9bi9vvlCNLs4WdYKDzFNuXX/2MgMp8d0jDm0ZaY9A5/Q5F+zMu1TvBo9MLtCR3tzrBAuLIMzpaM3sgAOWw5eW9lbNwsFPYwzM8pccws7v7fnqKI9r+BnxGd6YYVyC6YxcaBBRtdq/3YGMd0sdDcKxEL1CUOzKaJlkYJZS1MAa+Q7bM+TgGZTut0USQ6YC2+Qy6jQfgwtvYHVFnbis5ntRjXd/zYLHWMTRde98w0s6SkyJ5P7rLxWsdCP1PM1968kOAwWSgd4NhvgoclJDkGxiqyIN6ESmAtsf2bWQQ9IZ0+k8j3PxQOk2T/m8kKNjH5UL10uGGloGEKbo3eGNWCmjazz+oD9iZ4iBbfl3kdh3xEjR7MsI9/aXpU3yfu0CLBcU8Lf0q58p1mr45cM32HV2at3DzVp0Vab9KSwn0d9rMWWXDRThxcbm7/RqYDcQM8j+gRcDyzCfF7ukIIQ8V8Skc1828EvjMmh38MfESoB1UuvMdIZOhRB4l3x1XEIXgI2iUUXyUz+J5Lc5gOZk66F558ivv1NTQx7MPE/2QVubo8868FyjZgjVcF/b9iwnItGAx64ImCjhLb9Na450iCjyFQ9vTX/Os6Om2T2h D2Yqh9AF uMqwlYmnLhqobacXNWDpCoLkhNTILemvse6BwAkzT0aOGmWR/o2enmsmRXv0ukhWrBPhaqChUkCbznOEoLUzoRGoeEivUq6zEJ5iKLh8Va9VOSkEimeVuVh0uNM5JMMenvmTlXqq0GB6nhFduA+DfDdwYHnA19H5sEfZZUHLBloXoXvArOaFy/nlSlsW/dxx93SJg/UFDp1D7MyAg2+s71NCSyXavhZR9pTRB3h4A3YvprznHqp9cdXPtD2Sp2yeYJhzTtrnaDASwanAbM75tbLWUL4VcD+u20B7eYnlb5Ykz86HMywhLwbEhe+h5mhy0pk0uyi5FkEWwhfJSdXiHsQ9UULcg1c2FXIwa49ZjEERawV/UI2yo7Xwk+ixpl61E+tSBV5yEQQV3k1AeDYbgADafiOdUyjwUD0zZeD1eu1MUdTyrjMCe+8Hd0bhOC1aNMMjyBRK4KpE6QwL9pRynISqj/ndB2rRtckKApIpOWn9UMX+vKXPJKq8YpQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being visible to/affected by the child process). Signed-off-by: David Howells Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #10) - Drop bio_set_cleanup_mode(), open coding it instead. ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/blk-map.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index 0e2b0a861ba3..9c7ccea3f334 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -281,22 +281,21 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extraction_flags |= ITER_ALLOW_P2PDMA; + if (iov_iter_extract_will_pin(iter)) + bio_set_flag(bio, BIO_PAGE_PINNED); - bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { - struct page **pages, *stack_pages[UIO_FASTIOV]; + struct page *stack_pages[UIO_FASTIOV]; + struct page **pages = stack_pages; ssize_t bytes; size_t offs; int npages; - if (nr_vecs <= ARRAY_SIZE(stack_pages)) { - pages = stack_pages; - bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, extraction_flags); - } else { - bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, extraction_flags); - } + if (nr_vecs > ARRAY_SIZE(stack_pages)) + pages = NULL; + + bytes = iov_iter_extract_pages(iter, &pages, LONG_MAX, + nr_vecs, extraction_flags, &offs); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -318,7 +317,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (!bio_add_hw_page(rq->q, bio, page, n, offs, max_sectors, &same_page)) { if (same_page) - put_page(page); + bio_release_page(bio, page); break; } @@ -330,7 +329,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, * release the pages we didn't map into the bio, if any */ while (j < npages) - put_page(pages[j++]); + bio_release_page(bio, pages[j++]); if (pages != stack_pages) kvfree(pages); /* couldn't stuff something into bio? */