From patchwork Tue Feb 7 17:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13131879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6A8AC636D4 for ; Tue, 7 Feb 2023 17:13:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27DBC6B011F; Tue, 7 Feb 2023 12:13:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 253F06B0121; Tue, 7 Feb 2023 12:13:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16A966B0122; Tue, 7 Feb 2023 12:13:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 06D6B6B011F for ; Tue, 7 Feb 2023 12:13:25 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BB5D4AAA2D for ; Tue, 7 Feb 2023 17:13:24 +0000 (UTC) X-FDA: 80441141928.08.13DA4AA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id EA01918000F for ; Tue, 7 Feb 2023 17:13:21 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RJvJyKmh; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675790002; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AqhnRD2vIRBCcpahbYjaBxj+y0Ho2PQtP1suQNatsgY=; b=zB/4sOqTCVTHbruT1aJPDLXbQGBD8kx7V4OCqt3xq0zlSFK+oiy8QFlbOhTv31NZWP8y9B BuGAiEQh6e1xZywp74czRp4+4zTNcDZHis3CR1AmMK9oHsIq+FVjPqBnK8/0Jcxd54ztHa zldaEOeFkfMFFOczJh4J/bnd6+SDe6Y= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RJvJyKmh; spf=pass (imf24.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675790002; a=rsa-sha256; cv=none; b=QxB9iXGzj/qhRUz5Ypx/BzzLKSsI7uzIwhKNqSZExpiutz9Dm5RvT/MKBjBQ2s/mn38xRN adzhpn8PkSTgTDwrvnD8+5xQ3XlH/pD8pD7lwqAWWJRUvDuKNJvGNtR3pZn/HVqxypplaX SnM83FuVSza4W0v3eB8Vc13mxWsd6gI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675790001; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AqhnRD2vIRBCcpahbYjaBxj+y0Ho2PQtP1suQNatsgY=; b=RJvJyKmhmenVTxZEsQIhIQ5kNK7YJgir3HMrjukRpGebI6DsbEQQSwEZxy7g4NAkpVsnsO FL/C+LBJLD3HDvlDGKnsUdcAHJ+jFXmdT4aFHQ78OqRwwz52LGpXyPvIeiLhOtLaSokOl0 /dY1ICq5cQ/qpI3OfQ8bVVChRQZPbrQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-141-U0lXyxJNPFiRadUkuHNCCw-1; Tue, 07 Feb 2023 12:13:15 -0500 X-MC-Unique: U0lXyxJNPFiRadUkuHNCCw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CC8898027EB; Tue, 7 Feb 2023 17:13:14 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id C3B1940CF8EF; Tue, 7 Feb 2023 17:13:12 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com, Christoph Hellwig , John Hubbard Subject: [PATCH v12 01/10] vfs, iomap: Fix generic_file_splice_read() to avoid reversion of ITER_PIPE Date: Tue, 7 Feb 2023 17:12:56 +0000 Message-Id: <20230207171305.3716974-2-dhowells@redhat.com> In-Reply-To: <20230207171305.3716974-1-dhowells@redhat.com> References: <20230207171305.3716974-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EA01918000F X-Stat-Signature: 66aphsux8rzjmqhkejojp833mhb5pbw4 X-HE-Tag: 1675790001-581158 X-HE-Meta: U2FsdGVkX1+5Ql3DDIU/82DTAq4aGWnyXa+vs7G0cXfXKpGKVreZ/fxomzM/l3q6S0teD4mjgdnuNbGYhI/g4JJvqvj8kWadovOyyWGdKnT9MICuw/sWwwTujgddAy5NfJvFHMIqrw0Y8RiyYZpfPYR4mbDyK7VfiwmVbkHSMtJaZ6bySOD+KzPKlOWpRbg71g1o5ZZSeQmHsX/AOYshaHsRi3shvY9CTLACwPTZSsxdOU1XQmoq3QWjeA9Dke7BTjBQjCv4oJQsAH+P1f9TDv4nRkyM8cpMMx0gQc7TKr60SS616SrQohVxtrCjRUWKRMu3Vf3dnLJIRVR8S5CUszM3H78WycaQbvXeSTf75YGblOg0d2AZiXrOErgVZdBxP7MvnvwlDaOcsaEw1G7tSnvKrN0M+7PAxr+RXo2LQJTNMsa+Jb5rMwhbCzdf03ugl+YDew95xaobEH6xCmdr1EU0+N5HjpX+50bQeFXY4+FIfYCVN1OPB5hdSTuHr2DwPf5LhQfTV7iX36AeRa2yAvOycrEPQMWws76RJvyaASyTR85O92zD8q7USrmG+HH0EI61zwHnGwiiNM1cXEnv7ocpKCYhUf3TwBQM5UbQXVILdsaXXyb+1sjgxHCiC337a6AlDXFcefM3+qeqpyrBMDBsAOAt5DwFXjnf6gEf40WrPSK9EZVefrAA4NMj0HIt073WOsgneepBsjdnH79WSLGHaKoJbF4OQLvuREvw8ABMXkaXXfAZc+tOksX1/Svu/Wbl2xpkWxmPZvgyGTqdUa5VTJJt7gwUuSqpgnyO1TDEzwnVdQHknce6wdnQLFNmmuTzEMu/BeyE8IcPT1LZsYCwfAsIC+/CCu4Ff9Owgdo2cXpnB7QSMvxO1ezh//aILkPRsqip4quhERuss36TEgAqqA9XL/FDOTAGl096/kwFrzkvnBFigTI68J3wDK6VBjuYf6mEYkpc4wn5fQe UhsYspKO 6zVVVJQKJtZhVv9lYHmE6I04DFxZCTwTMM5R+JSf44joL384U7vteuS0kVr126cURagrUz0aStgTbp+WiECDQ0Olea9ss7drRM3MoBxx1pUmw9XRDo1BB7REjfF9bmy5ZS6ZBEekRLUA22hb3SFld9x40aNLAvfmG719fF/fQnc3lskOWanb9hB2AJocTOABzKjeNXNhAwkKiGKrtalJV3GoUIj/4kJq4uoo6/39I3++s5OIr1emwDIqa3YsSzmHkWealYjI6ybfRVnAHt0gWiPGqjLqYzIAfVRfm6zQaYWV7/WOX934wHuFmk6+KMTlUYQccKQQWs58x1eyIUYXf+6NefVBcDolj0AHC/A4qLP7VE+oQu6XDf5yxhRtAMiLbkijrZoF4VoJagf+m9Hqn0RydptklF9IMhCt8EnBKNkbZ6YFN4Gevt2GFPYNu/7ywqG7TrfMiDOyL1CqXliTiDAlsQU7wHn0m0hE1s5ilxkwNLJKPLkPulrffl2CJL/8+XVXQmdSfth7mqYIZxsdZZ1RPhRwldilYlc2QBFWJfnrr1C0OWiVe2MJqwACyheJaqIDbmsthqKldlsr5kBvJHcEwaOTGYmmpep79ISnZOIwoLsvFc+i6eq5a+9hsh1t8qcDReKpElhZKXrP4MNFOFcvlAkPo4sUWNQXlFAseVtrTgH0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the new iov_iter_extract_pages() function (added in a later patch), pages extracted from a non-user-backed iterator, such as ITER_PIPE, aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to shorten the iterator to just the data it is going to use - which causes the pipe buffers to be freed, even though they're attached to a bio and may get written to by DMA (thanks to Hillf Danton for spotting this[1]). This then causes massive memory corruption that is particularly noticable when the syzbot test[2] is run. The test boils down to: out = creat(argv[1], 0666); ftruncate(out, 0x800); lseek(out, 0x200, SEEK_SET); in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW); sendfile(out, in, NULL, 0x1dd00); run repeatedly in parallel. What I think is happening is that ftruncate() occasionally shortens the DIO read that's about to be made by sendfile's splice core by reducing i_size. Fix this by replacing the use of an ITER_PIPE iterator with an ITER_BVEC iterator for which reversion won't free the buffers. Bulk allocate all the buffers we think we're going to use in advance, do the read synchronously and only then trim the buffer down. The pages we did use get pushed into the pipe. This is more efficient by virtue of doing a bulk page allocation, but slightly less efficient by ignoring any partial page in the pipe. Note that this removes the only user of ITER_PIPE. Fixes: 920756a3306a ("block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages") Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com Signed-off-by: David Howells cc: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: linux-mm@kvack.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1] Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2] Signed-off-by: David Howells --- fs/splice.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 68 insertions(+), 8 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 5969b7a1d353..51778437f31f 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -295,24 +295,62 @@ void splice_shrink_spd(struct splice_pipe_desc *spd) * used as long as it has more or less sane ->read_iter(). * */ -ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, +ssize_t generic_file_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) { + LIST_HEAD(pages); struct iov_iter to; + struct bio_vec *bv; struct kiocb kiocb; - int ret; + struct page *page; + unsigned int head; + ssize_t ret; + size_t used, npages, chunk, remain, reclaim; + int i; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + npages = DIV_ROUND_UP(len, PAGE_SIZE); + + bv = kmalloc(array_size(npages, sizeof(bv[0])), GFP_KERNEL); + if (!bv) + return -ENOMEM; + + npages = alloc_pages_bulk_list(GFP_USER, npages, &pages); + if (!npages) { + kfree(bv); + return -ENOMEM; + } - iov_iter_pipe(&to, ITER_DEST, pipe, len); - init_sync_kiocb(&kiocb, in); + remain = len = min_t(size_t, len, npages * PAGE_SIZE); + + for (i = 0; i < npages; i++) { + chunk = min_t(size_t, PAGE_SIZE, remain); + page = list_first_entry(&pages, struct page, lru); + list_del_init(&page->lru); + bv[i].bv_page = page; + bv[i].bv_offset = 0; + bv[i].bv_len = chunk; + remain -= chunk; + } + + /* Do the I/O */ + iov_iter_bvec(&to, ITER_DEST, bv, npages, len); + init_sync_kiocb(&kiocb, file); kiocb.ki_pos = *ppos; - ret = call_read_iter(in, &kiocb, &to); + ret = call_read_iter(file, &kiocb, &to); + + reclaim = npages * PAGE_SIZE; + remain = 0; if (ret > 0) { + reclaim -= ret; + remain = ret; *ppos = kiocb.ki_pos; - file_accessed(in); + file_accessed(file); } else if (ret < 0) { - /* free what was emitted */ - pipe_discard_from(pipe, to.start_head); /* * callers of ->splice_read() expect -EAGAIN on * "can't put anything in there", rather than -EFAULT. @@ -321,6 +359,28 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, ret = -EAGAIN; } + /* Free any pages that didn't get touched at all. */ + for (; reclaim >= PAGE_SIZE; reclaim -= PAGE_SIZE) + __free_page(bv[--npages].bv_page); + + /* Push the remaining pages into the pipe. */ + head = pipe->head; + for (i = 0; i < npages; i++) { + struct pipe_buffer *buf = &pipe->bufs[head & (pipe->ring_size - 1)]; + + chunk = min_t(size_t, remain, PAGE_SIZE); + *buf = (struct pipe_buffer) { + .ops = &default_pipe_buf_ops, + .page = bv[i].bv_page, + .offset = 0, + .len = chunk, + }; + head++; + remain -= chunk; + } + pipe->head = head; + + kfree(bv); return ret; } EXPORT_SYMBOL(generic_file_splice_read);