From patchwork Thu Aug 25 15:24:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 791FCC32774 for ; Thu, 25 Aug 2022 15:24:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C48CF94000B; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC16394000D; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A626594000C; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 943E594000D for ; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 68BA680CBA for ; Thu, 25 Aug 2022 15:24:47 +0000 (UTC) X-FDA: 79838487414.04.F0B879C Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf23.hostedemail.com (Postfix) with ESMTP id 7A80A14000E for ; Thu, 25 Aug 2022 15:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=bxGrkGTTo43LTZ3Nm3zOWeDThYeDqrWiHdm9e0oQNtA=; b=kaCDplg1OtHM0O9TJKdP6tqye0 iCd6ssgN+loS7aD1+RS5HH/D/rM4BLjTvajFn813m4lsxD+O+Ek4zi+rZTHtuV58HvPV93K/HPRmv pEFdRa8OZIovbUc+urFK9Te6Py/uNojEQw1d+1K4uLXWMRkpHL96ZwVzgvy4yPbVQwZIJkO/cuNkN UA4ThK12k/0QDNvrphI0wQyJw3yMRn/IU4hBw8wrQacTuz/b6mE7Ah5KqfwbpIkoarZ7fslwWdOse zx3IIUBKsWxl1xBiEDOuD0qBbzxHqz2P2JUNN+FQA7oWNXiLDV9/VpPiUGiDPcOjomfOezHtajKbP sbBleevA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREij-0086M9-QV; Thu, 25 Aug 2022 09:24:38 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREid-0001eV-VM; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:18 -0600 Message-Id: <20220825152425.6296-2-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 1/8] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=kaCDplg1; spf=pass (imf23.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441086; a=rsa-sha256; cv=none; b=YjuREiWyeB8y9I0ReHxKjRJELd5VS3JV8PAIX9n8G+E8/h8pci2FPQF6YoqapGM6BsSo2e xU2l0QxPBFRBNUDf8VWwbPnW08j2Gye6GoD6H+ilrO1K8rN9IkHd07HVcjXwG2RzSCvVJC zsDGJccYanR3cvrvRy09xQEF8pLl4eE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bxGrkGTTo43LTZ3Nm3zOWeDThYeDqrWiHdm9e0oQNtA=; b=dDxJEWdqd64kpBQg+yV5wGe5VKG79PMZBbjmv1PheEqY+11d6EEOTYmekIsaNx6IhfpXqo WvT4OBRkcxtqlqYJ1aC83bzsMwkOC2GodQszILWBlAM+W8zlj4a1LjRv7zB9So4iKN+Qfz lI7DFKuJbnaCoUu1aPtFLbzMXCcpw2o= X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7A80A14000E X-Rspam-User: X-Stat-Signature: 994ktethdf9f4i195s9ydwms43376e7z Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=kaCDplg1; spf=pass (imf23.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1661441084-830111 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: GUP Callers that expect PCI P2PDMA pages can now set FOLL_PCI_P2PDMA to allow obtaining P2PDMA pages. If GUP is called without the flag and a P2PDMA page is found, it will return an error. FOLL_PCI_P2PDMA cannot be set if FOLL_LONGTERM is set. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- include/linux/mm.h | 1 + mm/gup.c | 22 +++++++++++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3bedc449c14d..37a3e91e6e77 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2891,6 +2891,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ #define FOLL_PIN 0x40000 /* pages must be released via unpin_user_page */ #define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ +#define FOLL_PCI_P2PDMA 0x100000 /* allow returning PCI P2PDMA pages */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each diff --git a/mm/gup.c b/mm/gup.c index 732825157430..79aea452619e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -566,6 +566,12 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, goto out; } + if (unlikely(!(flags & FOLL_PCI_P2PDMA) && + is_pci_p2pdma_page(page))) { + page = ERR_PTR(-EREMOTEIO); + goto out; + } + VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && !PageAnonExclusive(page), page); @@ -1015,6 +1021,9 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; + if ((gup_flags & FOLL_LONGTERM) && (gup_flags & FOLL_PCI_P2PDMA)) + return -EOPNOTSUPP; + if (vma_is_secretmem(vma)) return -EFAULT; @@ -2359,6 +2368,10 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, VM_BUG_ON(!pfn_valid(pte_pfn(pte))); page = pte_page(pte); + if (unlikely(!(flags & FOLL_PCI_P2PDMA) && + is_pci_p2pdma_page(page))) + goto pte_unmap; + folio = try_grab_folio(page, 1, flags); if (!folio) goto pte_unmap; @@ -2438,6 +2451,12 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr, undo_dev_pagemap(nr, nr_start, flags, pages); break; } + + if (!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { + undo_dev_pagemap(nr, nr_start, flags, pages); + break; + } + SetPageReferenced(page); pages[*nr] = page; if (unlikely(!try_grab_page(page, flags))) { @@ -2926,7 +2945,8 @@ static int internal_get_user_pages_fast(unsigned long start, if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM | FOLL_FORCE | FOLL_PIN | FOLL_GET | - FOLL_FAST_ONLY | FOLL_NOFAULT))) + FOLL_FAST_ONLY | FOLL_NOFAULT | + FOLL_PCI_P2PDMA))) return -EINVAL; if (gup_flags & FOLL_PIN) From patchwork Thu Aug 25 15:24:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66E67C28D13 for ; Thu, 25 Aug 2022 15:24:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E80FC940008; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBD96940007; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6B00940008; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AE2566B0075 for ; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 80165406E3 for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) X-FDA: 79838487372.06.D4531AC Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf28.hostedemail.com (Postfix) with ESMTP id 19F5FC0006 for ; Thu, 25 Aug 2022 15:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=JQ3qiW538t9ABcApNKq4W32OZxdHmb0EGXdVRgOkT1A=; b=mL1bcL2m3muagTl7k/hmjjD2sd WrGCiFwORUktHwhS6nJWPH2C/y3bRvzX6UhnbrEhLzyM3G7AGiNs7WpJfduS4yQ+TtLr3MyMHs5yL CWSJb9tMAb1GLhvJnfHKYifj41RK8KGr2qzUpybsOPuSQrV+eVgrLR6i1x2CbEKZtW3TREUfNc84H lxZYhh2cVE1+0Ja1vSY+HqqjdBAp/XLEhAVhSdmPse6kUBbQxwXI6q2R3hTK7ZJGQjXp9zHOAijYS GSTBGpijJgtCqQWh8U65wMlklqsAnSfTM+W1+Sh/d9x1x5cE8wNn9k8Ry2JK+22T576pcjmcyhi4i Epk0iVoQ==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREij-0086MA-QV; Thu, 25 Aug 2022 09:24:39 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREie-0001eY-4B; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:19 -0600 Message-Id: <20220825152425.6296-3-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 2/8] iov_iter: introduce iov_iter_get_pages_[alloc_]flags() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441086; a=rsa-sha256; cv=none; b=oKde6tLaP/Ieob2bVeZNEdE+0prxbJQFner0s21w7JtpWCa7YSuoyXMhN4zjcGbWcda8Hi 8Id24BGwoPo5vz7rzTVgtHMP54M/pCZFdPlBbxq79QsZrBoNRH5MaHZ5Zrwa1MQ6N5dhuX RO352+uI2emuz2z99e9JJoFye5j++YM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mL1bcL2m; spf=pass (imf28.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JQ3qiW538t9ABcApNKq4W32OZxdHmb0EGXdVRgOkT1A=; b=5iIve/sEo8egXvEq33Zd0Imj5xkJi26ww4ffbbGMsAvPVB3fs/yHIFi64wyLixCTUCI6nb 5HVDkfxCjIRlmSF6X9HJmqwmQSM1y2maWjoRo7GZK4bD7pouHq9e/OpvecZeto7MkDFDYv Ow3ONilS+18Ptjd1q/fEUZ75wi7VBXg= X-Stat-Signature: ae6wzsbnr5fjja4b6d5r6o4wzucag1x1 X-Rspamd-Queue-Id: 19F5FC0006 X-Rspam-User: X-Rspamd-Server: rspam06 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mL1bcL2m; spf=pass (imf28.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1661441084-946094 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add iov_iter_get_pages_flags() and iov_iter_get_pages_alloc_flags() which take a flags argument that is passed to get_user_pages_fast(). This is so that FOLL_PCI_P2PDMA can be passed when appropriate. Signed-off-by: Logan Gunthorpe Reviewed-by: John Hubbard --- include/linux/uio.h | 6 ++++++ lib/iov_iter.c | 40 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 5896af36199c..76ba69edb8c5 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -247,8 +247,14 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inode void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray, loff_t start, size_t count); +ssize_t iov_iter_get_pages_flags(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned maxpages, size_t *start, + unsigned int gup_flags); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); +ssize_t iov_iter_get_pages_alloc_flags(struct iov_iter *i, + struct page ***pages, size_t maxsize, size_t *start, + unsigned int gup_flags); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 4b7fce72e3e5..dedb78f3c655 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1427,7 +1427,8 @@ static struct page *first_bvec_segment(const struct iov_iter *i, static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, - unsigned int maxpages, size_t *start) + unsigned int maxpages, size_t *start, + unsigned int gup_flags) { unsigned int n; @@ -1439,7 +1440,6 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, maxsize = MAX_RW_COUNT; if (likely(user_backed_iter(i))) { - unsigned int gup_flags = 0; unsigned long addr; int res; @@ -1497,10 +1497,24 @@ ssize_t iov_iter_get_pages2(struct iov_iter *i, return 0; BUG_ON(!pages); - return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, start); + return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, + start, 0); } EXPORT_SYMBOL(iov_iter_get_pages2); +ssize_t iov_iter_get_pages_flags(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned maxpages, size_t *start, + unsigned int gup_flags) +{ + if (!maxpages) + return 0; + BUG_ON(!pages); + + return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, + start, gup_flags); +} +EXPORT_SYMBOL_GPL(iov_iter_get_pages_flags); + ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) @@ -1509,7 +1523,7 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, *pages = NULL; - len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start); + len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, 0); if (len <= 0) { kvfree(*pages); *pages = NULL; @@ -1518,6 +1532,24 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); +ssize_t iov_iter_get_pages_alloc_flags(struct iov_iter *i, + struct page ***pages, size_t maxsize, + size_t *start, unsigned int gup_flags) +{ + ssize_t len; + + *pages = NULL; + + len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, + gup_flags); + if (len <= 0) { + kvfree(*pages); + *pages = NULL; + } + return len; +} +EXPORT_SYMBOL_GPL(iov_iter_get_pages_alloc_flags); + size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { From patchwork Thu Aug 25 15:24:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66FF0C6498F for ; Thu, 25 Aug 2022 15:24:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77939940009; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 66A7794000C; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 46E36940009; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 150476B007B for ; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DA41480D1A for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) X-FDA: 79838487372.11.EC2A471 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf29.hostedemail.com (Postfix) with ESMTP id 8AF7B120016 for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=K5Jc8scvLUvhDTN44UnDRvk62eVtMjOsLaEb4UWxg7M=; b=TAse/YlGEzCihIsPGWyu6RMzTu yVK3Z1T9yYnlVtvmVzsxxqVjBgdqX7sn3+vugZG+YautHnMksjYjBxO4uRGs2yNGc5Zd70AHnj/8t OKgc66oyUrOurIsgqvHKdljgGD8qK7+t6en6l3GHHZ72iJvO63y0vSn64Yj1tDJxqglXS2TuNjpOC 5DDMrI6DkZCYkhfmF6vz1VZopGSU1I5Hl3KwpXsNddsfPYhPRCJsXyvh3WFofP9kODtJTQslzXHFP di3HWbMlwtbBe5NGrNBDOqE53/6YQC7Q13W2bgh1/aEpIKI05z9wd8VybMTyt/0ZG+4gsxawAYGoa 098afT0Q==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREij-0086MB-QX; Thu, 25 Aug 2022 09:24:39 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREie-0001ec-8l; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:20 -0600 Message-Id: <20220825152425.6296-4-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 3/8] block: add check when merging zone device pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b="TAse/YlG"; spf=pass (imf29.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441086; a=rsa-sha256; cv=none; b=q85wUEk2MRCRzq/oWEOxUyU7FwM3GHi7yMvOizydknvOk8H2i/RGKLn7fqZ/+GYAwjeQ2r rTazeQF/6Qi6DO+74sLw4TtQsX3yE7d8skPyY2qicjFTcxEV00wvvveA8Rlx+iqZp85mNg ONcBpC3XibgdRdzNRDraWfkgoD0WZ4k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K5Jc8scvLUvhDTN44UnDRvk62eVtMjOsLaEb4UWxg7M=; b=GCOsxVlmJpr+QR7r47susDMrVFgtiBzxalaH/MkLFch5leYmtpkoNoWm0CUQ608QbCegLu /fw3X2tZt9ebT7XSmuDDp2mTieXIVsfQ2seWzWACIjAN061VqsCFGciorTz4iIcDoChTsL qx0kznNTaqVwuesxw2JXriDwGh068/8= Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b="TAse/YlG"; spf=pass (imf29.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-Rspam-User: X-Stat-Signature: teg3rpdznca5sq3ekwncexwuchaxf4ec X-Rspamd-Queue-Id: 8AF7B120016 X-Rspamd-Server: rspam04 X-HE-Tag: 1661441086-312160 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Consecutive zone device pages should not be merged into the same sgl or bvec segment with other types of pages or if they belong to different pgmaps. Otherwise getting the pgmap of a given segment is not possible without scanning the entire segment. This helper returns true either if both pages are not zone device pages or both pages are zone device pages with the same pgmap. Add a helper to determine if zone device pages are mergeable and use this helper in page_is_mergeable(). Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- block/bio.c | 2 ++ include/linux/mmzone.h | 24 ++++++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/block/bio.c b/block/bio.c index 3d3a2678fea2..969607bc1f4d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -865,6 +865,8 @@ static inline bool page_is_mergeable(const struct bio_vec *bv, return false; if (xen_domain() && !xen_biovec_phys_mergeable(bv, page)) return false; + if (!zone_device_pages_have_same_pgmap(bv->bv_page, page)) + return false; *same_page = ((vec_end_addr & PAGE_MASK) == page_addr); if (*same_page) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index e24b40c52468..2c31915b057e 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -794,6 +794,25 @@ static inline bool is_zone_device_page(const struct page *page) { return page_zonenum(page) == ZONE_DEVICE; } + +/* + * Consecutive zone device pages should not be merged into the same sgl + * or bvec segment with other types of pages or if they belong to different + * pgmaps. Otherwise getting the pgmap of a given segment is not possible + * without scanning the entire segment. This helper returns true either if + * both pages are not zone device pages or both pages are zone device pages + * with the same pgmap. + */ +static inline bool zone_device_pages_have_same_pgmap(const struct page *a, + const struct page *b) +{ + if (is_zone_device_page(a) != is_zone_device_page(b)) + return false; + if (!is_zone_device_page(a)) + return true; + return a->pgmap == b->pgmap; +} + extern void memmap_init_zone_device(struct zone *, unsigned long, unsigned long, struct dev_pagemap *); #else @@ -801,6 +820,11 @@ static inline bool is_zone_device_page(const struct page *page) { return false; } +static inline bool zone_device_pages_have_same_pgmap(const struct page *a, + const struct page *b) +{ + return true; +} #endif static inline bool folio_is_zone_device(const struct folio *folio) From patchwork Thu Aug 25 15:24:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92E04C32774 for ; Thu, 25 Aug 2022 15:24:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 559AF940007; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50BC694000B; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 341CB94000A; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 033DC6B0075 for ; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C867D1403F2 for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) X-FDA: 79838487372.20.D2AA12E Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf14.hostedemail.com (Postfix) with ESMTP id 8503710000E for ; Thu, 25 Aug 2022 15:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=PNE4B8sXwOVMOk+3rqBkaNjhAgYrAxODM+UtKodjnl4=; b=blWJgVk3dMLXB9Hn8RuPfUoC0X 14dFNFjLX5NPIJM12TD3x1Ff5i8KItpOFIT6ngkqLr6P5dN+ZseUqea/oX0Jh6mbgxeufH94pmnxR YWNj1cKBdx6e89DpKvvqrl4M3NyL8YZeMHTJjoriIU82VWUxxsalH6aH3+dxyBMzORat8o9RYJ3su dRKf03bADuprGQ05RTIOCvqU8rIuPgfxIC3/k5rLPBXfZbxvyCfTrHn0Cs8V1r2nuN82JKFPGvNx6 ARAzsSfmXrNkHGfrMewamAQWvZgPLup5H2IFOL4sPhpUsJ/BBP0wHhvwu2+9BRzCGMyc6UVSukB28 Hb9Dh65g==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREij-0086MC-QX; Thu, 25 Aug 2022 09:24:38 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREie-0001ef-EU; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:21 -0600 Message-Id: <20220825152425.6296-5-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 4/8] lib/scatterlist: add check when merging zone device pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PNE4B8sXwOVMOk+3rqBkaNjhAgYrAxODM+UtKodjnl4=; b=vyGZhIP1BWyElwyx/oDkiljFUaWR+FTxEn3vAK8V6GM2B9i339ODJeo/U73zq8SzSakP4X QO4iAq6og91djlixzhZuIBXHkUHQu3gdh+JazScYU4LuA5gTQOaoH8egsQw8r+VdyO9GXP YJWr4qLAO96n/rJlhrfK0glwN2qr+pY= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=blWJgVk3; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf14.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441086; a=rsa-sha256; cv=none; b=XDZ9wxajey49hCe2KPAGbhQrD+v1FavmqCULGtJrqP2wRrbPUY2tsBtsFevoiDV1ETHPim GjQ2SNXdG/wzaJXfQi7tESPwGkkzA9/QLou59MsCZmev+qKIY7ANi40aTUPSR5W/TyFKM+ cjpCvLZHgVhR5jOQmMSIOTE1InP6oWk= Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=blWJgVk3; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf14.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspam-User: X-Rspamd-Queue-Id: 8503710000E X-Rspamd-Server: rspam10 X-Stat-Signature: g5dtpbn71dehkdazjw67a9pun8cfeydu X-HE-Tag: 1661441084-100096 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Consecutive zone device pages should not be merged into the same sgl or bvec segment with other types of pages or if they belong to different pgmaps. Otherwise getting the pgmap of a given segment is not possible without scanning the entire segment. This helper returns true either if both pages are not zone device pages or both pages are zone device pages with the same pgmap. Factor out the check for page mergability into a pages_are_mergable() helper and add a check with zone_device_pages_are_mergeable(). Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- lib/scatterlist.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/lib/scatterlist.c b/lib/scatterlist.c index c8c3d675845c..a0ad2a7959b5 100644 --- a/lib/scatterlist.c +++ b/lib/scatterlist.c @@ -410,6 +410,15 @@ static struct scatterlist *get_next_sg(struct sg_append_table *table, return new_sg; } +static bool pages_are_mergeable(struct page *a, struct page *b) +{ + if (page_to_pfn(a) != page_to_pfn(b) + 1) + return false; + if (!zone_device_pages_have_same_pgmap(a, b)) + return false; + return true; +} + /** * sg_alloc_append_table_from_pages - Allocate and initialize an append sg * table from an array of pages @@ -447,6 +456,7 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, unsigned int chunks, cur_page, seg_len, i, prv_len = 0; unsigned int added_nents = 0; struct scatterlist *s = sgt_append->prv; + struct page *last_pg; /* * The algorithm below requires max_segment to be aligned to PAGE_SIZE @@ -460,21 +470,17 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, return -EOPNOTSUPP; if (sgt_append->prv) { - unsigned long paddr = - (page_to_pfn(sg_page(sgt_append->prv)) * PAGE_SIZE + - sgt_append->prv->offset + sgt_append->prv->length) / - PAGE_SIZE; - if (WARN_ON(offset)) return -EINVAL; /* Merge contiguous pages into the last SG */ prv_len = sgt_append->prv->length; - while (n_pages && page_to_pfn(pages[0]) == paddr) { + last_pg = sg_page(sgt_append->prv); + while (n_pages && pages_are_mergeable(last_pg, pages[0])) { if (sgt_append->prv->length + PAGE_SIZE > max_segment) break; sgt_append->prv->length += PAGE_SIZE; - paddr++; + last_pg = pages[0]; pages++; n_pages--; } @@ -488,7 +494,7 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, for (i = 1; i < n_pages; i++) { seg_len += PAGE_SIZE; if (seg_len >= max_segment || - page_to_pfn(pages[i]) != page_to_pfn(pages[i - 1]) + 1) { + !pages_are_mergeable(pages[i], pages[i - 1])) { chunks++; seg_len = 0; } @@ -504,8 +510,7 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, for (j = cur_page + 1; j < n_pages; j++) { seg_len += PAGE_SIZE; if (seg_len >= max_segment || - page_to_pfn(pages[j]) != - page_to_pfn(pages[j - 1]) + 1) + !pages_are_mergeable(pages[j], pages[j - 1])) break; } From patchwork Thu Aug 25 15:24:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A64DC64990 for ; Thu, 25 Aug 2022 15:24:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A50B94000A; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9333A94000B; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A4D194000A; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2DB6D940007 for ; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F058AC0973 for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) X-FDA: 79838487372.09.D70ED06 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf22.hostedemail.com (Postfix) with ESMTP id 79A88C0002 for ; Thu, 25 Aug 2022 15:24:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=mLYWjwjaacuhTQSD3LMzZRV3Gg44uzd/+90ZlUZOo0E=; b=JlA02I9mYM/1XyzPhpPhzA8jen fB0btV3snf/z9KW6dTxn6u/abas7864PAvz+ynWXFzPL1rlGI2s0oRw7UP5yTir5n890MUigIrDsk zA3uu2g8QdinM4+B36OB6W5BEPoH1yqALmoP39OJfN4rqGUrjdfTbLcsqUme4Z8t54ow3fe5lfC/c Fl9hCoKvjhE1chLSEIBX9qOh/De48gdBzb/HewRblhZ8sJd79WaorHbd18g64W02PKJcBAlkL+Lii 02q17F11RI0GEWM8Baan/tqBiUlho3AxoVa7WA6H4kQJXp8aOTGBtLmwjDzjftTk2ncaWEi5g6yYK Qt6fGg3Q==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREim-0086MB-Bp; Thu, 25 Aug 2022 09:24:41 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREie-0001ei-J0; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:22 -0600 Message-Id: <20220825152425.6296-6-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 5/8] block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441086; a=rsa-sha256; cv=none; b=UtS8EZS+60GVN2vPmOhCGbGHOwiOQLNWi3DQ4UprWYciijtlGjhDmpXDlVcPIeUh9HMSLB isDUVRdR7cY/mkBLbVu1DzvygdNbYDiCxJbu8kEMZIQDDx7pOZdR1La0Yt1y5yLAIWwzwx g0iSPC4jbwqTILGaFFGJisFgIy9hxEE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=JlA02I9m; spf=pass (imf22.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mLYWjwjaacuhTQSD3LMzZRV3Gg44uzd/+90ZlUZOo0E=; b=QAO7kQqIPS/inmUJW6FWiK/U81jtPGeAVfhMttxzq/4RioDWNuyczBVrsRT3WI+nPkU7ju tbVKtH997obxEdvdBQF4uTGZPCAkh64pB7R8QqwKdA0R0lFxh2cM6yPTp26qiz/EhPxk2O nHlJbSE67MVIDJuIOthDwZ2P2n+jmcc= X-Rspam-User: Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=JlA02I9m; spf=pass (imf22.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-Rspamd-Server: rspam12 X-Stat-Signature: 7ipwr6w6uk9cm7atgrwiwp1gkm9maodr X-Rspamd-Queue-Id: 79A88C0002 X-HE-Tag: 1661441085-36183 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a bio's queue supports PCI P2PDMA, set FOLL_PCI_P2PDMA for iov_iter_get_pages_flags(). This allows PCI P2PDMA pages to be passed from userspace and enables the O_DIRECT path in iomap based filesystems and direct to block devices. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- block/bio.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/block/bio.c b/block/bio.c index 969607bc1f4d..ca09d79a0683 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1200,6 +1200,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; + unsigned int flags = 0; ssize_t size, left; unsigned len, i = 0; size_t offset, trim; @@ -1213,6 +1214,10 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); + if (bio->bi_bdev && bio->bi_bdev->bd_disk && + blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) + flags |= FOLL_PCI_P2PDMA; + /* * Each segment in the iov is required to be a block size multiple. * However, we may not be able to get the entire segment if it spans @@ -1220,8 +1225,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages2(iter, pages, UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset); + size = iov_iter_get_pages_flags(iter, pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, &offset, flags); if (unlikely(size <= 0)) return size ? size : -EFAULT; From patchwork Thu Aug 25 15:24:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D8FDC676B1 for ; Thu, 25 Aug 2022 15:24:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD50994000C; Thu, 25 Aug 2022 11:24:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2C3894000F; Thu, 25 Aug 2022 11:24:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7561594000C; Thu, 25 Aug 2022 11:24:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 50B9B94000D for ; Thu, 25 Aug 2022 11:24:48 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 23788AC73D for ; Thu, 25 Aug 2022 15:24:48 +0000 (UTC) X-FDA: 79838487456.11.41139DE Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf02.hostedemail.com (Postfix) with ESMTP id 164708002A for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=XnsEhea8heTLvEtJfQaMF6obPzklycKbPSf5ut+0ozc=; b=svROYF9VMXM1SZ5ev3XJ01x4LC 132cuMYxm0cuR1UgHM3+ccukTzKWFh4N4zBMoHf2MgOmzFLKWAMUgffufecNmtGwBVG1BfTSfJNgP DQ7SkRZM0drTSk17r0vSpQZMcNyBhMFwFi/NYCKX2/DfsVzJj+4R15Avwr7ZHj44aM3F5kurf1EXk XVGuyvzzF0FOND0stT084IVHnjhtD6wkCt7045x3i0hnT6j0pw4PatKrJmTCTR3vh8JcYX0U6hiIc TZuauDQKHVCn0wr8oOO6st5WkQOKC2Q+aQJadKK/pOvMUI6UaBMKBp1GzoXHe/SMIgfuBUh4yyrbv rU+Dlwxg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREik-0086MC-P0; Thu, 25 Aug 2022 09:24:39 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREie-0001el-Nq; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:23 -0600 Message-Id: <20220825152425.6296-7-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 6/8] block: set FOLL_PCI_P2PDMA in bio_map_user_iov() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441087; a=rsa-sha256; cv=none; b=sD6/7OnTb20a1Mdvuv1+HGMWi/DE7Ui/lVCP45zPNhGHerLHBeHWWKwOZCfAc7ApXQI/O+ Z8Jz9z/5assOwZfEnJjCX3nTNsZPlYRmPq/a9fF18+yNtFV6+cz8eLAWRLEL7LQ2MzUTtF o4Vuc5fuwM4NIEzO6H+LZvnoVffG9sM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=svROYF9V; spf=pass (imf02.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441087; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XnsEhea8heTLvEtJfQaMF6obPzklycKbPSf5ut+0ozc=; b=Iq/kg7MZTYLsTq9L6bXjB2IZLmt4t28HdCc/2hSAxBQisZfqc2VguYU5dQM0cocPAtqmFu U51AIoAWuD4YXpeDRCVbdaGkgQqAY8kTd35YajWACBiuojBrpkWpSQwSg1C/5OrpMzS+Kp seUwSXqnwKhQXrrRgmsbAJDUHJkTQXY= X-Stat-Signature: 1c5gkitkri9jky9m7tkj5rbuqoq7je4o X-Rspamd-Queue-Id: 164708002A X-Rspam-User: X-Rspamd-Server: rspam06 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=svROYF9V; spf=pass (imf02.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1661441086-867838 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a bio's queue supports PCI P2PDMA, set FOLL_PCI_P2PDMA for iov_iter_get_pages_flags(). This allows PCI P2PDMA pages to be passed from userspace and enables the NVMe passthru requests to use P2PDMA pages. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- block/blk-map.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/block/blk-map.c b/block/blk-map.c index 7196a6b64c80..1378f49ca5ca 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -236,6 +236,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, { unsigned int max_sectors = queue_max_hw_sectors(rq->q); unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); + unsigned int flags = 0; struct bio *bio; int ret; int j; @@ -248,13 +249,17 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, return -ENOMEM; bio_init(bio, NULL, bio->bi_inline_vecs, nr_vecs, req_op(rq)); + if (blk_queue_pci_p2pdma(rq->q)) + flags |= FOLL_PCI_P2PDMA; + while (iov_iter_count(iter)) { struct page **pages; ssize_t bytes; size_t offs, added = 0; int npages; - bytes = iov_iter_get_pages_alloc2(iter, &pages, LONG_MAX, &offs); + bytes = iov_iter_get_pages_alloc_flags(iter, &pages, LONG_MAX, + &offs, flags); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; From patchwork Thu Aug 25 15:24:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5F5BC6498F for ; Thu, 25 Aug 2022 15:24:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12A4B6B0078; Thu, 25 Aug 2022 11:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECF09940009; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D81526B007B; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C2045940007 for ; Thu, 25 Aug 2022 11:24:46 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 76FF6AC76E for ; Thu, 25 Aug 2022 15:24:46 +0000 (UTC) X-FDA: 79838487372.10.E469E8C Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf21.hostedemail.com (Postfix) with ESMTP id 009F71C0011 for ; Thu, 25 Aug 2022 15:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=jHRxLz2djGGS1TM542jkP+PzdqwM8h/KFYgaj6hQq10=; b=auQ8tkE4WiGZMxITEDr0NehtLa pjduVL2MTl1EiBVIWvrNWahaQ5IAFoEUwcLTSHpGK2r6Oe7SxBi+IoVGHvbBpi+CXWY2XBo0a7zhv pDZJqZJkhOYRFSM1hA9uDW1z4W3hMxIJBQnMkDchJtZJnzA20Pu7ybmXMSyg3/G8Or3YDlqu6sClq 3SyNxXu76FGeRui6SKCsyCsBGskr47P/3F2Zsog/zrqzp8FBoym3K0em4jkCFauHESXwUH9TW3iY8 B91rE1P2LZIVVI99tN4nwLTnWRNVd3Uejp5tslsOCEhyYNKMadbzbS8yFg8DsO9sv68gCv0XkiaT2 IuGjyLIg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREim-0086MC-BQ; Thu, 25 Aug 2022 09:24:41 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREie-0001eo-SF; Thu, 25 Aug 2022 09:24:32 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:24 -0600 Message-Id: <20220825152425.6296-8-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 7/8] PCI/P2PDMA: Allow userspace VMA allocations through sysfs X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661441086; a=rsa-sha256; cv=none; b=5Ozx5k9TkQ2iLzwbRl1dS627OX6I7Dk0uycbWD8I9bFK2Vb12r/7a124HtbUwnrZqlpVIr 9syUvgYkViSjBL010oXxBaSvycZDa1pzRkm1g3jcnIemTgKfe86eImJ26UP/xRb6KQKZ2x jC2nfhGq9/vVMZCQk2QKbs+SPdGMYCA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=auQ8tkE4; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf21.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661441086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jHRxLz2djGGS1TM542jkP+PzdqwM8h/KFYgaj6hQq10=; b=vk5RCrRo+42aHsnVNwkYDda40WLeeb4mmZ2q2kROfPNXfPRznA+7epQ2cu4lUm3q+kqRUJ OaNxbDO5rStL5M7IgC3T9eLSy2pcOF+buiaPYhpujQ4D3w4CbNNLunko69J1o8Y1HDd1bN qdza3CwXNhO9JDH1OJhoFXX3nQ3Sbs0= X-Stat-Signature: cqhrs9gxqa7untkxnuiqdm1qhk4q3sht X-Rspamd-Queue-Id: 009F71C0011 X-Rspam-User: X-Rspamd-Server: rspam11 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=auQ8tkE4; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf21.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-HE-Tag: 1661441084-541035 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a sysfs bin attribute called "allocate" under the existing "p2pmem" group. The only allowable operation on this file is the mmap() call. When mmap() is called on this attribute, the kernel allocates a chunk of memory from the genalloc and inserts the pages into the VMA. The dev_pagemap .page_free callback will indicate when these pages are no longer used and they will be put back into the genalloc. On device unbind, remove the sysfs file before the memremap_pages are cleaned up. This ensures unmap_mapping_range() is called on the files inode and no new mappings can be created. Signed-off-by: Logan Gunthorpe --- drivers/pci/p2pdma.c | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 4496a7c5c478..a6ed6bbca214 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -89,6 +89,90 @@ static ssize_t published_show(struct device *dev, struct device_attribute *attr, } static DEVICE_ATTR_RO(published); +static int p2pmem_alloc_mmap(struct file *filp, struct kobject *kobj, + struct bin_attribute *attr, struct vm_area_struct *vma) +{ + struct pci_dev *pdev = to_pci_dev(kobj_to_dev(kobj)); + size_t len = vma->vm_end - vma->vm_start; + struct pci_p2pdma *p2pdma; + struct percpu_ref *ref; + unsigned long vaddr; + void *kaddr; + int ret; + + /* prevent private mappings from being established */ + if ((vma->vm_flags & VM_MAYSHARE) != VM_MAYSHARE) { + pci_info_ratelimited(pdev, + "%s: fail, attempted private mapping\n", + current->comm); + return -EINVAL; + } + + if (vma->vm_pgoff) { + pci_info_ratelimited(pdev, + "%s: fail, attempted mapping with non-zero offset\n", + current->comm); + return -EINVAL; + } + + rcu_read_lock(); + p2pdma = rcu_dereference(pdev->p2pdma); + if (!p2pdma) { + ret = -ENODEV; + goto out; + } + + kaddr = (void *)gen_pool_alloc_owner(p2pdma->pool, len, (void **)&ref); + if (!kaddr) { + ret = -ENOMEM; + goto out; + } + + /* + * vm_insert_page() can sleep, so a reference is taken to mapping + * such that rcu_read_unlock() can be done before inserting the + * pages + */ + if (unlikely(!percpu_ref_tryget_live_rcu(ref))) { + ret = -ENODEV; + goto out_free_mem; + } + rcu_read_unlock(); + + for (vaddr = vma->vm_start; vaddr < vma->vm_end; vaddr += PAGE_SIZE) { + ret = vm_insert_page(vma, vaddr, virt_to_page(kaddr)); + if (ret) { + gen_pool_free(p2pdma->pool, (uintptr_t)kaddr, len); + return ret; + } + percpu_ref_get(ref); + put_page(virt_to_page(kaddr)); + kaddr += PAGE_SIZE; + len -= PAGE_SIZE; + } + + percpu_ref_put(ref); + + return 0; +out_free_mem: + gen_pool_free(p2pdma->pool, (uintptr_t)kaddr, len); +out: + rcu_read_unlock(); + return ret; +} + +static struct bin_attribute p2pmem_alloc_attr = { + .attr = { .name = "allocate", .mode = 0660 }, + .mmap = p2pmem_alloc_mmap, + /* + * Some places where we want to call mmap (ie. python) will check + * that the file size is greater than the mmap size before allowing + * the mmap to continue. To work around this, just set the size + * to be very large. + */ + .size = SZ_1T, +}; + static struct attribute *p2pmem_attrs[] = { &dev_attr_size.attr, &dev_attr_available.attr, @@ -96,11 +180,32 @@ static struct attribute *p2pmem_attrs[] = { NULL, }; +static struct bin_attribute *p2pmem_bin_attrs[] = { + &p2pmem_alloc_attr, + NULL, +}; + static const struct attribute_group p2pmem_group = { .attrs = p2pmem_attrs, + .bin_attrs = p2pmem_bin_attrs, .name = "p2pmem", }; +static void p2pdma_page_free(struct page *page) +{ + struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page->pgmap); + struct percpu_ref *ref; + + gen_pool_free_owner(pgmap->provider->p2pdma->pool, + (uintptr_t)page_to_virt(page), PAGE_SIZE, + (void **)&ref); + percpu_ref_put(ref); +} + +static const struct dev_pagemap_ops p2pdma_pgmap_ops = { + .page_free = p2pdma_page_free, +}; + static void pci_p2pdma_release(void *data) { struct pci_dev *pdev = data; @@ -152,6 +257,19 @@ static int pci_p2pdma_setup(struct pci_dev *pdev) return error; } +static void pci_p2pdma_unmap_mappings(void *data) +{ + struct pci_dev *pdev = data; + + /* + * Removing the alloc attribute from sysfs will call + * unmap_mapping_range() on the inode, teardown any existing userspace + * mappings and prevent new ones from being created. + */ + sysfs_remove_file_from_group(&pdev->dev.kobj, &p2pmem_alloc_attr.attr, + p2pmem_group.name); +} + /** * pci_p2pdma_add_resource - add memory for use as p2p memory * @pdev: the device to add the memory to @@ -198,6 +316,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->range.end = pgmap->range.start + size - 1; pgmap->nr_range = 1; pgmap->type = MEMORY_DEVICE_PCI_P2PDMA; + pgmap->ops = &p2pdma_pgmap_ops; p2p_pgmap->provider = pdev; p2p_pgmap->bus_offset = pci_bus_address(pdev, bar) - @@ -209,6 +328,11 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, goto pgmap_free; } + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_unmap_mappings, + pdev); + if (error) + goto pages_free; + p2pdma = rcu_dereference_protected(pdev->p2pdma, 1); error = gen_pool_add_owner(p2pdma->pool, (unsigned long)addr, pci_bus_address(pdev, bar) + offset, From patchwork Thu Aug 25 15:24:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12955039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 295D1ECAA25 for ; Thu, 25 Aug 2022 16:08:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DB34940008; Thu, 25 Aug 2022 12:08:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 789C5940007; Thu, 25 Aug 2022 12:08:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 652EE940008; Thu, 25 Aug 2022 12:08:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 570FA940007 for ; Thu, 25 Aug 2022 12:08:23 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 24DE91A0610 for ; Thu, 25 Aug 2022 16:08:23 +0000 (UTC) X-FDA: 79838597286.09.F555094 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf20.hostedemail.com (Postfix) with ESMTP id 8B58A1C0019 for ; Thu, 25 Aug 2022 16:08:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=Zco/cq2uLRvV+s+4HkHZAZF2wQKOR8FuVA8Uucgnt08=; b=ECzpmWpGfFN3FSN3MXglMySWFq 2/AyvUP5mc5dsTnuux+42Fb10+vKI4gPCs10aAmDZbYoZBwUmNwUGysjQ2BuYBQpsvZ9MWElt96CC bBjlqVLRj+av4OPvi1lgmo3s95qcB+a6q7ZpddFxRXCVbyQ00iLx/6NcZwRINyXwJUl/6x3FaBCFo J8pPdkOHtrhU+wGWGLJdb2zwmskLsaqFcwcodC4kLN7+M30fYzYMaIm8+MCZlyrayUO/+R1Znp3oy EiQoV2+fW6SxljxTyfuMnJ+vpiDzNUKchWev7FIQaywEwCZgXQU09yjQLzpHaoy4OsXgKoRj5NCXN Esl7IF9A==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oREik-0086M9-On; Thu, 25 Aug 2022 09:24:39 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1oREif-0001es-0Y; Thu, 25 Aug 2022 09:24:33 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates , Logan Gunthorpe Date: Thu, 25 Aug 2022 09:24:25 -0600 Message-Id: <20220825152425.6296-9-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220825152425.6296-1-logang@deltatee.com> References: <20220825152425.6296-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v9 8/8] ABI: sysfs-bus-pci: add documentation for p2pmem allocate X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661443702; a=rsa-sha256; cv=none; b=rnrKzEJq39foa9KBrRqcLjgWHdAZaNsYhj3Xmo0jNVORLjIc1impbog6XfQyuOXEWb8GTM b7SloensBqWP9xN8PGvxAQjU5iLeH02BXYzokNpRYVpPrIxfWNO+2mx0hsou6IHZKv0yUi P6SfLTRy5qHLfwzA7OH05TjCjg+13fo= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=ECzpmWpG; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf20.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661443702; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zco/cq2uLRvV+s+4HkHZAZF2wQKOR8FuVA8Uucgnt08=; b=u30LV9gNj1X+70Cv+DsYT0P3zKlqbe0NM9hD8sEG4mwByiPeMdL7nZQudRgT/9PzKyURDa vr7tCsXRsyQ8zjRLsalRsJSJqVRsUkLVwEOwyVh9Q1L3Eqh89CFvKcWSpztUpBg/ko67Uy PZZ0VHaot9Kg0JXacRghbLwLpb6KlVQ= X-Stat-Signature: ctooh3ng99sb79ngwwx1g8duouq4h51h X-Rspamd-Queue-Id: 8B58A1C0019 X-Rspam-User: X-Rspamd-Server: rspam11 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=ECzpmWpG; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf20.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-HE-Tag: 1661443699-428614 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add documentation for the p2pmem/allocate binary file which allows for allocating p2pmem buffers in userspace for passing to drivers that support them. (Currently only O_DIRECT to NVMe devices.) Signed-off-by: Logan Gunthorpe Reviewed-by: John Hubbard --- Documentation/ABI/testing/sysfs-bus-pci | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index 6fc2c2efe8ab..dca5e032b4fa 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci @@ -171,7 +171,7 @@ Description: binary file containing the Vital Product Data for the device. It should follow the VPD format defined in PCI Specification 2.1 or 2.2, but users should consider - that some devices may have incorrectly formatted data. + that some devices may have incorrectly formatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. @@ -407,6 +407,16 @@ Description: file contains a '1' if the memory has been published for use outside the driver that owns the device. +What: /sys/bus/pci/devices/.../p2pmem/allocate +Date: August 2022 +Contact: Logan Gunthorpe +Description: + Thes attribute allows mapping p2pmem into userspace. For + each mmap() call on this file, the kernel will allocate + a chunk of Peer-to-Peer memory for use in Peer-to-Peer + transactions. This memory can be used in O_DIRECT calls + to NVMe backed files for Peer-to-Peer copies. + What: /sys/bus/pci/devices/.../link/clkpm /sys/bus/pci/devices/.../link/l0s_aspm /sys/bus/pci/devices/.../link/l1_aspm