From patchwork Tue Dec 8 22:10:09 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 7802591 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 3F8369F387 for ; Tue, 8 Dec 2015 22:10:24 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 47F2B202F0 for ; Tue, 8 Dec 2015 22:10:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 161AC203F3 for ; Tue, 8 Dec 2015 22:10:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751098AbbLHWKV (ORCPT ); Tue, 8 Dec 2015 17:10:21 -0500 Received: from mga03.intel.com ([134.134.136.65]:26563 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750843AbbLHWKU (ORCPT ); Tue, 8 Dec 2015 17:10:20 -0500 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP; 08 Dec 2015 14:10:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,400,1444719600"; d="scan'208";a="9829441" Received: from phlsvsds.ph.intel.com ([10.228.195.38]) by fmsmga004.fm.intel.com with ESMTP; 08 Dec 2015 14:10:20 -0800 Received: from phlsvsds.ph.intel.com (localhost.localdomain [127.0.0.1]) by phlsvsds.ph.intel.com (8.13.8/8.13.8) with ESMTP id tB8MAIwe006937; Tue, 8 Dec 2015 17:10:18 -0500 Received: (from iweiny@localhost) by phlsvsds.ph.intel.com (8.13.8/8.13.8/Submit) id tB8MAHVn006934; Tue, 8 Dec 2015 17:10:17 -0500 X-Authentication-Warning: phlsvsds.ph.intel.com: iweiny set sender to ira.weiny@intel.com using -f From: ira.weiny@intel.com To: gregkh@linuxfoundation.org, devel@driverdev.osuosl.org Cc: dledford@redhat.com, linux-rdma@vger.kernel.org, Mitko Haralanov Subject: [PATCH v2 1/5] staging/rdma/hfi1: Convert to use get_user_pages_fast Date: Tue, 8 Dec 2015 17:10:09 -0500 Message-Id: <1449612613-6616-2-git-send-email-ira.weiny@intel.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1449612613-6616-1-git-send-email-ira.weiny@intel.com> References: <1449612613-6616-1-git-send-email-ira.weiny@intel.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Mitko Haralanov Convert hfi1_get_user_pages() to use get_user_pages_fast(), which is much fatster. The mm semaphore is still taken to update the pinned page count but is for a much shorter amount of time. Reviewed-by: Ira Weiny Signed-off-by: Mitko Haralanov --- drivers/staging/rdma/hfi1/file_ops.c | 8 +-- drivers/staging/rdma/hfi1/hfi.h | 4 +- drivers/staging/rdma/hfi1/user_pages.c | 97 +++++++++------------------------- 3 files changed, 32 insertions(+), 77 deletions(-) diff --git a/drivers/staging/rdma/hfi1/file_ops.c b/drivers/staging/rdma/hfi1/file_ops.c index 22037ce984c8..76580030f514 100644 --- a/drivers/staging/rdma/hfi1/file_ops.c +++ b/drivers/staging/rdma/hfi1/file_ops.c @@ -1670,8 +1670,8 @@ static int exp_tid_setup(struct file *fp, struct hfi1_tid_info *tinfo) * Now that we know how many free RcvArray entries we have, * we can pin that many user pages. */ - ret = hfi1_get_user_pages(vaddr + (mapped * PAGE_SIZE), - pinned, pages); + ret = hfi1_acquire_user_pages(vaddr + (mapped * PAGE_SIZE), + pinned, true, pages); if (ret) { /* * We can't continue because the pages array won't be @@ -1840,7 +1840,7 @@ static int exp_tid_free(struct file *fp, struct hfi1_tid_info *tinfo) } } flush_wc(); - hfi1_release_user_pages(pshadow, pcount); + hfi1_release_user_pages(pshadow, pcount, true); clear_bit(bitidx, &uctxt->tidusemap[idx]); map &= ~(1ULL<physshadow[tid] = 0; uctxt->tid_pg_list[tid] = NULL; pci_unmap_page(dd->pcidev, phys, PAGE_SIZE, PCI_DMA_FROMDEVICE); - hfi1_release_user_pages(&p, 1); + hfi1_release_user_pages(&p, 1, true); } } diff --git a/drivers/staging/rdma/hfi1/hfi.h b/drivers/staging/rdma/hfi1/hfi.h index 54ed6b36c1a7..467e1a9f8ed2 100644 --- a/drivers/staging/rdma/hfi1/hfi.h +++ b/drivers/staging/rdma/hfi1/hfi.h @@ -1551,8 +1551,8 @@ void hfi1_set_led_override(struct hfi1_pportdata *ppd, unsigned int val); */ #define DEFAULT_RCVHDR_ENTSIZE 32 -int hfi1_get_user_pages(unsigned long, size_t, struct page **); -void hfi1_release_user_pages(struct page **, size_t); +int hfi1_acquire_user_pages(unsigned long, size_t, bool, struct page **); +void hfi1_release_user_pages(struct page **, size_t, bool); static inline void clear_rcvhdrtail(const struct hfi1_ctxtdata *rcd) { diff --git a/drivers/staging/rdma/hfi1/user_pages.c b/drivers/staging/rdma/hfi1/user_pages.c index 9071afbd7bf4..692de658f0dc 100644 --- a/drivers/staging/rdma/hfi1/user_pages.c +++ b/drivers/staging/rdma/hfi1/user_pages.c @@ -49,59 +49,11 @@ */ #include +#include #include #include "hfi.h" -static void __hfi1_release_user_pages(struct page **p, size_t num_pages, - int dirty) -{ - size_t i; - - for (i = 0; i < num_pages; i++) { - if (dirty) - set_page_dirty_lock(p[i]); - put_page(p[i]); - } -} - -/* - * Call with current->mm->mmap_sem held. - */ -static int __hfi1_get_user_pages(unsigned long start_page, size_t num_pages, - struct page **p) -{ - unsigned long lock_limit; - size_t got; - int ret; - - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - if (num_pages > lock_limit && !capable(CAP_IPC_LOCK)) { - ret = -ENOMEM; - goto bail; - } - - for (got = 0; got < num_pages; got += ret) { - ret = get_user_pages(current, current->mm, - start_page + got * PAGE_SIZE, - num_pages - got, 1, 1, - p + got, NULL); - if (ret < 0) - goto bail_release; - } - - current->mm->pinned_vm += num_pages; - - ret = 0; - goto bail; - -bail_release: - __hfi1_release_user_pages(p, got, 0); -bail: - return ret; -} - /** * hfi1_map_page - a safety wrapper around pci_map_page() * @@ -116,41 +68,44 @@ dma_addr_t hfi1_map_page(struct pci_dev *hwdev, struct page *page, return phys; } -/** - * hfi1_get_user_pages - lock user pages into memory - * @start_page: the start page - * @num_pages: the number of pages - * @p: the output page structures - * - * This function takes a given start page (page aligned user virtual - * address) and pins it and the following specified number of pages. For - * now, num_pages is always 1, but that will probably change at some point - * (because caller is doing expected sends on a single virtually contiguous - * buffer, so we can do all pages at once). - */ -int hfi1_get_user_pages(unsigned long start_page, size_t num_pages, - struct page **p) +int hfi1_acquire_user_pages(unsigned long vaddr, size_t npages, bool writable, + struct page **pages) { + unsigned long pinned, lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; + bool can_lock = capable(CAP_IPC_LOCK); int ret; - down_write(¤t->mm->mmap_sem); + down_read(¤t->mm->mmap_sem); + pinned = current->mm->pinned_vm; + up_read(¤t->mm->mmap_sem); - ret = __hfi1_get_user_pages(start_page, num_pages, p); + if (pinned + npages > lock_limit && !can_lock) + return -ENOMEM; + ret = get_user_pages_fast(vaddr, npages, writable, pages); + if (ret < 0) + return ret; + + down_write(¤t->mm->mmap_sem); + current->mm->pinned_vm += ret; up_write(¤t->mm->mmap_sem); return ret; } -void hfi1_release_user_pages(struct page **p, size_t num_pages) +void hfi1_release_user_pages(struct page **p, size_t npages, bool dirty) { - if (current->mm) /* during close after signal, mm can be NULL */ - down_write(¤t->mm->mmap_sem); + size_t i; - __hfi1_release_user_pages(p, num_pages, 1); + for (i = 0; i < npages; i++) { + if (dirty) + set_page_dirty_lock(p[i]); + put_page(p[i]); + } - if (current->mm) { - current->mm->pinned_vm -= num_pages; + if (current->mm) { /* during close after signal, mm can be NULL */ + down_write(¤t->mm->mmap_sem); + current->mm->pinned_vm -= npages; up_write(¤t->mm->mmap_sem); } }