From patchwork Wed Jan 19 04:27:21 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Dillow X-Patchwork-Id: 488161 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter1.kernel.org (8.14.4/8.14.3) with ESMTP id p0J50ubT026308 for ; Wed, 19 Jan 2011 05:01:15 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750872Ab1ASFBN (ORCPT ); Wed, 19 Jan 2011 00:01:13 -0500 Received: from emroute4.ornl.gov ([160.91.86.27]:64328 "EHLO emroute4.ornl.gov" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750887Ab1ASFBM (ORCPT ); Wed, 19 Jan 2011 00:01:12 -0500 Received: from emroute4.ornl.gov ([127.0.0.1]) by emroute4.ornl.gov (PMDF V6.5-x5 #31823) with ESMTP id <0LF900BAV5PRBU@emroute4.ornl.gov> for linux-rdma@vger.kernel.org; Tue, 18 Jan 2011 23:27:27 -0500 (EST) Received: from CONVERSION-DAEMON.emroute4.ornl.gov by emroute4.ornl.gov (PMDF V6.5-x5 #31823) id <0LF900C015PRXB@emroute4.ornl.gov> for linux-rdma@vger.kernel.org; Tue, 18 Jan 2011 23:27:27 -0500 (EST) Received: from lap75545.ornl.gov (lap75545.ornl.gov [160.91.210.54]) by emroute4.ornl.gov (PMDF V6.5-x5 #31823) with ESMTPS id <0LF900B825PRBL@emroute4.ornl.gov> for linux-rdma@vger.kernel.org; Tue, 18 Jan 2011 23:27:27 -0500 (EST) Received: from lap75545.ornl.gov (localhost.localdomain [127.0.0.1]) by lap75545.ornl.gov (8.14.4/8.14.3) with ESMTP id p0J4RQjW026226 for ; Tue, 18 Jan 2011 23:27:26 -0500 Received: (from dad@localhost) by lap75545.ornl.gov (8.14.4/8.14.4/Submit) id p0J4RQuP026223 for linux-rdma@vger.kernel.org; Tue, 18 Jan 2011 23:27:26 -0500 Date: Tue, 18 Jan 2011 23:27:21 -0500 From: David Dillow Subject: [RFC 7/8] IB/srp: try to use larger FMR sizes to cover our mappings In-reply-to: <1295411242-26148-1-git-send-email-dillowda@ornl.gov> To: linux-rdma@vger.kernel.org Message-id: <1295411242-26148-8-git-send-email-dillowda@ornl.gov> X-Mailer: git-send-email 1.7.3.4 References: <1295411242-26148-1-git-send-email-dillowda@ornl.gov> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter1.kernel.org [140.211.167.41]); Wed, 19 Jan 2011 05:01:15 +0000 (UTC) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 0121530..1da8b25 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -2300,7 +2300,7 @@ static void srp_add_one(struct ib_device *device) struct ib_device_attr *dev_attr; struct ib_fmr_pool_param fmr_param; struct srp_host *host; - int fmr_page_shift, s, e, p; + int max_pages_per_fmr, fmr_page_shift, s, e, p; dev_attr = kmalloc(sizeof *dev_attr, GFP_KERNEL); if (!dev_attr) @@ -2340,17 +2340,24 @@ static void srp_add_one(struct ib_device *device) if (IS_ERR(srp_dev->mr)) goto err_pd; - memset(&fmr_param, 0, sizeof fmr_param); - fmr_param.pool_size = SRP_FMR_POOL_SIZE; - fmr_param.dirty_watermark = SRP_FMR_DIRTY_SIZE; - fmr_param.cache = 1; - fmr_param.max_pages_per_fmr = SRP_FMR_SIZE; - fmr_param.page_shift = fmr_page_shift; - fmr_param.access = (IB_ACCESS_LOCAL_WRITE | - IB_ACCESS_REMOTE_WRITE | - IB_ACCESS_REMOTE_READ); - - srp_dev->fmr_pool = ib_create_fmr_pool(srp_dev->pd, &fmr_param); + for (max_pages_per_fmr = SRP_FMR_SIZE; + max_pages_per_fmr >= SRP_FMR_MIN_SIZE; + max_pages_per_fmr /= 2, srp_dev->fmr_max_size /= 2) { + memset(&fmr_param, 0, sizeof fmr_param); + fmr_param.pool_size = SRP_FMR_POOL_SIZE; + fmr_param.dirty_watermark = SRP_FMR_DIRTY_SIZE; + fmr_param.cache = 1; + fmr_param.max_pages_per_fmr = SRP_FMR_SIZE; + fmr_param.page_shift = fmr_page_shift; + fmr_param.access = (IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_WRITE | + IB_ACCESS_REMOTE_READ); + + srp_dev->fmr_pool = ib_create_fmr_pool(srp_dev->pd, &fmr_param); + if (!IS_ERR(srp_dev->fmr_pool)) + break; + } + if (IS_ERR(srp_dev->fmr_pool)) srp_dev->fmr_pool = NULL; diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h index cf69621..020caf0 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.h +++ b/drivers/infiniband/ulp/srp/ib_srp.h @@ -69,7 +69,8 @@ enum { SRP_TAG_NO_REQ = ~0U, SRP_TAG_TSK_MGMT = 1U << 31, - SRP_FMR_SIZE = 256, + SRP_FMR_SIZE = 512, + SRP_FMR_MIN_SIZE = 128, SRP_FMR_POOL_SIZE = 1024, SRP_FMR_DIRTY_SIZE = SRP_FMR_POOL_SIZE / 4,