From patchwork Wed Dec 5 13:24:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10714203 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E1C51923 for ; Wed, 5 Dec 2018 13:25:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1DC942CF32 for ; Wed, 5 Dec 2018 13:25:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 121D32CF33; Wed, 5 Dec 2018 13:25:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9382E2CF37 for ; Wed, 5 Dec 2018 13:25:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727505AbeLENZP (ORCPT ); Wed, 5 Dec 2018 08:25:15 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:36162 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727498AbeLENZP (ORCPT ); Wed, 5 Dec 2018 08:25:15 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 5 Dec 2018 15:31:11 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wB5DPAfM000838; Wed, 5 Dec 2018 15:25:10 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id wB5DPAdC011324; Wed, 5 Dec 2018 15:25:10 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id wB5DPALQ011323; Wed, 5 Dec 2018 15:25:10 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, monis@mellanox.com, guyle@mellanox.com, aviadye@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 6/6] verbs: Let caller pre-fetch a sub-region of ODP MR in rc_pingpong Date: Wed, 5 Dec 2018 15:24:51 +0200 Message-Id: <1544016291-10815-7-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1544016291-10815-1-git-send-email-yishaih@mellanox.com> References: <1544016291-10815-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Moni Shoua Add option (-P) to pre-fetch the memory that the application uses. This option is applicable only with ODP MR and it should prevent a page fault later on when the HCA tries to read or write from or to the memory. The pre-fetch is done once, just after creating the MR. Signed-off-by: Moni Shoua Reviewed-by: Guy Levi Signed-off-by: Yishai Hadas --- libibverbs/examples/rc_pingpong.c | 29 ++++++++++++++++++++++++++++- libibverbs/man/ibv_rc_pingpong.1 | 7 +++++-- 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/libibverbs/examples/rc_pingpong.c b/libibverbs/examples/rc_pingpong.c index d4c5df5..8b2253d 100644 --- a/libibverbs/examples/rc_pingpong.c +++ b/libibverbs/examples/rc_pingpong.c @@ -57,6 +57,7 @@ enum { static int page_size; static int use_odp; +static int prefetch_mr; static int use_ts; static int validate_buf; static int use_dm; @@ -430,6 +431,22 @@ static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size, goto clean_dm; } + if (prefetch_mr) { + struct ibv_sge sg_list; + int ret; + + sg_list.lkey = ctx->mr->lkey; + sg_list.addr = (uintptr_t)ctx->buf; + sg_list.length = size; + + ret = ibv_advise_mr(ctx->pd, IBV_ADVISE_MR_ADVICE_PREFETCH_WRITE, + IB_UVERBS_ADVISE_MR_FLAG_FLUSH, + &sg_list, 1); + + if (ret) + fprintf(stderr, "Couldn't prefetch MR(%d). Continue anyway\n", ret); + } + if (use_ts) { struct ibv_cq_init_attr_ex attr_ex = { .cqe = rx_depth + 1, @@ -717,6 +734,7 @@ static void usage(const char *argv0) printf(" -e, --events sleep on CQ events (default poll)\n"); printf(" -g, --gid-idx= local port gid index\n"); printf(" -o, --odp use on demand paging\n"); + printf(" -P, --prefetch prefetch an ODP MR\n"); printf(" -t, --ts get CQE with timestamp\n"); printf(" -c, --chk validate received buffer\n"); printf(" -j, --dm use device memory\n"); @@ -764,13 +782,14 @@ int main(int argc, char *argv[]) { .name = "events", .has_arg = 0, .val = 'e' }, { .name = "gid-idx", .has_arg = 1, .val = 'g' }, { .name = "odp", .has_arg = 0, .val = 'o' }, + { .name = "prefetch", .has_arg = 0, .val = 'P' }, { .name = "ts", .has_arg = 0, .val = 't' }, { .name = "chk", .has_arg = 0, .val = 'c' }, { .name = "dm", .has_arg = 0, .val = 'j' }, {} }; - c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:eg:otcj", + c = getopt_long(argc, argv, "p:d:i:s:m:r:n:l:eg:oPtcj", long_options, NULL); if (c == -1) @@ -832,6 +851,9 @@ int main(int argc, char *argv[]) case 'o': use_odp = 1; break; + case 'P': + prefetch_mr = 1; + break; case 't': use_ts = 1; break; @@ -861,6 +883,11 @@ int main(int argc, char *argv[]) return 1; } + if (!use_odp && prefetch_mr) { + fprintf(stderr, "prefetch is valid only with on-demand memory region\n"); + return 1; + } + if (use_ts) { ts.comp_recv_max_time_delta = 0; ts.comp_recv_min_time_delta = 0xffffffff; diff --git a/libibverbs/man/ibv_rc_pingpong.1 b/libibverbs/man/ibv_rc_pingpong.1 index ad5c834..5561fe5 100644 --- a/libibverbs/man/ibv_rc_pingpong.1 +++ b/libibverbs/man/ibv_rc_pingpong.1 @@ -8,12 +8,12 @@ ibv_rc_pingpong \- simple InfiniBand RC transport test .B ibv_rc_pingpong [\-p port] [\-d device] [\-i ib port] [\-s size] [\-m size] [\-r rx depth] [\-n iters] [\-l sl] [\-e] [\-g gid index] -[\-o] [\-t] \fBHOSTNAME\fR +[\-o] [\-P] [\-t] \fBHOSTNAME\fR .B ibv_rc_pingpong [\-p port] [\-d device] [\-i ib port] [\-s size] [\-m size] [\-r rx depth] [\-n iters] [\-l sl] [\-e] [\-g gid index] -[\-o] [\-t] +[\-o] [\-P] [\-t] .SH DESCRIPTION .PP @@ -58,6 +58,9 @@ local port \fIGIDINDEX\fR \fB\-o\fR, \fB\-\-odp\fR use on demand paging .TP +\fB\-P\fR, \fB\-\-prefetch=\fR +prefetch an ODP MR +.TP \fB\-t\fR, \fB\-\-ts\fR get CQE with timestamp .TP