From patchwork Mon Apr 14 17:01:06 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Ishchuk X-Patchwork-Id: 3985561 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 7BA6C9F336 for ; Mon, 14 Apr 2014 20:12:45 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 9E5BF201B9 for ; Mon, 14 Apr 2014 20:12:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 98A5D20179 for ; Mon, 14 Apr 2014 20:12:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754663AbaDNUMm (ORCPT ); Mon, 14 Apr 2014 16:12:42 -0400 Received: from e8.ny.us.ibm.com ([32.97.182.138]:34420 "EHLO e8.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754391AbaDNUMm (ORCPT ); Mon, 14 Apr 2014 16:12:42 -0400 Received: from /spool/local by e8.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 14 Apr 2014 16:12:36 -0400 Received: from d01dlp03.pok.ibm.com (9.56.250.168) by e8.ny.us.ibm.com (192.168.1.108) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Mon, 14 Apr 2014 16:10:14 -0400 Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 289FDC93361; Mon, 14 Apr 2014 12:58:24 -0400 (EDT) Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by b01cxnp23034.gho.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id s3EGwSIV6816148; Mon, 14 Apr 2014 16:58:28 GMT Received: from d01av04.pok.ibm.com (localhost [127.0.0.1]) by d01av04.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s3EGwRJI025456; Mon, 14 Apr 2014 12:58:28 -0400 Received: from drpkvmt5.endicott.ibm.com ([9.60.13.5]) by d01av04.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s3EGwMgI025130; Mon, 14 Apr 2014 12:58:27 -0400 From: Alexey Ishchuk To: linux-rdma@vger.kernel.org Cc: arlin.r.davis@intel.com, gilr@dev.mellanox.co.il, roland@kernel.org, linux-s390@vger.kernel.org, Alexey Ishchuk Subject: [PATCH 2/5] libibverbs: add new verb command support Date: Mon, 14 Apr 2014 19:01:06 +0200 Message-Id: <1397494869-43842-3-git-send-email-alexey_ishchuk@ru.ibm.com> X-Mailer: git-send-email 1.8.1.4 In-Reply-To: <1397494869-43842-1-git-send-email-alexey_ishchuk@ru.ibm.com> References: <1397494869-43842-1-git-send-email-alexey_ishchuk@ru.ibm.com> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14041420-0320-0000-0000-000002FF4518 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The current implementation of some Infiniband verbs uses mapped memory to directly access the PCI I/O memory from userspace programs to perform I/O operations. On the s390x platform the PCI I/O memory can be accessed only using special privileged CPU instructions and since those instructions are privileged, they cannot be used from userspace programs. This makes impossible to use mapped memory areas to access the PCI I/O memory from userspace on s390x platform and aplications cannot use the Infiniband verbs on that platform without modification. There are two approaches could be implemented to solve this problem: * using a page fault handler to intercept mapped memory area access errors, and handle them in the handler by issuing the appropriate privileged CPU instructions; * modification of the existing verbs to avoid the mapped memory areas usage on the s390x platform. The page fault handler solution is the most complex one because it requires not only modifcation of the virtual memory handling in Linux kernel but also makes the developer to provide code for all the CPU instrutions which work with memory program interpretation. This approcach requires lots of lines of code and noticable overhead during the program execution. The modification of the existing verbs solution is much simpler and more realible. It requires modification of the libraries provided in the DAPL support packages to replace the usage of mapped memory areas used to access the device UAR and Blueflame page with the device driver write primitive calls supplying a special verb command to kernelspace. The modification of the existing verbs solution has been choosen for implementation. Userspace library libibverbs contains platform independent part of the Infiniband verb implementation. To allow the privileged CPU instruction execution in kernel context on request from the userspace programs, this patch adds a new user verb command IB_USER_VERBS_CMD_KWRITE_MMIO and its parameter list structure, which is used to identify the PCI I/O operation request in the kernelspace driver. Also, it adds s390x platform specific definitions for the macro mb, rmb, wmb, and wc_wmb. Signed-off-by: Alexey Ishchuk --- include/infiniband/arch.h | 7 +++++++ include/infiniband/kern-abi.h | 6 ++++-- 2 files changed, 11 insertions(+), 2 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- a/include/infiniband/arch.h +++ b/include/infiniband/arch.h @@ -115,6 +115,13 @@ static inline uint64_t ntohll(uint64_t x #define wmb() mb() #define wc_wmb() wmb() +#elif defined(__s390x__) + +#define mb() { asm volatile("" : : : "memory"); } /* for s390x */ +#define rmb() mb() /* for s390x */ +#define wmb() mb() /* for s390x */ +#define wc_wmb() wmb() /* for s390x */ + #else #warning No architecture specific defines found. Using generic implementation. --- a/include/infiniband/kern-abi.h +++ b/include/infiniband/kern-abi.h @@ -91,7 +91,8 @@ enum { IB_USER_VERBS_CMD_OPEN_XRCD, IB_USER_VERBS_CMD_CLOSE_XRCD, IB_USER_VERBS_CMD_CREATE_XSRQ, - IB_USER_VERBS_CMD_OPEN_QP + IB_USER_VERBS_CMD_OPEN_QP, + IB_USER_VERBS_CMD_KWRITE_MMIO }; #define IB_USER_VERBS_CMD_COMMAND_MASK 0xff @@ -904,7 +905,8 @@ enum { IB_USER_VERBS_CMD_OPEN_XRCD_V2 = -1, IB_USER_VERBS_CMD_CLOSE_XRCD_V2 = -1, IB_USER_VERBS_CMD_CREATE_XSRQ_V2 = -1, - IB_USER_VERBS_CMD_OPEN_QP_V2 = -1 + IB_USER_VERBS_CMD_OPEN_QP_V2 = -1, + IB_USER_VERBS_CMD_KWRITE_MMIO_V2 = -1 }; struct ibv_modify_srq_v3 {