From patchwork Fri Mar 24 10:45:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 9642575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9D23E6020B for ; Fri, 24 Mar 2017 10:47:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B411926E81 for ; Fri, 24 Mar 2017 10:47:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A916528236; Fri, 24 Mar 2017 10:47:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16E3B26E81 for ; Fri, 24 Mar 2017 10:47:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935660AbdCXKrl (ORCPT ); Fri, 24 Mar 2017 06:47:41 -0400 Received: from mail-wm0-f52.google.com ([74.125.82.52]:37435 "EHLO mail-wm0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751714AbdCXKri (ORCPT ); Fri, 24 Mar 2017 06:47:38 -0400 Received: by mail-wm0-f52.google.com with SMTP id n11so9892417wma.0 for ; Fri, 24 Mar 2017 03:47:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1GbOFbIi2BJpn5goZmfrSE69FnaOfD87fGHo/THzkII=; b=epoTOGlQi+kZIj+vSpMn5+IjJwNGRDUjTjhh4HlPhHiIXHlP6l9MlJXNz89ajxiWj0 7WhJPBD/5DFfV9ANKXld3Yn4DlE5HSLXxoSwTd6RYi8X7+IFP4KdbH/RQE6LtS1EM8H9 nqekozXtjHE8tq3SCtgKwgRtfIlWh92oJn9BCNlxewMR2cNCDs9hWABdFJmpPlbkLBp8 bFMkwsEC4N/EP9WnaqFbQkkzjtgnhI2kRNyqe840aFxUNgOLZS6LZTCOzp8yD0MEwY2S wfHveMYKKR/8cwTxEMrdKyFdO7z/+LUJK9TE8J9skhRghTMQdyuWB9zz0Y5xjRvOYZeD Wkjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1GbOFbIi2BJpn5goZmfrSE69FnaOfD87fGHo/THzkII=; b=q3qTEpV7qJsm/g1tfTxc8MExAuWAcHwngSNnmY61Zoh7+J1ZnW0pklaLmS9QeHm5FA TegxY5LHvos1MVj5UA8PmRMxVYWbJTut8HEPmFczmcCyGuAsnWn7XV9IGbWFgGlGSMmF D3UvkRZ8nshxNGxYWCL+XP9UwsGKYNco8huugWiyFmN0f4IQe7tc2kZTLd/2t2WnjF1u 6ExQQGuvzRyrBQdktTUOp9qXrjxUfGI86B+aype7+DkZO6jDkx7iRYuQ0RkmT/B+nSt8 LgCve5UD2Na7A59ufGzzMCedPa+kRkzCPWv/mR7R6CrgxcaGjymwTgWKmHZoO0QbUW3g n02w== X-Gm-Message-State: AFeK/H2/Krc8kvM0CB0wLZu7UQ2ALgRR1l2PA6XagKc+lNuL5SwZJkv2PdhoQ4EcsKPYs4lM X-Received: by 10.28.29.138 with SMTP id d132mr2185182wmd.40.1490352456300; Fri, 24 Mar 2017 03:47:36 -0700 (PDT) Received: from jinpu-GA-870A-USB3.pb.local ([62.217.45.26]) by smtp.gmail.com with ESMTPSA id s17sm2404458wrc.25.2017.03.24.03.47.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 24 Mar 2017 03:47:35 -0700 (PDT) From: Jack Wang X-Google-Original-From: Jack Wang To: linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Cc: dledford@redhat.com, axboe@kernel.dk, hch@lst.de, mail@fholler.de, Milind.dumbare@gmail.com, yun.wang@profitbricks.com, Jack Wang , Kleber Souza , Danil Kipnis , Roman Pen Subject: [PATCH 06/28] ibtrs_clt: add header file shared only in ibtrs_client Date: Fri, 24 Mar 2017 11:45:21 +0100 Message-Id: <1490352343-20075-7-git-send-email-jinpu.wangl@profitbricks.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1490352343-20075-1-git-send-email-jinpu.wangl@profitbricks.com> References: <1490352343-20075-1-git-send-email-jinpu.wangl@profitbricks.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jack Wang Signed-off-by: Jack Wang Signed-off-by: Kleber Souza Signed-off-by: Danil Kipnis Signed-off-by: Roman Pen --- .../ulp/ibtrs_client/ibtrs_clt_internal.h | 244 +++++++++++++++++++++ 1 file changed, 244 insertions(+) create mode 100644 drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_internal.h diff --git a/drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_internal.h b/drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_internal.h new file mode 100644 index 0000000..7274b2d --- /dev/null +++ b/drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_internal.h @@ -0,0 +1,244 @@ +/* + * InfiniBand Transport Layer + * + * Copyright (c) 2014 - 2017 ProfitBricks GmbH. All rights reserved. + * Authors: Fabian Holler < mail@fholler.de> + * Jack Wang + * Kleber Souza + * Danil Kipnis + * Roman Pen + * Milind Dumbare + * + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions, and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * substantially similar to the "NO WARRANTY" disclaimer below + * ("Disclaimer") and any redistribution must be conditioned upon + * including a substantially similar Disclaimer requirement for further + * binary redistribution. + * 3. Neither the names of the above-listed copyright holders nor the names + * of any contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * Alternatively, this software may be distributed under the terms of the + * GNU General Public License ("GPL") version 2 as published by the Free + * Software Foundation. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING + * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGES. + * + */ + +#if !defined(IBTRS_CLT_INTERNAL_H) +#define IBTRS_CLT_INTERNAL_H + +#include + +enum ssm_state { + _SSM_STATE_MIN, + SSM_STATE_IDLE, + SSM_STATE_IDLE_RECONNECT, + SSM_STATE_WF_INFO, + SSM_STATE_WF_INFO_RECONNECT, + SSM_STATE_OPEN, + SSM_STATE_OPEN_RECONNECT, + SSM_STATE_CONNECTED, + SSM_STATE_RECONNECT, + SSM_STATE_RECONNECT_IMM, + SSM_STATE_CLOSE_DESTROY, + SSM_STATE_CLOSE_RECONNECT, + SSM_STATE_CLOSE_RECONNECT_IMM, + SSM_STATE_DISCONNECTED, + SSM_STATE_DESTROYED, + _SSM_STATE_MAX +}; + +enum ibtrs_fast_reg { + IBTRS_FAST_MEM_NONE, + IBTRS_FAST_MEM_FR, + IBTRS_FAST_MEM_FMR +}; + +struct ibtrs_stats_reconnects { + u32 successful_cnt; + u32 fail_cnt; +}; + +struct ibtrs_stats_wc_comp { + u32 max_wc_cnt; + u32 cnt; + u64 total_cnt; +}; + +struct ibtrs_stats_cpu_migration { + atomic_t *from; + int *to; +}; + +struct ibtrs_clt_stats_rdma_stats { + u64 cnt_read; + u64 size_total_read; + u64 cnt_write; + u64 size_total_write; + + u16 inflight; +}; + +#define MIN_LOG_SG 2 +#define MAX_LOG_SG 5 +#define MAX_LIN_SG BIT(MIN_LOG_SG) +#define SG_DISTR_LEN (MAX_LOG_SG - MIN_LOG_SG + MAX_LIN_SG + 1) + +struct ibtrs_clt_stats_rdma_lat_entry { + u64 read; + u64 write; +}; + +#define MAX_LOG_LATENCY 16 +#define MIN_LOG_LATENCY 0 + +struct ibtrs_clt_stats_user_ib_msgs { + u32 recv_msg_cnt; + u32 sent_msg_cnt; + u64 recv_size; + u64 sent_size; +}; + +struct ibtrs_clt_stats { + struct ibtrs_stats_cpu_migration cpu_migr; + struct ibtrs_clt_stats_rdma_stats *rdma_stats; + u64 *sg_list_total; + u64 **sg_list_distr; + struct ibtrs_stats_reconnects reconnects; + struct ibtrs_clt_stats_rdma_lat_entry **rdma_lat_distr; + struct ibtrs_clt_stats_rdma_lat_entry *rdma_lat_max; + struct ibtrs_clt_stats_user_ib_msgs user_ib_msgs; + struct ibtrs_stats_wc_comp *wc_comp; +}; + +struct ibtrs_session { + struct list_head list; /* global session list */ + wait_queue_head_t wait_q; + enum ssm_state state; + struct ibtrs_con *con; + struct ib_session ib_sess; + struct ib_device *ib_device; + struct ibtrs_iu *rdma_info_iu; + struct ibtrs_iu *sess_info_iu; + struct ibtrs_iu *dummy_rx_iu; + struct ibtrs_iu **usr_rx_ring; + struct ibtrs_iu **io_tx_ius; + + spinlock_t u_msg_ius_lock ____cacheline_aligned; + struct list_head u_msg_ius_list; + + struct rdma_req *reqs; + struct ib_fmr_pool *fmr_pool; + atomic_t ib_sess_initialized; + bool io_bufs_initialized; + size_t pdu_sz; + void *priv; + struct workqueue_struct *sm_wq; + struct workqueue_struct *msg_wq; + struct delayed_work heartbeat_dwork; + u32 heartbeat_timeout_srv_ms; + struct delayed_work reconnect_dwork; + struct ibtrs_heartbeat heartbeat; + atomic_t refcount; + u8 active_cnt; + bool enable_rdma_lat; + u8 ver; + u8 connected_cnt; + u32 retry_cnt; + s16 max_reconnect_attempts; + u8 reconnect_delay_sec; + void *tags; + unsigned long *tags_map; + wait_queue_head_t tags_wait; + u64 *srv_rdma_addr; + u32 srv_rdma_buf_rkey; + u32 max_io_size; + u32 max_req_size; + u32 chunk_size; + u32 max_desc; + u32 queue_depth; + u16 user_queue_depth; + enum ibtrs_fast_reg fast_reg_mode; + u64 mr_page_mask; + u32 mr_page_size; + u32 mr_max_size; + u32 max_pages_per_mr; + int max_sge; + struct sockaddr_storage peer_addr; + struct sockaddr_storage self_addr; + struct completion *destroy_completion; + struct kobject kobj; + struct kobject kobj_stats; + char addr[IBTRS_ADDRLEN]; + char hostname[MAXHOSTNAMELEN]; + struct ibtrs_clt_stats stats; + wait_queue_head_t mu_iu_wait_q; + wait_queue_head_t mu_buf_wait_q; + atomic_t peer_usr_msg_bufs; + struct completion *ib_sess_destroy_completion; +}; + +#define TAG_SIZE(sess) (sizeof(struct ibtrs_tag) + (sess)->pdu_sz) +#define GET_TAG(sess, idx) ((sess)->tags + TAG_SIZE(sess) * idx) + +/** + * ibtrs_clt_reconnect() - Reconnect the session + * @sess: Session handler + */ +int ibtrs_clt_reconnect(struct ibtrs_session *sess); + +void ibtrs_clt_set_max_reconnect_attempts(struct ibtrs_session *sess, + s16 value); + +s16 ibtrs_clt_get_max_reconnect_attempts(const struct ibtrs_session *sess); +int ibtrs_clt_get_user_queue_depth(struct ibtrs_session *sess); +int ibtrs_clt_set_user_queue_depth(struct ibtrs_session *sess, u16 queue_depth); +int ibtrs_clt_reset_sg_list_distr_stats(struct ibtrs_session *sess, + bool enable); +int ibtrs_clt_stats_sg_list_distr_to_str(struct ibtrs_session *sess, + char *buf, size_t len); +int ibtrs_clt_reset_rdma_lat_distr_stats(struct ibtrs_session *sess, + bool enable); +ssize_t ibtrs_clt_stats_rdma_lat_distr_to_str(struct ibtrs_session *sess, + char *page, size_t len); +int ibtrs_clt_reset_cpu_migr_stats(struct ibtrs_session *sess, bool enable); +int ibtrs_clt_stats_migration_cnt_to_str(struct ibtrs_session *sess, char *buf, + size_t len); +int ibtrs_clt_reset_reconnects_stat(struct ibtrs_session *sess, bool enable); +int ibtrs_clt_stats_reconnects_to_str(struct ibtrs_session *sess, char *buf, + size_t len); +int ibtrs_clt_reset_user_ib_msgs_stats(struct ibtrs_session *sess, bool enable); +int ibtrs_clt_stats_user_ib_msgs_to_str(struct ibtrs_session *sess, char *buf, + size_t len); +int ibtrs_clt_reset_wc_comp_stats(struct ibtrs_session *sess, bool enable); +int ibtrs_clt_stats_wc_completion_to_str(struct ibtrs_session *sess, char *buf, + size_t len); +int ibtrs_clt_reset_rdma_stats(struct ibtrs_session *sess, bool enable); +ssize_t ibtrs_clt_stats_rdma_to_str(struct ibtrs_session *sess, + char *page, size_t len); +bool ibtrs_clt_sess_is_connected(const struct ibtrs_session *sess); +int ibtrs_clt_reset_all_stats(struct ibtrs_session *sess, bool enable); +ssize_t ibtrs_clt_reset_all_help(struct ibtrs_session *sess, + char *page, size_t len); + +#endif