From patchwork Fri Mar 24 10:45:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 9642631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 641546020B for ; Fri, 24 Mar 2017 10:48:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7AEC327F07 for ; Fri, 24 Mar 2017 10:48:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FD5F2823D; Fri, 24 Mar 2017 10:48:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9CD6428236 for ; Fri, 24 Mar 2017 10:48:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935668AbdCXKsK (ORCPT ); Fri, 24 Mar 2017 06:48:10 -0400 Received: from mail-wr0-f177.google.com ([209.85.128.177]:35203 "EHLO mail-wr0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752325AbdCXKry (ORCPT ); Fri, 24 Mar 2017 06:47:54 -0400 Received: by mail-wr0-f177.google.com with SMTP id u1so6754279wra.2 for ; Fri, 24 Mar 2017 03:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wQQH3pF8vA7ZbhY8Lqw0VKsMqgZJOZGAyJHQ/hB0m+o=; b=IWwA0DmlSrxWUZpeHYtnM+ibIAGggRFnbhRIQoEJGMScv7HD4FO89VVpHuZpYpuv0i 2LwZv+cwME/UI10+gPQHeODAXORKyclmJ2QTiBqrAzZ0jT46nb5Io7GhswSnur+4coyF jdJT/rs09+/krVKpMcHT8wjuMGFOqVWCUD8m4SEea5R8UL+w6yqAPu0rk/q2F7tHeY9H YT1WyysM8J6f7X0S+uCyAuUiyV0i+7qj3/SphbSIx6DnmRXd8rPLiw8ZIs1r9Y/36kot 6TG2sn/ZVbi7pBfHmUJ52LYAl8OVn38ZVKHr2Uk7tkOGJGdjbsHlt41ty7UnAkfSgJJt 962A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wQQH3pF8vA7ZbhY8Lqw0VKsMqgZJOZGAyJHQ/hB0m+o=; b=MB/X5z41A/6ys/vhtoPM3AUvVPfPlL0pUeqRdYPTPt8utwz1OabsJa1DviZ0455h0L 0IDuZ1gX3tSiX1010L0ZzI0/qa3izOCn4Rw5GVaUFApKAcuu2oO5hsLT3GxSQYeO9ORZ OGNb/Zbor7Bp2YReD2CBGvoqcCaaQ2G2ZtUpouJkSqVCTwRK33GWrKA3nqyykGXBdzx6 5xnH3qXryXpxH1pEXMLkvTlgC/cAuAjjX+JlePH3/FGPp9sUR6t0F7OPbXByWEDdnh8g QAPKHVdXsfzoj4OZcudcXpuW1C0nWfwAEbX8y667y1XBbGKM0P34mkQzfHhoqQq4MVjE CO6A== X-Gm-Message-State: AFeK/H1weBe4CH6v3EWpwhbn1kiVH/EbOVv3Ch2+9LM3xoR21y9CkETkHzO6UmOYgYI2ymA8 X-Received: by 10.223.162.212 with SMTP id t20mr6736991wra.122.1490352467124; Fri, 24 Mar 2017 03:47:47 -0700 (PDT) Received: from jinpu-GA-870A-USB3.pb.local ([62.217.45.26]) by smtp.gmail.com with ESMTPSA id s17sm2404458wrc.25.2017.03.24.03.47.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 24 Mar 2017 03:47:46 -0700 (PDT) From: Jack Wang X-Google-Original-From: Jack Wang To: linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Cc: dledford@redhat.com, axboe@kernel.dk, hch@lst.de, mail@fholler.de, Milind.dumbare@gmail.com, yun.wang@profitbricks.com, Jack Wang , Kleber Souza , Danil Kipnis , Roman Pen Subject: [PATCH 17/28] ibnbd_clt: add header shared in ibnbd_client Date: Fri, 24 Mar 2017 11:45:32 +0100 Message-Id: <1490352343-20075-18-git-send-email-jinpu.wangl@profitbricks.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1490352343-20075-1-git-send-email-jinpu.wangl@profitbricks.com> References: <1490352343-20075-1-git-send-email-jinpu.wangl@profitbricks.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jack Wang Signed-off-by: Jack Wang Signed-off-by: Kleber Souza Signed-off-by: Danil Kipnis Signed-off-by: Roman Pen --- drivers/block/ibnbd_client/ibnbd_clt.h | 231 +++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 drivers/block/ibnbd_client/ibnbd_clt.h diff --git a/drivers/block/ibnbd_client/ibnbd_clt.h b/drivers/block/ibnbd_client/ibnbd_clt.h new file mode 100644 index 0000000..3f0db78 --- /dev/null +++ b/drivers/block/ibnbd_client/ibnbd_clt.h @@ -0,0 +1,231 @@ +/* + * InfiniBand Network Block Driver + * + * Copyright (c) 2014 - 2017 ProfitBricks GmbH. All rights reserved. + * Authors: Fabian Holler < mail@fholler.de> + * Jack Wang + * Kleber Souza + * Danil Kipnis + * Roman Pen + * Milind Dumbare + * + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions, and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * substantially similar to the "NO WARRANTY" disclaimer below + * ("Disclaimer") and any redistribution must be conditioned upon + * including a substantially similar Disclaimer requirement for further + * binary redistribution. + * 3. Neither the names of the above-listed copyright holders nor the names + * of any contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * Alternatively, this software may be distributed under the terms of the + * GNU General Public License ("GPL") version 2 as published by the Free + * Software Foundation. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING + * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGES. + * + */ + +#ifndef _IBNBD_CLT_H +#define _IBNBD_CLT_H +#include +#include /* for wait_queue_head_t */ +#include /* for sockaddr_in */ +#include /* for sockaddr_in */ +#include +#include "ibnbd_clt_log.h" +#include "../ibnbd_inc/ibnbd.h" +#include "../ibnbd_inc/ibnbd-proto.h" /* ibnbd protocol messages */ +#include /* for ibtrs api */ +#include + +#define IP_PREFIX "ip:" +#define IP_PREFIX_LEN strlen(IP_PREFIX) +#define GID_PREFIX "gid:" +#define GID_PREFIX_LEN strlen(GID_PREFIX) + +#define BMAX_SEGMENTS 31 +#define RECONNECT_DELAY 30 +#define MAX_RECONNECTS -1 + +enum ibnbd_dev_state { + DEV_STATE_INIT, + DEV_STATE_INIT_CLOSED, + DEV_STATE_CLOSED, + DEV_STATE_UNMAPPED, + DEV_STATE_OPEN +}; + +enum ibnbd_queue_mode { + BLK_MQ, + BLK_RQ +}; + +struct ibnbd_iu { + struct request *rq; + struct ibtrs_tag *tag; + struct ibnbd_dev *dev; + struct ibnbd_msg_io msg; + int errno; + struct scatterlist sglist[BMAX_SEGMENTS]; +}; + +struct ibnbd_cpu_qlist { + struct list_head requeue_list; + spinlock_t requeue_lock; + unsigned int cpu; +}; + +enum sess_state { + SESS_STATE_READY, + SESS_STATE_DISCONNECTED, + SESS_STATE_DESTROYED, +}; + +struct ibnbd_session { + struct list_head list; + struct ibtrs_session *sess; + struct ibnbd_cpu_qlist __percpu + *cpu_queues; + DECLARE_BITMAP(cpu_queues_bm, NR_CPUS); + int __percpu *cpu_rr; /* per-cpu var for CPU round-robin */ + atomic_t busy; + int queue_depth; + u32 max_io_size; + struct blk_mq_tag_set tag_set; + struct mutex lock; /* protects state and devs_list */ + struct list_head devs_list; /* list of struct ibnbd_dev */ + struct kref refcount; + struct sockaddr_storage addr; + char str_addr[IBTRS_ADDRLEN]; + char hostname[MAXHOSTNAMELEN]; + enum sess_state state; + u8 ver; /* protocol version */ + struct completion *sess_info_compl; +}; + +struct ibnbd_work { + struct work_struct work; + struct ibnbd_session *sess; +}; + +/** + * Submission queues. + */ +struct ibnbd_queue { + struct list_head requeue_list; + unsigned long in_list; + struct ibnbd_dev *dev; + struct blk_mq_hw_ctx *hctx; +}; + +struct ibnbd_dev { + struct list_head g_list; + struct ibnbd_session *sess; + struct request_queue *queue; + struct ibnbd_queue *hw_queues; + struct delayed_work rq_delay_work; + u32 device_id; + u32 clt_device_id; + struct completion *open_compl; /* completion for open msg */ + int open_errno; + struct mutex lock; + enum ibnbd_dev_state dev_state; + enum ibnbd_queue_mode queue_mode; + enum ibnbd_io_mode io_mode; /* user requested */ + enum ibnbd_io_mode remote_io_mode; /* server really used */ + /* local Idr index - used to track minor number allocations. */ + char pathname[NAME_MAX]; + enum ibnbd_access_mode access_mode; + bool read_only; + bool rotational; + u32 max_hw_sectors; + u32 max_write_same_sectors; + u32 max_discard_sectors; + u32 discard_zeroes_data; + u32 discard_granularity; + u32 discard_alignment; + u16 secure_discard; + u16 physical_block_size; + u16 logical_block_size; + u16 max_segments; + size_t nsectors; + u64 size; /* device size in bytes */ + struct list_head list; + struct gendisk *gd; + struct kobject kobj; + char blk_symlink_name[NAME_MAX]; + struct completion *close_compl; + atomic_t refcount; +}; + +static inline const char *ibnbd_queue_mode_str(enum ibnbd_queue_mode mode) +{ + switch (mode) { + case BLK_RQ: + return "rq"; + case BLK_MQ: + return "mq"; + default: + return "unknown"; + } +} + +int ibnbd_close_device(struct ibnbd_dev *dev, bool force); +struct ibnbd_session *ibnbd_create_session(const struct sockaddr_storage *addr); +struct ibnbd_session *ibnbd_clt_find_sess(const struct sockaddr_storage *addr); +void ibnbd_clt_sess_release(struct kref *ref); +struct ibnbd_dev *ibnbd_client_add_device(struct ibnbd_session *sess, + const char *pathname, + enum ibnbd_access_mode access_mode, + enum ibnbd_queue_mode queue_mode, + enum ibnbd_io_mode io_mode); +void ibnbd_destroy_gen_disk(struct ibnbd_dev *dev); +int ibnbd_addr_to_str(const struct sockaddr_storage *addr, + char *buf, size_t len); +bool ibnbd_clt_dev_is_open(struct ibnbd_dev *dev); +bool ibnbd_clt_dev_is_mapped(const char *pathname); +int open_remote_device(struct ibnbd_dev *dev); + +const char *ibnbd_clt_get_io_mode(const struct ibnbd_dev *dev); + +#define ERR_DEVS(sess, fmt, ...) \ +({ struct ibnbd_dev *dev; \ + \ + mutex_lock(&sess->lock); \ + list_for_each_entry(dev, &sess->devs_list, list) \ + pr_err("ibnbd L%d <%s@%s> ERR:" fmt, \ + __LINE__, dev->pathname, dev->sess->str_addr,\ + ##__VA_ARGS__); \ + mutex_unlock(&sess->lock); \ +}) + +#define INFO_DEVS(sess, fmt, ...) \ +({ struct ibnbd_dev *dev; \ + \ + mutex_lock(&sess->lock); \ + list_for_each_entry(dev, &sess->devs_list, list) \ + pr_info("ibnbd <%s@%s> ERR:" fmt, \ + dev->pathname, dev->sess->str_addr,\ + ##__VA_ARGS__); \ + mutex_unlock(&sess->lock); \ +}) +#endif /* _IBNBD_CLT_H */