From patchwork Fri Feb 2 14:08:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Pen X-Patchwork-Id: 10196833 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 028E06037D for ; Fri, 2 Feb 2018 14:11:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E7BAB28E86 for ; Fri, 2 Feb 2018 14:11:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E66B228E8B; Fri, 2 Feb 2018 14:11:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B2C028E89 for ; Fri, 2 Feb 2018 14:11:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752068AbeBBOLP (ORCPT ); Fri, 2 Feb 2018 09:11:15 -0500 Received: from mail-wm0-f51.google.com ([74.125.82.51]:54658 "EHLO mail-wm0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752083AbeBBOK6 (ORCPT ); Fri, 2 Feb 2018 09:10:58 -0500 Received: by mail-wm0-f51.google.com with SMTP id i186so12855329wmi.4 for ; Fri, 02 Feb 2018 06:10:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Bj7P0J9nwg8JP7rQpVA8KflASW2rUaBiA95aqkEsZ60=; b=hZsmmMQGj/e0m2unCVuwLSIoMQR0NMKnVCjO3QZ/yqBhi6LFHRTlwK5IOEg2P/D3jX Wmk9gpaFk+HdowiUOtHRxDMFod4TAnRG81LzvmytozUcwf5WPehT9L24QzrJihbdC7wX cXWzh5DseXB+G4xIvAgU1LssXZpaMtKB3ulHGVmM503RgGRQnrrPwqe6h7epxSpAMpi9 1rM0aQGJ6EZojIbAirF+1Y89WyPIJIRn28tPTcW6XKqf58930hfw8AizlLSJKHneZahI C8sSNRAmcs3DFosj5ZZXdhIPCdGnHG/1vCBR78YHdCs+Io3sLqMsLErzmEUNVNNBhiWB vv0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Bj7P0J9nwg8JP7rQpVA8KflASW2rUaBiA95aqkEsZ60=; b=AOVyLQGYoKv27iFPblDz/z9v6jA+cnp5mtJXnOayAH6Zq4uMd+RGkv5nQ3ljbO+NYw j8IWIgPbvfr873NSEFKFaTQkJMVte9wAkJU+cASlsavvGSWjAlXkETU3qij3LNInZcV/ P0lMyX6grXAyE3NSesq0t+WWaaa/axtoiboCCmHCpJpg3B0l1yrS8GYRshQfGCpeUFOZ NI4yybEYPc7PXBhCz1YtvNKwikrRr7WiMm2VHfeVDX5tTjlPtMucJWcslPUHO15zXE/y 5bECxQPDdKA7vju9wvpuq96n+/vCNyWioaOkk9PG+4lCRddzM7ZMvwKRRxlgDNhAsXYB TSDg== X-Gm-Message-State: AKwxytcnrcAiHZlGHCU5zu3JLj4NI2p9fkijRvBGfWyRDvRoUFc5KPTT misawEnMblxVS6NB1zyJE5GsNqHe X-Google-Smtp-Source: AH8x227DmOsPIXSY1X+l6Bs0xNtEOkpNcZ6hTDwjUNAd9OBaiazk2pJ/yG8+7kgmktWEN8nR7YbtsA== X-Received: by 10.28.61.68 with SMTP id k65mr28116522wma.147.1517580656845; Fri, 02 Feb 2018 06:10:56 -0800 (PST) Received: from pb.pb.local ([62.217.45.26]) by smtp.gmail.com with ESMTPSA id v186sm798819wmf.17.2018.02.02.06.10.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Feb 2018 06:10:56 -0800 (PST) From: Roman Pen To: linux-block@vger.kernel.org, linux-rdma@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Sagi Grimberg , Bart Van Assche , Or Gerlitz , Roman Pen , Danil Kipnis , Jack Wang Subject: [PATCH 13/24] ibtrs: a bit of documentation Date: Fri, 2 Feb 2018 15:08:53 +0100 Message-Id: <20180202140904.2017-14-roman.penyaev@profitbricks.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20180202140904.2017-1-roman.penyaev@profitbricks.com> References: <20180202140904.2017-1-roman.penyaev@profitbricks.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP README with description of major sysfs entries. Signed-off-by: Roman Pen Signed-off-by: Danil Kipnis Cc: Jack Wang --- drivers/infiniband/ulp/ibtrs/README | 238 ++++++++++++++++++++++++++++++++++++ 1 file changed, 238 insertions(+) diff --git a/drivers/infiniband/ulp/ibtrs/README b/drivers/infiniband/ulp/ibtrs/README new file mode 100644 index 000000000000..ed506c7e202d --- /dev/null +++ b/drivers/infiniband/ulp/ibtrs/README @@ -0,0 +1,238 @@ +**************************** +InfiniBand Transport (IBTRS) +**************************** + +IBTRS (InfiniBand Transport) is a reliable high speed transport library +which provides support to establish optimal number of connections +between client and server machines using RDMA (InfiniBand, RoCE, iWarp) +transport. It is optimized to transfer (read/write) IO blocks. + +In its core interface it follows the BIO semantics of providing the +possibility to either write data from an sg list to the remote side +or to request ("read") data transfer from the remote side into a given +sg list. + +IBTRS provides I/O fail-over and load-balancing capabilities by using +multipath I/O (see "add_path" and "mp_policy" configuration entries). + +IBTRS is used by the IBNBD (Infiniband Network Block Device) modules. + +====================== +Client Sysfs Interface +====================== + +This chapter describes only the most important files of sysfs interface +on client side. + +Entries under /sys/kernel/ibtrs_client/ +======================================= + +When a user of IBTRS API creates a new session, a directory entry with +the name of that session is created. + +Entries under /sys/kernel/ibtrs_client// +====================================================== + +add_path (RW) +------------- + +Adds a new path (connection) to an existing session. Expected format is the +following: + + <[source addr,]destination addr> + + *addr ::= [ ip: | gid: ] + +max_reconnect_attempts (RW) +--------------------------- + +Maximum number reconnect attempts the client should make before giving up +after connection breaks unexpectedly. + +mp_policy (RW) +-------------- + +Multipath policy specifies which path should be selected on each IO: + + round-robin (0): + select path in per CPU round-robin manner. + + min-inflight (1): + select path with minimum inflights. + +Entries under /sys/kernel/ibtrs_client//paths/ +============================================================ + + +Each path belonging to a given session is listed here by its destination +address. When a new path is added to a session by writing to the "add_path" +entry, a directory with the corresponding destination address is created. + +Entries under /sys/kernel/ibtrs_client//paths// +======================================================================== + +state (R) +--------- + +Contains "connected" if the session is connected to the peer and fully +functional. Otherwise the file contains "disconnected" + +reconnect (RW) +-------------- + +Write "1" to the file in order to reconnect the path. +Operation is blocking and returns 0 if reconnect was successfull. + +disconnect (RW) +--------------- + +Write "1" to the file in order to disconnect the path. +Operation blocks until IBTRS path is disconnected. + +remove_path (RW) +---------------- + +Write "1" to the file in order to disconnected and remove the path +from the session. Operation blocks until the path is disconnected +and removed from the session. + +Entries under /sys/kernel/ibtrs_client//paths//stats/ +============================================================================== + +Write "0" to any file in that directory to reset corresponding statistics. + +reset_all (RW) +-------------- + +Read will return usage help, write 0 will clear all the statistics. + +sg_entries (RW) +--------------- + +Data to be transfered via RDMA is passed to IBTRS as scather-gather +list. A scather-gather list can contain multiple entries. +Scather-gather list with less entries require less processing power +and can therefore transfered faster. The file sg_entries outputs a +per-CPU distribution table for the number of entries in the +scather-gather lists, that were passed to the IBTRS API function +ibtrs_clt_request (READ or WRITE). + +cpu_migration (RW) +------------------ + +IBTRS expects that each HCA IRQ is pinned to a separate CPU. If it's +not the case, the processing of an I/O response could be processed on a +different CPU than where it was originally submitted. This file shows +how many interrupts where generated on a non expected CPU. +"from:" is the CPU on which the IRQ was expected, but not generated. +"to:" is the CPU on which the IRQ was generated, but not expected. + +reconnects (RW) +--------------- + +Contains 2 unsigned int values, the first one records number of successful +reconnects in the path lifetime, the second one records number of failed +reconnects in the path lifetime. + +rdma_lat (RW) +------------- + +Latency distribution of IBTRS requests. +The format is: + 1 ms: + 2 ms: + 4 ms: + 8 ms: + 16 ms: + ... + 65536 ms: + >= 65536 ms: + maximum ms: + +wc_completion (RW) +------------------ + +Contains 2 unsigned int values, the first one records max number of work +requests processed in work_completion in session lifetime, the second +one records average number of work requests processed in work_completion +in session lifetime. + +rdma (RW) +--------- + +Contains statistics regarding rdma operations and inflight operations. +The output consists of 6 values: + + \ + + +====================== +Server Sysfs Interface +====================== + +Entries under /sys/kernel/ibtrs_server/ +======================================= + +When a user of IBTRS API creates a new session on a client side, a +directory entry with the name of that session is created in here. + +Entries under /sys/kernel/ibtrs_server//paths/ +============================================================ + +When new path is created by writing to "add_path" entry on client side, +a directory entry with source address is created on server. + +Entries under /sys/kernel/ibtrs_server//paths// +========================================================================== + +disconnect (RW) +--------------- + +When "1" is written to the file, the IBTRS session is being disconnected. +Oprations is non-blocking and returns control immediately to the caller. + +hca_name (R) +------------ + +Contains the the name of HCA the connection established on. + +hca_port (R) +------------ + +Contains the port number of active port traffic is going through. + +Entries under /sys/kernel/ibtrs_server//paths//stats/ +================================================================================ + +When "0" is written to a file in this directory, the corresponding counters +will be reset. + +reset_all (RW) +-------------- + +Read will return usage help, write 0 will clear all the counters about +stats. + +rdma (RW) +--------- + +Contains statistics regarding rdma operations and inflight operations. +The output consists of 5 values: + + + +wc_completion (RW) +------------------ + +Contains 3 values, the first one is int, records max number of work +requests processed in work_completion in session lifetime, the second +one long int records total number of work requests processed in +work_completion in session lifetime and the 3rd one long int records +total number of calls to the cq completion handler. Devision of 2nd +number through 3rd gives the average number of completions processed +in completion handler. + +Contact +------- + +Mailing list: "IBNBD/IBTRS Storage Team"