From patchwork Sun Oct 14 18:55:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 10640797 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51B1F17E3 for ; Sun, 14 Oct 2018 18:56:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 08C4B28B9F for ; Sun, 14 Oct 2018 18:56:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F13CE293CC; Sun, 14 Oct 2018 18:56:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 66D0828BEA for ; Sun, 14 Oct 2018 18:56:11 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id AB34E21FCF4; Sun, 14 Oct 2018 11:55:51 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp3.ccs.ornl.gov (smtp3.ccs.ornl.gov [160.91.203.39]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 570A621F50B for ; Sun, 14 Oct 2018 11:55:39 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp3.ccs.ornl.gov (Postfix) with ESMTP id E3EAF223E; Sun, 14 Oct 2018 14:55:34 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id E1DF02AC; Sun, 14 Oct 2018 14:55:34 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Sun, 14 Oct 2018 14:55:32 -0400 Message-Id: <1539543332-28679-11-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1539543332-28679-1-git-send-email-jsimmons@infradead.org> References: <1539543332-28679-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 10/10] lustre: lnet: make LNET_MAX_IOV dependent on page size X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Simmons , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP The default behavior of LNet is to always use 256 pages which is LNET_MAX_IOV and that LNET_MAX_PAYLOAD is always one megabyte. This assumes pages are always 4K in size which is not the case. This cause bulk I/O errors when using platforms like PowerPC or ARM which tend to use 64K pages. This is resolved by first making LNET_MAX_PAYLOAD always one megabyte since this is what the configuring sets it too by default and no one ever changes it. In theory it could set it to as high as 16MB but that will cause the I/O errors since the ptlrpc layer expects the packets to be always 1 megabyte in size. Also it would be better to make the maximum payload a per network setup configurations instead of for everything. Second we make LNET_MAX_IOV equal to LNET_MAX_PAYLOAD divided by the PAGE_SIZE. This way packets will always be the LNET_MAX_PAYLOAD in size but the number of pages used, LNET_MAX_IOV will vary depending on the platform it is creating packets on. Signed-off-by: James Simmons WC-bug-id: https://jira.whamcloud.com/browse/LU-10157 Reviewed-on: https://review.whamcloud.com/31559 Reviewed-by: Wang Shilong Reviewed-by: Ruth Klundt Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- drivers/staging/lustre/include/linux/lnet/lib-types.h | 10 ++++------ drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h | 3 --- drivers/staging/lustre/lnet/Kconfig | 10 ---------- drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c | 4 ++-- drivers/staging/lustre/lustre/include/lustre_net.h | 1 + 5 files changed, 7 insertions(+), 21 deletions(-) diff --git a/drivers/staging/lustre/include/linux/lnet/lib-types.h b/drivers/staging/lustre/include/linux/lnet/lib-types.h index 19f7b11..8951a53 100644 --- a/drivers/staging/lustre/include/linux/lnet/lib-types.h +++ b/drivers/staging/lustre/include/linux/lnet/lib-types.h @@ -37,6 +37,7 @@ #define __LNET_LIB_TYPES_H__ #include +#include #include #include #include @@ -46,12 +47,9 @@ #include /* Max payload size */ -#define LNET_MAX_PAYLOAD CONFIG_LNET_MAX_PAYLOAD -#if (LNET_MAX_PAYLOAD < LNET_MTU) -# error "LNET_MAX_PAYLOAD too small - error in configure --with-max-payload-mb" -#elif (LNET_MAX_PAYLOAD > (PAGE_SIZE * LNET_MAX_IOV)) -# error "LNET_MAX_PAYLOAD too large - error in configure --with-max-payload-mb" -#endif +#define LNET_MAX_PAYLOAD LNET_MTU + +#define LNET_MAX_IOV (LNET_MAX_PAYLOAD >> PAGE_SHIFT) /* forward refs */ struct lnet_libmd; diff --git a/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h b/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h index 1ecf18e..e440100 100644 --- a/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h +++ b/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h @@ -522,9 +522,6 @@ struct lnet_md { #define LNET_MTU_BITS 20 #define LNET_MTU (1 << LNET_MTU_BITS) -/** limit on the number of fragments in discontiguous MDs */ -#define LNET_MAX_IOV 256 - /** * Options for the MD structure. See lnet_md::options. */ diff --git a/drivers/staging/lustre/lnet/Kconfig b/drivers/staging/lustre/lnet/Kconfig index ad049e6..6062a82 100644 --- a/drivers/staging/lustre/lnet/Kconfig +++ b/drivers/staging/lustre/lnet/Kconfig @@ -8,16 +8,6 @@ config LNET case of Lustre routers only the LNet layer is required. Lately other projects are also looking into using LNet as their networking API as well. -config LNET_MAX_PAYLOAD - int "Lustre lnet max transfer payload (default 1MB)" - depends on LNET - default "1048576" - help - This option defines the maximum size of payload in bytes that lnet - can put into its transport. - - If unsure, use default. - config LNET_SELFTEST tristate "Lustre networking self testing" depends on LNET diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c index 66aa45f..68a8963 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c @@ -1414,7 +1414,7 @@ static void kiblnd_destroy_fmr_pool_list(struct list_head *head) static int kiblnd_alloc_fmr_pool(struct kib_fmr_poolset *fps, struct kib_fmr_pool *fpo) { struct ib_fmr_pool_param param = { - .max_pages_per_fmr = LNET_MAX_PAYLOAD / PAGE_SIZE, + .max_pages_per_fmr = LNET_MAX_IOV, .page_shift = PAGE_SHIFT, .access = (IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE), @@ -1474,7 +1474,7 @@ static int kiblnd_alloc_freg_pool(struct kib_fmr_poolset *fps, frd->frd_mr = ib_alloc_mr(fpo->fpo_hdev->ibh_pd, fastreg_gaps ? IB_MR_TYPE_SG_GAPS : IB_MR_TYPE_MEM_REG, - LNET_MAX_PAYLOAD / PAGE_SIZE); + LNET_MAX_IOV); if (IS_ERR(frd->frd_mr)) { rc = PTR_ERR(frd->frd_mr); CERROR("Failed to allocate ib_alloc_mr: %d\n", rc); diff --git a/drivers/staging/lustre/lustre/include/lustre_net.h b/drivers/staging/lustre/lustre/include/lustre_net.h index 2df72c7..ce7e98c 100644 --- a/drivers/staging/lustre/lustre/include/lustre_net.h +++ b/drivers/staging/lustre/lustre/include/lustre_net.h @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include