From patchwork Wed Jun 5 23:24:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 10978041 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1760613AD for ; Wed, 5 Jun 2019 23:25:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09E1F283C3 for ; Wed, 5 Jun 2019 23:25:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F2C0C28A15; Wed, 5 Jun 2019 23:25:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1ECF3283C3 for ; Wed, 5 Jun 2019 23:25:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726502AbfFEXZK (ORCPT ); Wed, 5 Jun 2019 19:25:10 -0400 Received: from mail-eopbgr30054.outbound.protection.outlook.com ([40.107.3.54]:39078 "EHLO EUR03-AM5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726603AbfFEXZK (ORCPT ); Wed, 5 Jun 2019 19:25:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s5BroFyUPF9gicmWnc/xL3WNtl/53BqrtJm+rcRtAHs=; b=eYvNAb2kdqBgU15/GPU0s3xzah80q/OW/P6GwPEty8im46ErOef9dNXMBEI/8gdKhNtVXV/LMo5YtAf0PT59WCqOxLOU5kygMsekM3g+7zuwsuFSrvyMtmB6auCOosHOTd+Wps+BQSv8QobXXtEwQcsN3WCKL4OhApLXEL6ui94= Received: from DB8PR05MB5898.eurprd05.prod.outlook.com (20.179.9.32) by DB8PR05MB6105.eurprd05.prod.outlook.com (20.179.10.223) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1943.22; Wed, 5 Jun 2019 23:24:53 +0000 Received: from DB8PR05MB5898.eurprd05.prod.outlook.com ([fe80::4008:6417:32d4:6031]) by DB8PR05MB5898.eurprd05.prod.outlook.com ([fe80::4008:6417:32d4:6031%5]) with mapi id 15.20.1965.011; Wed, 5 Jun 2019 23:24:53 +0000 From: Saeed Mahameed To: "David S. Miller" , Jason Gunthorpe , Doug Ledford CC: Michael Chan , Andy Gospodarek , Tal Gilboa , "linux-rdma@vger.kernel.org" , "netdev@vger.kernel.org" , Yamin Friedman , Max Gurtovoy , Saeed Mahameed Subject: [for-next 9/9] RDMA/core: Provide RDMA DIM support for ULPs Thread-Topic: [for-next 9/9] RDMA/core: Provide RDMA DIM support for ULPs Thread-Index: AQHVG/XhmNzLXsetGES4OHmfYyEsgg== Date: Wed, 5 Jun 2019 23:24:53 +0000 Message-ID: <20190605232348.6452-10-saeedm@mellanox.com> References: <20190605232348.6452-1-saeedm@mellanox.com> In-Reply-To: <20190605232348.6452-1-saeedm@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-originating-ip: [209.116.155.178] x-clientproxiedby: BYAPR02CA0045.namprd02.prod.outlook.com (2603:10b6:a03:54::22) To DB8PR05MB5898.eurprd05.prod.outlook.com (2603:10a6:10:a4::32) authentication-results: spf=none (sender IP is ) smtp.mailfrom=saeedm@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 478db7b8-2872-41b3-d228-08d6ea0d0376 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020);SRVR:DB8PR05MB6105; x-ms-traffictypediagnostic: DB8PR05MB6105: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2089; x-forefront-prvs: 00594E8DBA x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(396003)(346002)(39850400004)(136003)(366004)(376002)(51234002)(199004)(189003)(4326008)(316002)(5660300002)(2906002)(66946007)(53936002)(186003)(8936002)(66476007)(8676002)(81166006)(66446008)(1076003)(478600001)(86362001)(50226002)(81156014)(66556008)(14454004)(107886003)(6116002)(73956011)(7736002)(305945005)(446003)(11346002)(64756008)(3846002)(14444005)(71190400001)(99286004)(26005)(25786009)(110136005)(71200400001)(486006)(2616005)(68736007)(6512007)(66066001)(6436002)(76176011)(6486002)(54906003)(36756003)(6506007)(386003)(102836004)(5024004)(256004)(476003)(52116002);DIR:OUT;SFP:1101;SCL:1;SRVR:DB8PR05MB6105;H:DB8PR05MB5898.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: N5w9y0oRlneDfpF5Dq7Q68UEgGf1jzSitrfqz6uRiHRPCm7ZcUv5FJ7wIZwOch7uIYeNCrQrlZ4KhKAtJdD5tRPj47FcOs4D3JGT9/C2Eyra2V0Ff5XEhnSJ3jO6GdaZ1ibXWDqHovkIA1zWNhM2RRsWHL8e9kl2KQUF/rJvkiXJi+HpVQkNdgfqtI9sg1ESzWeOn7HCO2al/bv4NRcn2jC/e96z7DIKdX+nazNLNZ4fUxRQS/Y8Gw+g6VukhNi8EIW8YbXRTv3eeZC+nHPMPs/DI+hydOwwSgoUv/mN2q4IrHmn9paKYNTnU2NvGKF4vOzVuGCf7Osi9mbPiGfwRi4mE18CXbaWx+TWL5hImojUcn/aXBQjXGs0XGMD914npRo0V6KaEFNe7a/yA5lyQjYJ6tiajr9yh2D0Vo6aVfM= MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 478db7b8-2872-41b3-d228-08d6ea0d0376 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Jun 2019 23:24:53.6914 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: saeedm@mellanox.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR05MB6105 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yamin Friedman Added the interface in the infiniband driver that applies the rdma_dim adaptive moderation. There is now a special function for allocating an ib_cq that uses rdma_dim. Performance improvement (ConnectX-5 100GbE, x86) running FIO benchmark over NVMf between two equal end-hosts with 56 cores across a Mellanox switch using null_blk device: READS without DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 3.8GiB/s | 7.7M | 1401 usec | 2442 usec 4k | 7.0GiB/s | 1.8M | 4817 usec | 6587 usec 64k | 10.7GiB/s| 175k | 9896 usec | 10028 usec IO WRITES without DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 3.6GiB/s | 7.5M | 1434 usec | 2474 usec 4k | 6.3GiB/s | 1.6M | 938 usec | 1221 usec 64k | 10.7GiB/s| 175k | 8979 usec | 12780 usec IO READS with DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 4GiB/s | 8.2M | 816 usec | 889 usec 4k | 10.1GiB/s| 2.65M| 3359 usec | 5080 usec 64k | 10.7GiB/s| 175k | 9896 usec | 10028 usec IO WRITES with DIM: blk size | BW | IOPS | 99th percentile latency | 99.99th latency 512B | 3.9GiB/s | 8.1M | 799 usec | 922 usec 4k | 9.6GiB/s | 2.5M | 717 usec | 1004 usec 64k | 10.7GiB/s| 176k | 8586 usec | 12256 usec The rdma_dim algorithm was designed to measure the effectiveness of moderation on the flow in a general way and thus should be appropriate for all RDMA storage protocols. Signed-off-by: Yamin Friedman Reviewed-by: Max Gurtovoy Signed-off-by: Saeed Mahameed --- drivers/infiniband/core/cq.c | 78 ++++++++++++++++++- drivers/net/ethernet/mellanox/mlx4/Kconfig | 1 + .../net/ethernet/mellanox/mlx5/core/Kconfig | 1 + include/rdma/ib_verbs.h | 27 ++++++- 4 files changed, 102 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index a4c81992267c..326d928d2763 100644 --- a/drivers/infiniband/core/cq.c +++ b/drivers/infiniband/core/cq.c @@ -14,6 +14,7 @@ #include #include #include +#include /* # of WCs to poll for with a single call to ib_poll_cq */ #define IB_POLL_BATCH 16 @@ -26,6 +27,32 @@ #define IB_POLL_FLAGS \ (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS) +static void ib_cq_rdma_dim_work(struct work_struct *w) +{ + struct dim *dim = container_of(w, struct dim, work); + struct ib_cq *cq = container_of(dim, struct ib_cq, dim); + + u16 usec = rdma_dim_prof[dim->profile_ix].usec; + u16 comps = rdma_dim_prof[dim->profile_ix].comps; + + dim->state = DIM_START_MEASURE; + + cq->device->ops.modify_cq(cq, comps, usec); +} + +static bool rdma_dim_init(struct dim *dim, struct ib_cq *cq) +{ + if (!cq->device->ops.modify_cq) + return false; + + memset(dim, 0, sizeof(*dim)); + dim->state = DIM_START_MEASURE; + dim->tune_state = DIM_GOING_RIGHT; + dim->profile_ix = RDMA_DIM_START_PROFILE; + + return true; +} + static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *wcs, int batch) { @@ -98,6 +125,24 @@ static int ib_poll_handler(struct irq_poll *iop, int budget) return completed; } +static int ib_poll_dim_handler(struct irq_poll *iop, int budget) +{ + struct ib_cq *cq = container_of(iop, struct ib_cq, iop); + int completed; + struct dim *dim = &cq->dim; + + completed = __ib_process_cq(cq, budget, cq->wc, IB_POLL_BATCH); + if (completed < budget) { + irq_poll_complete(&cq->iop); + if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) + irq_poll_sched(&cq->iop); + } + + rdma_dim(dim, completed); + + return completed; +} + static void ib_cq_completion_softirq(struct ib_cq *cq, void *private) { irq_poll_sched(&cq->iop); @@ -105,14 +150,18 @@ static void ib_cq_completion_softirq(struct ib_cq *cq, void *private) static void ib_cq_poll_work(struct work_struct *work) { - struct ib_cq *cq = container_of(work, struct ib_cq, work); + struct ib_cq *cq = container_of(work, struct ib_cq, + work); int completed; completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, cq->wc, IB_POLL_BATCH); + if (completed >= IB_POLL_BUDGET_WORKQUEUE || ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0) queue_work(cq->comp_wq, &cq->work); + else if (cq->dim_used) + rdma_dim(&cq->dim, completed); } static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) @@ -129,6 +178,7 @@ static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) * @poll_ctx: context to poll the CQ from. * @caller: module owner name. * @udata: Valid user data or NULL for kernel object + * @use_dim: use dynamic interrupt moderation * * This is the proper interface to allocate a CQ for in-kernel users. A * CQ allocated with this interface will automatically be polled from the @@ -138,7 +188,8 @@ static void ib_cq_completion_workqueue(struct ib_cq *cq, void *private) struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, int nr_cqe, int comp_vector, enum ib_poll_context poll_ctx, - const char *caller, struct ib_udata *udata) + const char *caller, struct ib_udata *udata, + bool use_dim) { struct ib_cq_init_attr cq_attr = { .cqe = nr_cqe, @@ -173,13 +224,30 @@ struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, case IB_POLL_SOFTIRQ: cq->comp_handler = ib_cq_completion_softirq; - irq_poll_init(&cq->iop, IB_POLL_BUDGET_IRQ, ib_poll_handler); + if (use_dim) + cq->dim_used = rdma_dim_init(&cq->dim, cq); + + if (cq->dim_used) { + irq_poll_init(&cq->iop, IB_POLL_BUDGET_IRQ, + ib_poll_dim_handler); + INIT_WORK(&cq->dim.work, ib_cq_rdma_dim_work); + } else { + irq_poll_init(&cq->iop, IB_POLL_BUDGET_IRQ, + ib_poll_handler); + } + ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); break; case IB_POLL_WORKQUEUE: case IB_POLL_UNBOUND_WORKQUEUE: cq->comp_handler = ib_cq_completion_workqueue; INIT_WORK(&cq->work, ib_cq_poll_work); + if (use_dim) + cq->dim_used = rdma_dim_init(&cq->dim, cq); + + if (cq->dim_used) + INIT_WORK(&cq->dim.work, ib_cq_rdma_dim_work); + ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); cq->comp_wq = (cq->poll_ctx == IB_POLL_WORKQUEUE) ? ib_comp_wq : ib_comp_unbound_wq; @@ -217,10 +285,14 @@ void ib_free_cq_user(struct ib_cq *cq, struct ib_udata *udata) break; case IB_POLL_SOFTIRQ: irq_poll_disable(&cq->iop); + if (cq->dim_used) + cancel_work_sync(&cq->dim.work); break; case IB_POLL_WORKQUEUE: case IB_POLL_UNBOUND_WORKQUEUE: cancel_work_sync(&cq->work); + if (cq->dim_used) + cancel_work_sync(&cq->dim.work); break; default: WARN_ON_ONCE(1); diff --git a/drivers/net/ethernet/mellanox/mlx4/Kconfig b/drivers/net/ethernet/mellanox/mlx4/Kconfig index e69c3c31e701..93cd25997b24 100644 --- a/drivers/net/ethernet/mellanox/mlx4/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx4/Kconfig @@ -28,6 +28,7 @@ config MLX4_CORE tristate depends on PCI select NET_DEVLINK + select DIMLIB default n config MLX4_DEBUG diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig index 7845aa5bf6be..ef292fbb53c9 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig @@ -7,6 +7,7 @@ config MLX5_CORE tristate "Mellanox 5th generation network adapters (ConnectX series) core driver" depends on PCI select NET_DEVLINK + select DIMLIB imply PTP_1588_CLOCK imply VXLAN imply MLXFW diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 0742095355f2..7b03fb3e4f0b 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -65,6 +65,7 @@ #include #include #include +#include #define IB_FW_VERSION_NAME_MAX ETHTOOL_FWVERS_LEN @@ -1638,6 +1639,8 @@ struct ib_cq { * Implementation details of the RDMA core, don't use in drivers: */ struct rdma_restrack_entry res; + struct dim dim; + bool dim_used; }; struct ib_srq { @@ -3746,7 +3749,8 @@ static inline int ib_post_recv(struct ib_qp *qp, struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private, int nr_cqe, int comp_vector, enum ib_poll_context poll_ctx, - const char *caller, struct ib_udata *udata); + const char *caller, struct ib_udata *udata, + bool use_dim); /** * ib_alloc_cq_user: Allocate kernel/user CQ @@ -3764,7 +3768,7 @@ static inline struct ib_cq *ib_alloc_cq_user(struct ib_device *dev, struct ib_udata *udata) { return __ib_alloc_cq_user(dev, private, nr_cqe, comp_vector, poll_ctx, - KBUILD_MODNAME, udata); + KBUILD_MODNAME, udata, false); } /** @@ -3785,6 +3789,25 @@ static inline struct ib_cq *ib_alloc_cq(struct ib_device *dev, void *private, NULL); } +/** + * ib_alloc_cq_dim: Allocate kernel CQ with dynamic interrupt moderation + * @dev: The IB device + * @private: Private data attached to the CQE + * @nr_cqe: Number of CQEs in the CQ + * @comp_vector: Completion vector used for the IRQs + * @poll_ctx: Context used for polling the CQ + * + * NOTE: for user cq use ib_alloc_cq_user with valid udata! + */ +static inline struct ib_cq *ib_alloc_cq_dim(struct ib_device *dev, + void *private, int nr_cqe, + int comp_vector, + enum ib_poll_context poll_ctx) +{ + return __ib_alloc_cq_user(dev, private, nr_cqe, comp_vector, poll_ctx, + KBUILD_MODNAME, NULL, true); +} + /** * ib_free_cq_user - Free kernel/user CQ * @cq: The CQ to free