From patchwork Wed Oct 10 06:27:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 10633999 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6867D112B for ; Wed, 10 Oct 2018 06:27:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5837F29622 for ; Wed, 10 Oct 2018 06:27:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B05229633; Wed, 10 Oct 2018 06:27:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D70F229622 for ; Wed, 10 Oct 2018 06:27:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726004AbeJJNs0 (ORCPT ); Wed, 10 Oct 2018 09:48:26 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:34247 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725757AbeJJNs0 (ORCPT ); Wed, 10 Oct 2018 09:48:26 -0400 Received: by mail-pg1-f195.google.com with SMTP id g12-v6so2016798pgs.1 for ; Tue, 09 Oct 2018 23:27:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=4Op7D7amJ+7yo2VDNgb3jv3vkhd7Jut52RDtf2vWFBQ=; b=DvvSHqY6HyQWn8a0gwgVeO3C8OWM0k7jjBw457a7x2FgquGCr0w36Pfy/IPwZDtkUd XfCYAu0dkogU9FVlmvyd+63EM3Yo233un4vOtm/+GAiu9gO/fX9ByM3vOE2ddmTALFaq Ic0SRRs0vhVr12l/nxEJBQRBwnq8+6BWvV0LU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=4Op7D7amJ+7yo2VDNgb3jv3vkhd7Jut52RDtf2vWFBQ=; b=HeLntOQbGpYBzWf6DqrjjnGaYacjNEX1hwUnncW/xzfJz9tw8guBrgZPxAGPeaI/3U kXIS9Ew/5zKw+0+hpcU310mJrHlBwYLY43S/hMY5cP+yglFHYi0TNv0TDa7dObPZoNdq MFWYOW10X2MXfycq2eSe/xHqDoadoiz8C+ujGwdSxJC0/kxMR72KmYWKkhiUsc5nZJfR rknCisOzyb964S3OmKnRzGVMU8DAQCnohKZc3duPMBRvuj0gkCRjqsAaR/EBvREg9XtJ TUTgiipt7dTgJInWqUpJAYIm9wdse+6Ag3Wi1KjwAl1CaHuPJmn1DamkxEKd7LdT9qA/ YMVA== X-Gm-Message-State: ABuFfohpy2BYMLhPza68M/ai9zGFjYobGQXQoTBRqZzUoeZQBkccXkBP 1L2x0yLwRr7VuUAArNX9Oodepg== X-Google-Smtp-Source: ACcGV60UyUUH96zJKnYwCMrbynWxIAe9lxGpIzqKHs5P12L4W65NjhEf+dyZ4e4SPCidrf7mp1znYQ== X-Received: by 2002:a63:f:: with SMTP id 15-v6mr28421799pga.15.1539152868114; Tue, 09 Oct 2018 23:27:48 -0700 (PDT) Received: from neo00-el73.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id r65-v6sm31330790pfj.5.2018.10.09.23.27.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Oct 2018 23:27:47 -0700 (PDT) From: Devesh Sharma To: jgg@mellanox.com, dledford@redhat.com Cc: linux-rdma@vger.kernel.org, NMoreyChaisemartin@suse.de, Devesh Sharma , JD Zheng , Jonathan Richardson Subject: [PATCH rdma-core V2 2/4] bnxt_re/lib: Remove db_lock around doorbell ring Date: Wed, 10 Oct 2018 02:27:28 -0400 Message-Id: <1539152850-24357-3-git-send-email-devesh.sharma@broadcom.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1539152850-24357-1-git-send-email-devesh.sharma@broadcom.com> References: <1539152850-24357-1-git-send-email-devesh.sharma@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Remove the lock around the doorbell. It isn't necessary and was a bottleneck for multithreaded clients. Performance of spdk improves by 40% with it removed. Signed-off-by: Devesh Sharma Signed-off-by: JD Zheng Signed-off-by: Jonathan Richardson --- providers/bnxt_re/db.c | 9 ++++----- providers/bnxt_re/main.c | 1 - providers/bnxt_re/main.h | 1 - providers/bnxt_re/memory.h | 10 ---------- providers/bnxt_re/verbs.c | 2 -- 5 files changed, 4 insertions(+), 19 deletions(-) diff --git a/providers/bnxt_re/db.c b/providers/bnxt_re/db.c index a79f871..85da182 100644 --- a/providers/bnxt_re/db.c +++ b/providers/bnxt_re/db.c @@ -36,7 +36,7 @@ * Description: Doorbell handling functions. */ -#include +#include #include "main.h" static void bnxt_re_ring_db(struct bnxt_re_dpi *dpi, @@ -44,11 +44,10 @@ static void bnxt_re_ring_db(struct bnxt_re_dpi *dpi, { __le64 *dbval; - pthread_spin_lock(&dpi->db_lock); dbval = (__le64 *)&hdr->indx; - udma_to_device_barrier(); - iowrite64(dpi->dbpage, dbval); - pthread_spin_unlock(&dpi->db_lock); + mmio_wc_start(); + mmio_write64_le(dpi->dbpage, *dbval); + mmio_flush_writes(); } static void bnxt_re_init_db_hdr(struct bnxt_re_db_hdr *hdr, uint32_t indx, diff --git a/providers/bnxt_re/main.c b/providers/bnxt_re/main.c index 54e3cc3..a1ba06a 100644 --- a/providers/bnxt_re/main.c +++ b/providers/bnxt_re/main.c @@ -167,7 +167,6 @@ static void bnxt_re_free_context(struct ibv_context *ibvctx) * allocated in this context. */ if (cntx->udpi.dbpage && cntx->udpi.dbpage != MAP_FAILED) { - pthread_spin_destroy(&cntx->udpi.db_lock); munmap(cntx->udpi.dbpage, dev->pg_size); cntx->udpi.dbpage = NULL; } diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index 16f55f9..0b5c749 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -59,7 +59,6 @@ struct bnxt_re_dpi { __u32 dpindx; __u64 *dbpage; - pthread_spinlock_t db_lock; }; struct bnxt_re_pd { diff --git a/providers/bnxt_re/memory.h b/providers/bnxt_re/memory.h index aac0ff3..75564c4 100644 --- a/providers/bnxt_re/memory.h +++ b/providers/bnxt_re/memory.h @@ -81,16 +81,6 @@ static inline unsigned long roundup_pow_of_two(unsigned long val) int bnxt_re_alloc_aligned(struct bnxt_re_queue *que, uint32_t pg_size); void bnxt_re_free_aligned(struct bnxt_re_queue *que); -static inline void iowrite64(__u64 *dst, __le64 *src) -{ - *(volatile __le64 *)dst = *src; -} - -static inline void iowrite32(__u32 *dst, __le32 *src) -{ - *(volatile __le32 *)dst = *src; -} - /* Basic queue operation */ static inline uint32_t bnxt_re_is_que_full(struct bnxt_re_queue *que) { diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 9ce1454..8e9fd8c 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -110,8 +110,6 @@ struct ibv_pd *bnxt_re_alloc_pd(struct ibv_context *ibvctx) (void)ibv_cmd_dealloc_pd(&pd->ibvpd); goto out; } - pthread_spin_init(&cntx->udpi.db_lock, - PTHREAD_PROCESS_PRIVATE); } return &pd->ibvpd;