From patchwork Tue Jun 10 14:16:13 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hal Rosenstock X-Patchwork-Id: 4329141 X-Patchwork-Delegate: hal@mellanox.com Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 76711BEEAA for ; Tue, 10 Jun 2014 14:16:26 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7D10A20274 for ; Tue, 10 Jun 2014 14:16:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E5543201F5 for ; Tue, 10 Jun 2014 14:16:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752207AbaFJOQV (ORCPT ); Tue, 10 Jun 2014 10:16:21 -0400 Received: from mail-wi0-f177.google.com ([209.85.212.177]:62083 "EHLO mail-wi0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751058AbaFJOQV (ORCPT ); Tue, 10 Jun 2014 10:16:21 -0400 Received: by mail-wi0-f177.google.com with SMTP id f8so6237131wiw.10 for ; Tue, 10 Jun 2014 07:16:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=pnuuyT2NCIiJHlzRsxS+RXpj5I1bfG9uiESs43nIQX4=; b=P2YRlsTANbRno2pLVEQY8ACNK6uv9gOTUAW8Fig9+JQV1uswtNbzt5Ycqt7FFDJF6v XJSZmyiksO6ITRxpcounUzfmIpe3sWr35fwrd9zjN8TC6qKjoZbHvIQQP7FVuL9laCqk NU/ST0Rh9qtLmO3Kc5HfxSyEYmSDVwhzRcapBvN15OE+41L9DMIhcp1ZErc7zEBGfVqq Pt3q+dd7gQ1HpI5pVZFabr0gA14ZA0IgR0puffz1O/zy6gaTuPSIFr5zjnbUlJ5fXO+P 75oAB7YOTOgX/IFEUEcON0yRfflDbWGNThnxZu4kPCTSRFFTNsLEJNfQOf5iAGTHnDSs xmVw== X-Gm-Message-State: ALoCoQlLr/17LFp9qL+cWX/S7TWzFV5GcB57UqwLxZwVvtDM1b+9QH4jSbF8PkTCp3PFniQl2EgG X-Received: by 10.194.178.99 with SMTP id cx3mr42269971wjc.54.1402409778776; Tue, 10 Jun 2014 07:16:18 -0700 (PDT) Received: from [192.168.1.102] (c-98-229-118-119.hsd1.ma.comcast.net. [98.229.118.119]) by mx.google.com with ESMTPSA id cx5sm29827404wjb.8.2014.06.10.07.16.17 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 10 Jun 2014 07:16:17 -0700 (PDT) Message-ID: <5397132D.4020603@dev.mellanox.co.il> Date: Tue, 10 Jun 2014 10:16:13 -0400 From: Hal Rosenstock User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: "linux-rdma (linux-rdma@vger.kernel.org)" Subject: [PATCH opensm] osm_vendor_ibumad.c: Better match table eviction strategy Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Rather than just evict LRU transaction when the match table is full, evict the LRU non SMP transaction and if none exist then evict the LRU SMP transaction. Signed-off-by: Hal Rosenstock --- libvendor/osm_vendor_ibumad.c | 56 +++++++++++++++++++++++++++++++--------- 1 files changed, 43 insertions(+), 13 deletions(-) diff --git a/libvendor/osm_vendor_ibumad.c b/libvendor/osm_vendor_ibumad.c index d9ed13a..94a2783 100644 --- a/libvendor/osm_vendor_ibumad.c +++ b/libvendor/osm_vendor_ibumad.c @@ -187,15 +187,21 @@ static osm_madw_t *get_madw(osm_vendor_t * p_vend, ib_net64_t * tid, return 0; } +/* + * If match table full, evict LRU (least recently used) transaction. + * Maintain 2 LRUs: one for SMPs, and one for others (GS). + * Evict LRU GS transaction if one is available and only evict LRU SMP + * transaction if no other choice. + */ static void put_madw(osm_vendor_t * p_vend, osm_madw_t * p_madw, ib_net64_t tid, uint8_t mgmt_class) { - umad_match_t *m, *e, *old_lru, *lru = 0; + umad_match_t *m, *e, *old_lru, *lru = 0, *lru_smp = 0; osm_madw_t *p_req_madw; osm_umad_bind_info_t *p_bind; ib_net64_t old_tid; - uint32_t oldest = ~0; + uint32_t oldest = ~0, oldest_smp = ~0; uint8_t old_mgmt_class; pthread_mutex_lock(&p_vend->match_tbl_mutex); @@ -210,15 +216,30 @@ put_madw(osm_vendor_t * p_vend, osm_madw_t * p_madw, ib_net64_t tid, pthread_mutex_unlock(&p_vend->match_tbl_mutex); return; } - if (oldest >= m->version) { - oldest = m->version; - lru = m; + if (m->mgmt_class == IB_MCLASS_SUBN_DIR || + m->mgmt_class == IB_MCLASS_SUBN_LID) { + if (oldest_smp >= m->version) { + oldest_smp = m->version; + lru_smp = m; + } + } else { + if (oldest >= m->version) { + oldest = m->version; + lru = m; + } } } - old_lru = lru; - old_tid = lru->tid; - old_mgmt_class = lru->mgmt_class; + if (oldest != ~0) { + old_lru = lru; + old_tid = lru->tid; + old_mgmt_class = lru->mgmt_class; + } else { + CL_ASSERT(oldest_smp != ~0); + old_lru = lru_smp; + old_tid = lru_smp->tid; + old_mgmt_class = lru_smp->mgmt_class; + } p_req_madw = old_lru->v; p_bind = p_req_madw->h_bind; p_req_madw->status = IB_CANCELED; @@ -226,11 +247,20 @@ put_madw(osm_vendor_t * p_vend, osm_madw_t * p_madw, ib_net64_t tid, pthread_mutex_lock(&p_vend->cb_mutex); (*p_bind->send_err_callback) (p_bind->client_context, p_req_madw); pthread_mutex_unlock(&p_vend->cb_mutex); - lru->tid = tid; - lru->mgmt_class = mgmt_class; - lru->v = p_madw; - lru->version = - cl_atomic_inc((atomic32_t *) & p_vend->mtbl.last_version); + if (mgmt_class == IB_MCLASS_SUBN_DIR || + mgmt_class == IB_MCLASS_SUBN_LID) { + lru_smp->tid = tid; + lru_smp->mgmt_class = mgmt_class; + lru_smp->v = p_madw; + lru_smp->version = + cl_atomic_inc((atomic32_t *) & p_vend->mtbl.last_version); + } else { + lru->tid = tid; + lru->mgmt_class = mgmt_class; + lru->v = p_madw; + lru->version = + cl_atomic_inc((atomic32_t *) & p_vend->mtbl.last_version); + } pthread_mutex_unlock(&p_vend->match_tbl_mutex); OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5402: " "evicting entry %p (tid was 0x%" PRIx64