From patchwork Fri Apr 19 18:05:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Hefty, Sean" X-Patchwork-Id: 10909547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48FBF1515 for ; Fri, 19 Apr 2019 18:58:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B7AA28DF5 for ; Fri, 19 Apr 2019 18:58:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FF7028DF9; Fri, 19 Apr 2019 18:58:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B404E28DF5 for ; Fri, 19 Apr 2019 18:58:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728659AbfDSS6y convert rfc822-to-8bit (ORCPT ); Fri, 19 Apr 2019 14:58:54 -0400 Received: from mga11.intel.com ([192.55.52.93]:51808 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728652AbfDSS6x (ORCPT ); Fri, 19 Apr 2019 14:58:53 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Apr 2019 11:05:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,370,1549958400"; d="scan'208";a="150788073" Received: from orsmsx106.amr.corp.intel.com ([10.22.225.133]) by FMSMGA003.fm.intel.com with ESMTP; 19 Apr 2019 11:05:54 -0700 Received: from orsmsx109.amr.corp.intel.com ([169.254.11.52]) by ORSMSX106.amr.corp.intel.com ([169.254.1.121]) with mapi id 14.03.0415.000; Fri, 19 Apr 2019 11:05:53 -0700 From: "Hefty, Sean" To: "linux-rdma (linux-rdma@vger.kernel.org)" Subject: [PATCH rdma-core 5/5] rsockets: Periodically wake-up threads from rpoll Thread-Topic: [PATCH rdma-core 5/5] rsockets: Periodically wake-up threads from rpoll Thread-Index: AdT22kOzhx/krXsPSL+5tLGaipsMRg== Date: Fri, 19 Apr 2019 18:05:52 +0000 Message-ID: <1828884A29C6694DAF28B7E6B8A82373B3E1F933@ORSMSX109.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMzgwYzg2MTAtNDA3Yi00MmEyLWEyOTMtODEyYTVlMzA3OGFlIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoicWk3NEFtTmFueUN2aXRqa2o0bFJlaitQQ1cwdWZrZWJkZThoV3c3TDJcL1hUd3Rvd3JKczlFTDJFbjlSZnFXRmUifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.600.7 dlp-reaction: no-action x-originating-ip: [10.22.254.139] MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The changes to rpoll to use a signaling fd to wake up blocked threads, combined with suspending polling while rsockets states may be changing _should_ prevent any threads from blocking indefinitely in rpoll() when a desired state change occurs. However, to be extra safe and proactive that we don't end up hanging an application, we periodically wake up any polling thread, so that it can recheck its rsocket states. This is simply a proactive measure to allow apps to make forward progress and not in response to any known issues. The sleeping interval is set to an arbitrary value of 2 seconds, but can be overridden using config files. Signed-off-by: Sean Hefty --- librdmacm/man/rsocket.7.in | 6 +++++- librdmacm/rsocket.c | 14 +++++++++++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/librdmacm/man/rsocket.7.in b/librdmacm/man/rsocket.7.in index 5b777690..7dc479ec 100644 --- a/librdmacm/man/rsocket.7.in +++ b/librdmacm/man/rsocket.7.in @@ -1,5 +1,5 @@ .\" Licensed under the OpenIB.org BSD license (FreeBSD Variant) - See COPYING.md -.TH "RSOCKET" 7 "2013-01-21" "librdmacm" "Librdmacm Programmer's Manual" librdmacm +.TH "RSOCKET" 7 "2019-04-16" "librdmacm" "Librdmacm Programmer's Manual" librdmacm .SH NAME rsocket \- RDMA socket API .SH SYNOPSIS @@ -147,6 +147,10 @@ iomap_size - default size of remote iomapping table .P polling_time - default number of microseconds to poll for data before waiting .P +wake_up_interval - maximum number of milliseconds to block in poll. +This value is used to safe guard against potential application hangs +in rpoll(). +.P All configuration files should contain a single integer value. Values may be set by issuing a command similar to the following example. .P diff --git a/librdmacm/rsocket.c b/librdmacm/rsocket.c index ed5e0fe3..bd67619b 100644 --- a/librdmacm/rsocket.c +++ b/librdmacm/rsocket.c @@ -129,6 +129,7 @@ static uint16_t def_rqsize = 384; static uint32_t def_mem = (1 << 17); static uint32_t def_wmem = (1 << 17); static uint32_t polling_time = 10; +static int wake_up_interval = 2000; /* * Immediate data format is determined by the upper bits @@ -543,6 +544,10 @@ static void rs_configure(void) fclose(f); } + if ((f = fopen(RS_CONF_DIR "/wake_up_interval", "r"))) { + failable_fscanf(f, "%d", &wake_up_interval); + fclose(f); + } if ((f = fopen(RS_CONF_DIR "/inline_default", "r"))) { failable_fscanf(f, "%hu", &def_inline); fclose(f); @@ -3259,7 +3264,7 @@ int rpoll(struct pollfd *fds, nfds_t nfds, int timeout) struct pollfd *rfds; uint64_t start_time; uint32_t poll_time = 0; - int ret; + int pollsleep, ret; do { ret = rs_poll_check(fds, nfds); @@ -3288,10 +3293,13 @@ int rpoll(struct pollfd *fds, nfds_t nfds, int timeout) timeout -= (int) ((rs_time_us() - start_time) / 1000); if (timeout <= 0) return 0; + pollsleep = min(timeout, wake_up_interval); + } else { + pollsleep = wake_up_interval; } - ret = poll(rfds, nfds + 1, timeout); - if (ret <= 0) { + ret = poll(rfds, nfds + 1, pollsleep); + if (ret < 0) { rs_poll_exit(); break; }