From patchwork Wed Nov 15 15:27:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDBA0C54FB9 for ; Wed, 15 Nov 2023 15:28:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229768AbjKOP2C (ORCPT ); Wed, 15 Nov 2023 10:28:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229646AbjKOP2B (ORCPT ); Wed, 15 Nov 2023 10:28:01 -0500 Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6279A181 for ; Wed, 15 Nov 2023 07:27:58 -0800 (PST) Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-53e2308198eso10605100a12.1 for ; Wed, 15 Nov 2023 07:27:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062077; x=1700666877; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sQqIrb54h0DMj/KbzsRVn/beLRIDs48LlOFxYh9J+hc=; b=XCKOe+dSprctWvh/VitzHxzNn9vyO9iArDOqNdcpeD+/id/HjVjAFqDjdU9mRdkst2 8fk2Ind8BPAGy6D/LMnQUVstIHtKfY5DXxuc/xRkmgPBzoxIiwcImGIXmZwUBvx0jIAv Gf4xRRDMlw3Oj/J89j0KEhT1z+e+B8nSRyFy24PpJP5ZDwG9bzf4V/sj3WVekDZS05Fz dpNhydI8ptvKg2U2bAjfwghoZVBa6CDJfJ9oI1HkiUs9c/vsvDH4/xABNR7kQ5OL+I7+ 9XwWb4jQA+CltGvDT0L6BslBMEZJUkLxBHD804bCNLXUFvTa+crFt3z/o0GH15S9na1O FejA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062077; x=1700666877; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sQqIrb54h0DMj/KbzsRVn/beLRIDs48LlOFxYh9J+hc=; b=Iw/Yj9eguNZ16R/+nJrJuwBnfMyNkBjPXBeD23BQgxRdUNW8lnM3KTqpMI3fdd7GhL r7bHeq5tRlS3frqMe1lVKM3Ih9+B96WcYp6wbG3iyiCXjpazB6opBCBGo1Sj+sCweoA+ GaivY2VMk/59BkG4+X59spEoL5RrzXhr1s/gcClhevCSUcVBWsSfKRY1zR1PhjpnR3HI FduxiWVbyZjm9AMhOJ24iLvI4dY3Yc1GCb3M2UGb21g3Ay8IHnxSiqbIvFt2f+sK5x7F vhjFBx1iuiN923NqSYp4IB90n6tgn0yCyage2nevy8IQ2WvhgaScbu01k9AwIoiIjngg m+og== X-Gm-Message-State: AOJu0YyBF96819s73J2T9aStFMxQi4kXPZC0tTzd4cluMZ86gkQF05gi YLdPQ3LQQrUi/hjMSR2n20aE5U5MzDnxY9gY/qk= X-Google-Smtp-Source: AGHT+IHjY/r1KGkFXBxFP7fcaa/vVQj/KwIB9v+CXa1QvdELX45vSAvUYUz959j421l2IfKlte7LFA== X-Received: by 2002:a05:6402:1507:b0:543:4ff0:4fd5 with SMTP id f7-20020a056402150700b005434ff04fd5mr7833088edw.39.1700062076697; Wed, 15 Nov 2023 07:27:56 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:27:56 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH for-next 1/9] RDMA/rtrs-clt: Add warning logs for RDMA events Date: Wed, 15 Nov 2023 16:27:41 +0100 Message-Id: <20231115152749.424301-2-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Some RDMA CM events can trigger connection close or recovery for a certain rtrs_clt_path. Such close/recovery triggers should happen after an appropriate log message, since they can lead to IO failures. Signed-off-by: Md Haris Iqbal Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 07261523c554..984a4a1db3c8 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -2025,6 +2025,8 @@ static int rtrs_clt_rdma_cm_handler(struct rdma_cm_id *cm_id, /* * Device removal is a special case. Queue close and return 0. */ + rtrs_wrn_rl(s, "CM event: %s, status: %d\n", rdma_event_msg(ev->event), + ev->status); rtrs_clt_close_conns(clt_path, false); return 0; default: From patchwork Wed Nov 15 15:27:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456950 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31456C2BB3F for ; Wed, 15 Nov 2023 15:28:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229696AbjKOP2D (ORCPT ); Wed, 15 Nov 2023 10:28:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbjKOP2C (ORCPT ); Wed, 15 Nov 2023 10:28:02 -0500 Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D347A18A for ; Wed, 15 Nov 2023 07:27:58 -0800 (PST) Received: by mail-ed1-x52c.google.com with SMTP id 4fb4d7f45d1cf-547e7de7b6fso851979a12.0 for ; Wed, 15 Nov 2023 07:27:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062077; x=1700666877; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5mnD/MS4Zvh8PQ5PRgjBo4e3nhT20PGw9b1s4Mi6YtI=; b=ESwh1QHX/n7xwJFfDgzTvhuLYzKlP165nlZ2V87A9jFB+3DJzel9gHGrGOANAlWIRD qvKUTTnR2uBiRb6GL8+Z1Xct101sslxaZWNN6ld7nLEE+/5X62jWaYwDke1oJaxXqMyf 8O+BWVw2rLTCbfLrjY/8vL2mon0K99eIe5eqQjN1ceVuUA/6AiOq0qNuAZZXjCIHEspU knyuedH5K2xDI4LuDg8SpU6UnmLonxldRBIm+VB/y0iRfkKVjqC9kFGGiymbZH5MsQaK zoENIO+IoDRW9TEo0Rd907mlANO1+Ni8BEVeco0rBLU9lld+qJtu76LRl1AoGaVRLm1j aa2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062077; x=1700666877; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5mnD/MS4Zvh8PQ5PRgjBo4e3nhT20PGw9b1s4Mi6YtI=; b=Knx9zMBnslMAn4csTalnAO6GPqyJXA7tVvf+6YQXbw95S7jYKD/RKvusueAyFESm/m 2RxiQnrkepPAn1A5UBkWkFLa1IbcDoYTYExhsNtlnO24f6SYeDh8xCU3pQZIgHGVV9SW SsDY4fGwQbNVfej/gfE3BpMgFBoFPXQqIyMH/3TLldfmd3V+O0lG8V68/IMBLrrpkn4z k1fBtI8s+pCvxbzQh7RJIAKMDEGMVupIF7pcB8Xdbi9znoxlkXyUF7k7Nvd3DexJTfoH nFYgIJ90oLIY37gxexYMsbKqBdLe9WLay/ATqjhbW3lL4XY0VCO53+rE1HNhiLxcncJA pjTw== X-Gm-Message-State: AOJu0Yx7lSc4pVbQjYTsBw5FjXfcrYnpAsq5veh1VcMIJzVCzztz4vdp jklIaLNh9yFvk8c4PWmC6oz4xWzD6ed9mIxH7yk= X-Google-Smtp-Source: AGHT+IGYoQE9Tu/vD8pkOIT/Du3aPz3MxfK0eg+PHcsndw5EBbKnBmzGwTe33hPufdKxzpOVMYFsbQ== X-Received: by 2002:a05:6402:115a:b0:53d:b59c:8f8d with SMTP id g26-20020a056402115a00b0053db59c8f8dmr5421531edw.8.1700062077286; Wed, 15 Nov 2023 07:27:57 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:27:57 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Florian-Ewald Mueller , Grzegorz Prajsner Subject: [PATCH for-next 2/9] RDMA/rtrs-srv: Do not unconditionally enable irq Date: Wed, 15 Nov 2023 16:27:42 +0100 Message-Id: <20231115152749.424301-3-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jack Wang When IO is completed, rtrs can be called in softirq context, unconditionally enabling irq could cause panic. To be on safe side, use spin_lock_irqsave and spin_unlock_irqrestore instread. Signed-off-by: Jack Wang Signed-off-by: Florian-Ewald Mueller Signed-off-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 75e56604e462..ab4200041fd3 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -65,8 +65,9 @@ static bool rtrs_srv_change_state(struct rtrs_srv_path *srv_path, { enum rtrs_srv_state old_state; bool changed = false; + unsigned long flags; - spin_lock_irq(&srv_path->state_lock); + spin_lock_irqsave(&srv_path->state_lock, flags); old_state = srv_path->state; switch (new_state) { case RTRS_SRV_CONNECTED: @@ -87,7 +88,7 @@ static bool rtrs_srv_change_state(struct rtrs_srv_path *srv_path, } if (changed) srv_path->state = new_state; - spin_unlock_irq(&srv_path->state_lock); + spin_unlock_irqrestore(&srv_path->state_lock, flags); return changed; } From patchwork Wed Nov 15 15:27:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3405EC072A2 for ; Wed, 15 Nov 2023 15:28:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231586AbjKOP2D (ORCPT ); Wed, 15 Nov 2023 10:28:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229989AbjKOP2C (ORCPT ); Wed, 15 Nov 2023 10:28:02 -0500 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D96F19D for ; Wed, 15 Nov 2023 07:27:59 -0800 (PST) Received: by mail-ed1-x52a.google.com with SMTP id 4fb4d7f45d1cf-53e07db272cso10696695a12.3 for ; Wed, 15 Nov 2023 07:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062078; x=1700666878; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oXVhHS+esCbUis83o9HgRtgY9C2LVzgM15KGu0G+oHs=; b=Et4L7PyhSvUFMKsbre+FL7woptaHtWNky4XILXsN3a98joPTcwbcv6ObQpjJcJZS9Q iwZZ/L28GvsFHQqrOJGDNnA9pir56OOa+UMrY1omTdVyQR2qZXy+kEtoOK1qFeVvmbuU 6wgk3dDwGTj0QnP6gj8I6hHPyVB9tle5LubPdyHLCBVanqvZ96fEnntxQ35dqqvT53Mn jtZENoot5Af7Q6d1xxIl1KFYv1yvdIW3UrRlvir5et0Iazzxz8QZKtD3l3QlLro1WSJs 343ItU1XDP13ynr4zov/Dd5gdfv3ankz61vvLqsD2/Zhkcka3Sy5fyIjsQa5kNhFvSI7 emQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062078; x=1700666878; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oXVhHS+esCbUis83o9HgRtgY9C2LVzgM15KGu0G+oHs=; b=muIBSECIKzNgIQ39J0EcjA2ahTKeZu9uKFNb0aawYqd4HSwCGfDTirfPPL5hpAdL2j mix+6Gn1+M48YhifyuAhtyjezus07OBYn2AAOmVdTSFo3Sv6CEoD7lDB0wHGs/EIZcj5 6JGHgAi2UCjCzodiSwuPS+T8Ql2pdKKCClD8RdHU5l3ugJMGJo3aqa9qp8vO1HBujkOq ret7PmWWFOtuI3seTBqdIO0a9N9LGTMYZxTwXURVfBt+oI5d9anRY71vT/77uClKeiRA LzQB3bTTl/O23F+F5J87dE2kQDKRZy+PPhzKhpcNxe+DkP4yIxLTG7KDyZ6Fyw3QpszY onfg== X-Gm-Message-State: AOJu0YxZ2EERbpbYHGCdGo2D31pmdObA1VO8kR0JnJybq0xDvSl0Y0eP 4YY6zQ35oLA+BEmggMAXVg8/PAiaW+UTlEGeAoM= X-Google-Smtp-Source: AGHT+IHU11tSAaiOQYuQKIMF6+OBUWDUgBSyvA/13IHjTvOKVWca6MKAZ2FW4a2xY5kWC2ZKDvOw8g== X-Received: by 2002:aa7:c045:0:b0:543:5f3:c92 with SMTP id k5-20020aa7c045000000b0054305f30c92mr9476082edo.36.1700062077971; Wed, 15 Nov 2023 07:27:57 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:27:57 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH for-next 3/9] RDMA/rtrs-clt: Start hb after path_up Date: Wed, 15 Nov 2023 16:27:43 +0100 Message-Id: <20231115152749.424301-4-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jack Wang If we start hb too early, it will confuse server side to close the session. Signed-off-by: Jack Wang Reviewed-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 984a4a1db3c8..83ebd9be760e 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -2352,8 +2352,6 @@ static int init_conns(struct rtrs_clt_path *clt_path) if (err) goto destroy; - rtrs_start_hb(&clt_path->s); - return 0; destroy: @@ -2627,6 +2625,7 @@ static int init_path(struct rtrs_clt_path *clt_path) goto out; } rtrs_clt_path_up(clt_path); + rtrs_start_hb(&clt_path->s); out: mutex_unlock(&clt_path->init_mutex); From patchwork Wed Nov 15 15:27:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74C0FC5AD4C for ; Wed, 15 Nov 2023 15:28:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229738AbjKOP2F (ORCPT ); Wed, 15 Nov 2023 10:28:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbjKOP2D (ORCPT ); Wed, 15 Nov 2023 10:28:03 -0500 Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E64912F for ; Wed, 15 Nov 2023 07:28:00 -0800 (PST) Received: by mail-lf1-x12e.google.com with SMTP id 2adb3069b0e04-507bd644a96so9875498e87.3 for ; Wed, 15 Nov 2023 07:28:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062078; x=1700666878; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1s0YFa68fF+ubtjF9HVVaUhPUqjPwA2OrrJXt0ep3L8=; b=hAP5XEOG6qPS5qtZbGs//WZTveHa7WxjeGQj4cxAab2xiXg/lf9Xq+Dj67Fhp3S3rC rCdI0MGBhzRVfEJqdUlhhmz85HmSDetkoifEgW2DUd8Rfd0kS8nRmA+zvggXYu07Q1j6 E7rf+PvlmQjC1pUl9hcbMHLpF8c3r+QBxw0S7aRAErD99wUZTh+hFMF9K1XC66XC24ZH WSJxawUYLV3yXxQ3uP847+JxhUDo55PEC+FN5I2tdgR5kHTDqiLhnr/+yHkI34/bY2U0 puy7BPHr/xmZhhgIqGxNjpK/ISQiyXtKD4dGXzqPiU2ipGWas0qiXvLp53mKt8Fpgdj4 Zxlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062078; x=1700666878; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1s0YFa68fF+ubtjF9HVVaUhPUqjPwA2OrrJXt0ep3L8=; b=H5M8qReMWI+6G8SyJ9Kxfnt2mT8dNu0zD9iZWev9V8umw1rpTFjKl0jYS2vtE+W6rj udFbr/8DRWDHRuguS4Ohx6H9wPoQfPR062FfYoGKj/gn7Rjw92ZdaOPFel3r/U8n+7TN 0Wdatbhxr/Rj7vRyqy7rZfKAHFkQRkYDDbAwNWsnOmyNAwwweMDRouc+UX5dC+Rd1MSx jM6uEzAfqtfhSNgg3u4KxhZ6SEglJD8MjbXypS4ZCOBCbK3Jr8sTXN2sKPB+MimZPYO3 prGN6iPIt2e0tRtCmI4kZvxXEJ8pJXVUUISkUX+/fAuAzuipkyAdXRR4t7S763AnQRbs Nbtw== X-Gm-Message-State: AOJu0YwjX2E+cWkEvgcdl0OYHCtu/vdHvefyy5SpRWRPD4T5l64Rytze 7cy1bCE9HBW68dC3br/wmUseZ3mKrY8Xlt1vLUI= X-Google-Smtp-Source: AGHT+IEeCFvnCYTpbmzbDNGab2vfngGxrmvPQcZxjGRGNZbPcW8s2HcOpPDcxgEE4vbzuqDnj2Supg== X-Received: by 2002:ac2:558b:0:b0:4fb:bef0:948e with SMTP id v11-20020ac2558b000000b004fbbef0948emr8232855lfg.5.1700062078585; Wed, 15 Nov 2023 07:27:58 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:27:58 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH for-next 4/9] RDMA/rtrs-srv: Check return values while processing info request Date: Wed, 15 Nov 2023 16:27:44 +0100 Message-Id: <20231115152749.424301-5-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org While processing info request, it could so happen that the srv_path goes to CLOSING state, cause of any of the error events from RDMA. That state change should be picked up while trying to change the state in process_info_req, by checking the return value. In case the state change call in process_info_req fails, we fail the processing. We should also check the return value for rtrs_srv_path_up, since it sends a link event to the client above, and the client can fail for any reason. Signed-off-by: Md Haris Iqbal Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index ab4200041fd3..4be0e5b132d4 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -710,20 +710,23 @@ static void rtrs_srv_info_rsp_done(struct ib_cq *cq, struct ib_wc *wc) WARN_ON(wc->opcode != IB_WC_SEND); } -static void rtrs_srv_path_up(struct rtrs_srv_path *srv_path) +static int rtrs_srv_path_up(struct rtrs_srv_path *srv_path) { struct rtrs_srv_sess *srv = srv_path->srv; struct rtrs_srv_ctx *ctx = srv->ctx; - int up; + int up, ret = 0; mutex_lock(&srv->paths_ev_mutex); up = ++srv->paths_up; if (up == 1) - ctx->ops.link_ev(srv, RTRS_SRV_LINK_EV_CONNECTED, NULL); + ret = ctx->ops.link_ev(srv, RTRS_SRV_LINK_EV_CONNECTED, NULL); mutex_unlock(&srv->paths_ev_mutex); /* Mark session as established */ - srv_path->established = true; + if (!ret) + srv_path->established = true; + + return ret; } static void rtrs_srv_path_down(struct rtrs_srv_path *srv_path) @@ -852,7 +855,12 @@ static int process_info_req(struct rtrs_srv_con *con, goto iu_free; kobject_get(&srv_path->kobj); get_device(&srv_path->srv->dev); - rtrs_srv_change_state(srv_path, RTRS_SRV_CONNECTED); + err = rtrs_srv_change_state(srv_path, RTRS_SRV_CONNECTED); + if (!err) { + rtrs_err(s, "rtrs_srv_change_state(), err: %d\n", err); + goto iu_free; + } + rtrs_srv_start_hb(srv_path); /* @@ -861,7 +869,11 @@ static int process_info_req(struct rtrs_srv_con *con, * all connections are successfully established. Thus, simply notify * listener with a proper event if we are the first path. */ - rtrs_srv_path_up(srv_path); + err = rtrs_srv_path_up(srv_path); + if (err) { + rtrs_err(s, "rtrs_srv_path_up(), err: %d\n", err); + goto iu_free; + } ib_dma_sync_single_for_device(srv_path->s.dev->ib_dev, tx_iu->dma_addr, From patchwork Wed Nov 15 15:27:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF218C2BB3F for ; Wed, 15 Nov 2023 15:28:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231734AbjKOP2G (ORCPT ); Wed, 15 Nov 2023 10:28:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230460AbjKOP2E (ORCPT ); Wed, 15 Nov 2023 10:28:04 -0500 Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D663518A for ; Wed, 15 Nov 2023 07:28:00 -0800 (PST) Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-507973f3b65so9805607e87.3 for ; Wed, 15 Nov 2023 07:28:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062079; x=1700666879; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pIwdTD2wwefJLEoIwT50HmG4gzgICRaFcaA956d/E3E=; b=TKy3uMYc+Pa+Q2iD1FW/m8cVPr5jgHtCbfB18hLdcCr1IE07H382ou22yoweEw/hT9 4uxQ+UQMv/WbqCacz3now7vAO0upKG/QVoQhUDNwZ7CTtVAAzvRtOPhUAhLfynuPispX 3OhoeRdEts5SqRQwfjhpWVjVNpMypfkP6GgvD4O4Ci4LNdDOy1K5mEpBBUJVYrzJ2Yxd 54XiVGF7lGL9m7Bv6Q/yPJOlSYUFUjGsridO1UOu8nSPaCtIajf7awiO5vXtTgyMn/xs xF2VOSUBKclro6YF64gc4Ul/8HO06ZNzkqyuTfROl2BdqqVMRcnhASoSQFh61OBsf3a9 QY3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062079; x=1700666879; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pIwdTD2wwefJLEoIwT50HmG4gzgICRaFcaA956d/E3E=; b=cSQRpCfMN16OTpBbilC9EnWj4QDZWKW87oFSdxrLib6YFN2ub0Ppiszh+PhXYOVND0 8ReumhRLWMviiBukfkyRJg0TKib3ltcJEWiWUL+ooP3F5hhs++OdLVhPWcVNbiBVaewg Tt2SOQXAi/QHEtiMdYRPPnqlRa7LvY7zr3NJ1a/voqaFuTSGpC7AqBeIspA7tKNGowTj XrP590zcbUzCTdKbgP7Nfrv2VstCm5LSkfG5qN68UjA918JzE2xf76c+oEClTO77umzT jMSDTgz95UrVIRLLa5f89XuYfS8LG7Zi5ETmv9+xy4iQAsCfi3VKKTb3f2DW3MR2hI3X paKg== X-Gm-Message-State: AOJu0YycYfhROqdKkTO+8TAKwL55j1VPzdLCOGYAa4qqqygi1+3V0KNM 4GS6vc4i0200KbBSQuu1GozgHJM6vRwmZv9uROA= X-Google-Smtp-Source: AGHT+IE1c3FTS23VgwV/ElRn+gXXcmeNiBx3kmLJ3vwJUOu4GKj4JqYDsWP8inku9PIVoehklqK5Gg== X-Received: by 2002:ac2:4d05:0:b0:509:8d88:b770 with SMTP id r5-20020ac24d05000000b005098d88b770mr9528869lfi.39.1700062079230; Wed, 15 Nov 2023 07:27:59 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:27:58 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH for-next 5/9] RDMA/rtrs-srv: Free srv_mr iu only when always_invalidate is true Date: Wed, 15 Nov 2023 16:27:45 +0100 Message-Id: <20231115152749.424301-6-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Since srv_mr->iu is allocated and used only when always_invalidate is true, free it only when always_invalidate is true. Signed-off-by: Md Haris Iqbal Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 4be0e5b132d4..925b71481c62 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -551,7 +551,10 @@ static void unmap_cont_bufs(struct rtrs_srv_path *srv_path) struct rtrs_srv_mr *srv_mr; srv_mr = &srv_path->mrs[i]; - rtrs_iu_free(srv_mr->iu, srv_path->s.dev->ib_dev, 1); + + if (always_invalidate) + rtrs_iu_free(srv_mr->iu, srv_path->s.dev->ib_dev, 1); + ib_dereg_mr(srv_mr->mr); ib_dma_unmap_sg(srv_path->s.dev->ib_dev, srv_mr->sgt.sgl, srv_mr->sgt.nents, DMA_BIDIRECTIONAL); From patchwork Wed Nov 15 15:27:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D50AC54FB9 for ; Wed, 15 Nov 2023 15:28:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231895AbjKOP2H (ORCPT ); Wed, 15 Nov 2023 10:28:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231641AbjKOP2E (ORCPT ); Wed, 15 Nov 2023 10:28:04 -0500 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61603101 for ; Wed, 15 Nov 2023 07:28:01 -0800 (PST) Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-9c603e2354fso197391666b.1 for ; Wed, 15 Nov 2023 07:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062080; x=1700666880; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5pS2yg4z7QyRI3Eox8JCayMQqgQn/h9ZIZWLeABnOtg=; b=CA0emUV58/S1kb+vDX1/YVeOeXlW8HG/105BtAONiJtxLeW31hgFZveyB8w5rEozkp EQIybMSih9mtc2L9n1T9NxEIIjOSsn0sbHWsq6cpUIz7AIyl6Kq6AvRq0LbLDlxMeF7B zPNZIk4lrinE5azuWN2DnaOwQw0T3qWmzFnSEW5jzWHRvDxt+7aszNolAjy2tBVFcPbB 4w5gjQvSVzfc0nZRuJU6tBrYmxy7ZFPL2R3W0awstxxpnKj0hf2/ZdyiCafb1PHoRInQ 8UrqKi0ZbBZGgRHnC1V/9yUl66sNY/nu5o6q4ySV22dIgO7zlMdIqdlRS+LIDkE0wU1o VtuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062080; x=1700666880; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5pS2yg4z7QyRI3Eox8JCayMQqgQn/h9ZIZWLeABnOtg=; b=T9vIWv7t+oy2mEWguvE4iasipQErjJh/YURJ2k+MBateJXBc0egH3MSLYMAeUFGpPv L0R/EvTTFpRbBx07lYV0e9cDqFlwyMSBbGbBCYf/u5cUtsChMn2/9k69yipBoVxSoFl1 +93hPAUBgc6MJd4nenrlta+sJZhGGYM4HN8yu+mBApRVAQgamzHi1qhNtvgIem9wLmjF ZT4g1dx/Ar64eWdBxSZFMn8Nq1aXnKQOki1owlvk1Jk1ppn5EDbVShgL4FUOQ8hMM/Qu 3eT9Mgpn7GahrkBG7cgaMPxtndXjPE/cHll0Xy0m5T1YTII7pr5oAhszL5uaRKKncf0k Xtvw== X-Gm-Message-State: AOJu0YzGZifPRrwp8nrWx+pIORgJ5Q/ifZxcXyyKt64zLqBP4nbyx+WO ojAzsTZyn9POlK2U0kydsZmft+8kAgL0gtOvBj0= X-Google-Smtp-Source: AGHT+IHczSDIxRj6Xa3wpKd5AcelE5lcYLtzBYtcQQ1NUOwank5S2LZR7tMOrC8xocbWLGyFtDN7MA== X-Received: by 2002:a17:907:868e:b0:9ae:5a56:be32 with SMTP id qa14-20020a170907868e00b009ae5a56be32mr5528013ejc.38.1700062079893; Wed, 15 Nov 2023 07:27:59 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:27:59 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Santosh Kumar Pradhan , Grzegorz Prajsner Subject: [PATCH for-next 6/9] RDMA/rtrs-srv: Destroy path files after making sure no IOs in-flight Date: Wed, 15 Nov 2023 16:27:46 +0100 Message-Id: <20231115152749.424301-7-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Destroying path files may lead to the freeing of rdma_stats. This creates the following race. An IO is in-flight, or has just passed the session state check in process_read/process_write. The close_work gets triggered and the function rtrs_srv_close_work() starts and does destroy path which frees the rdma_stats. After this the function process_read/process_write resumes and tries to update the stats through the function rtrs_srv_update_rdma_stats This commit solves the problem by moving the destroy path function to a later point. This point makes sure any inflights are completed. This is done by qp drain, and waiting for all in-flights through ops_id. Signed-off-by: Md Haris Iqbal Signed-off-by: Santosh Kumar Pradhan Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-srv.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 925b71481c62..1d33efb8fb03 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -1532,7 +1532,6 @@ static void rtrs_srv_close_work(struct work_struct *work) srv_path = container_of(work, typeof(*srv_path), close_work); - rtrs_srv_destroy_path_files(srv_path); rtrs_srv_stop_hb(srv_path); for (i = 0; i < srv_path->s.con_num; i++) { @@ -1552,6 +1551,8 @@ static void rtrs_srv_close_work(struct work_struct *work) /* Wait for all completion */ wait_for_completion(&srv_path->complete_done); + rtrs_srv_destroy_path_files(srv_path); + /* Notify upper layer if we are the last path */ rtrs_srv_path_down(srv_path); From patchwork Wed Nov 15 15:27:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF541C5AE4B for ; Wed, 15 Nov 2023 15:28:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229646AbjKOP2H (ORCPT ); Wed, 15 Nov 2023 10:28:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229989AbjKOP2F (ORCPT ); Wed, 15 Nov 2023 10:28:05 -0500 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0174419D for ; Wed, 15 Nov 2023 07:28:02 -0800 (PST) Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-5409bc907edso10590265a12.0 for ; Wed, 15 Nov 2023 07:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062080; x=1700666880; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jW7bALUgNM17QBq2CNcnNNwpzrRY3Ezp5tcU/Q9UywY=; b=efIk6KZyckCv46IfwFBOjwPd5rw4YGVFnhW0baWn8x29ZCSE/YTjJl40w2WWZG7ulz HeD39T/omyEvtCdmHQzA2l728pv1CoUhcjMlJl2hERSghzKA2h8hn9rs7cGn68GF3t67 IPG10CfrRX7Ed+5TWYYFzDdiHE115mZ8FaIbgfTPMBmOWpcXNAMOTyxJFs6ahTu7me+X Rr84GdobUiYdgpX8A6MqjotiXKqwbCtLrMZn4QZBxu5zLLQ6A/6y0CjViIrntWmy2mj3 5SqZ77nEvTcvNaMxt+CX35iNoABLeb8idIG+/v/FQudZgr3gbJIWmL4JaB50VMDWzm4E Cqcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062080; x=1700666880; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jW7bALUgNM17QBq2CNcnNNwpzrRY3Ezp5tcU/Q9UywY=; b=va2dMdZLOpaHyQgq1fv7dxeouw8YYNAUFuD9TMLqpnX6ylmZzdeTy9+yNSwMmioVCO l25gJQL88FjAzErXC2ZGIugw/pD8lGMH7Bny04ZS4sNcdnorzgLQaxr2SNDgWqmImdSj XXqhB01/6Esj/H1ErLDqJugKDNBbJQBD2liMhCuS+bG95cs0gr6rxs5zhCm7m+ucCdVU DrEGexdYfI/2kt5ygjGpEUaH/a9B7tq8dtm5TQPwkwPHyD1BepBT4G0yDLiRlsl0r8O+ +B7W2EsbQxT1tklJQQtKJuLFGboBOu/1HocMXFzak73fnaYkWJ5nbTTUDaqzlaDfcfN/ lbtw== X-Gm-Message-State: AOJu0Yw9DmELtYUsbcLrKvtieo6SSsPxTfwgmp42e3COjB8dW7D3Kxrg 9m2oUEV5IE5rKCmcgnAROjCBYHpxoCPm+D5QLls= X-Google-Smtp-Source: AGHT+IGY3wozw7Qqci1EWIIG01Xmokcqppl1xVnIv4T8Nj+r/fYY5N5keHZf462VJ0kReqXdUhv4Ug== X-Received: by 2002:aa7:d8d2:0:b0:53e:5957:aa4e with SMTP id k18-20020aa7d8d2000000b0053e5957aa4emr10425072eds.20.1700062080515; Wed, 15 Nov 2023 07:28:00 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.27.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:28:00 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH for-next 7/9] RDMA/rtrs-clt: Fix the max_send_wr setting Date: Wed, 15 Nov 2023 16:27:47 +0100 Message-Id: <20231115152749.424301-8-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jack Wang For each write request, we need Request, Response Memory Registration, Local Invalidate. Signed-off-by: Jack Wang Reviewed-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 83ebd9be760e..df10d29c3fe9 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -1699,7 +1699,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con) clt_path->s.dev_ref++; max_send_wr = min_t(int, wr_limit, /* QD * (REQ + RSP + FR REGS or INVS) + drain */ - clt_path->queue_depth * 3 + 1); + clt_path->queue_depth * 4 + 1); max_recv_wr = min_t(int, wr_limit, clt_path->queue_depth * 3 + 1); max_send_sge = 2; From patchwork Wed Nov 15 15:27:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456956 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85C4BC072A2 for ; Wed, 15 Nov 2023 15:28:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229989AbjKOP2I (ORCPT ); Wed, 15 Nov 2023 10:28:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbjKOP2G (ORCPT ); Wed, 15 Nov 2023 10:28:06 -0500 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3B9D101 for ; Wed, 15 Nov 2023 07:28:02 -0800 (PST) Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-53e04b17132so10677759a12.0 for ; Wed, 15 Nov 2023 07:28:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062081; x=1700666881; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KW3a8FMsS31luQZTd2cWIZE9yQSnciSmaftHSgr3QiA=; b=Pbl4xroQYvm+7aML2VCmKqy/ix8kWcTIGKOAnUnVq5UI5fb/6gYVDgYXRWNfVMC6HI KiuRIvV+gmtnevFYOt5trHVnGkElGpXOOAFMtcBAP3ZlIU6oviLnXjB59TDdt84BHtsi XfveWYihxIlMNUSM2KW/T0sjVsKJtqDFH4D55TGM1oXgumKnTGu31mUq7liqHVheqg2Q EP5AjhF3clW+Hh+oHKhSqZiZwKeu7bEk0N0ItmGNZxePk1au7yHiuvbiSuXdi06PsnN9 zxA4tdcqUv8hS0s9HJgJn6lzuCwsjXQW00EBAbMP6QfIY8vcX8SKQe8SIu0R+kIwNp7N U34Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062081; x=1700666881; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KW3a8FMsS31luQZTd2cWIZE9yQSnciSmaftHSgr3QiA=; b=Nq515ViEn1zI8v/8OSIdi5GfRYtWDkGYWCb7nselL+J4fOCLeJ194vAsS+Abr6XTFl 94J407esK0TkcUT3NV5ci5mEPm4bVNLoJ6Qp7pj1vTVb3trsds+7ZxS+zPhvD/YpsWJU T31Fzuwf7bS7IFSTkfX3fI5P6lMOMAqS5XnN7vAWXeW60QTYLVWZDj6eJYXjlqJq57hr 77hM2Ui9XK84hKM8BHO4h10GtF32GY8nJZtqmM++moKBZgv1drBFUqkMwRBqf4yk/Gpl I+J/QakRNw0huTphOWELHIXQNObZIeFmZaIwpOqhveBtnvTpZSTF35EuFBHmLB/aDXFl Op1g== X-Gm-Message-State: AOJu0YyGqjjkYXHTMIwnB8RY2vEcumyiCvDwcyRWgDv/y+WG/HjLIvMo QN+W8O16aDP/1yN7JWlNC3PpZzabhN4kGBq/+sM= X-Google-Smtp-Source: AGHT+IGegLniqBBH85Ky9JK/BGpLKWwt4QhEwUiTqHSljPIyblh2+CwC1FWMsm+Z5zDuDb3kLrPAvA== X-Received: by 2002:aa7:d847:0:b0:543:5281:2ad8 with SMTP id f7-20020aa7d847000000b0054352812ad8mr9834481eds.18.1700062081238; Wed, 15 Nov 2023 07:28:01 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.28.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:28:00 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Grzegorz Prajsner Subject: [PATCH for-next 8/9] RDMA/rtrs-clt: Remove the warnings for req in_use check Date: Wed, 15 Nov 2023 16:27:48 +0100 Message-Id: <20231115152749.424301-9-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jack Wang As we chain the WR during write request: memory registration, rdma write, local invalidate, if only the last WR fail to send due to send queue overrun, the server can send back the reply, while client mark the req->in_use to false in case of error in rtrs_clt_req when error out from rtrs_post_rdma_write_sg. Signed-off-by: Jack Wang Reviewed-by: Md Haris Iqbal Signed-off-by: Grzegorz Prajsner --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index df10d29c3fe9..8c5054d18402 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -384,7 +384,7 @@ static void complete_rdma_req(struct rtrs_clt_io_req *req, int errno, struct rtrs_clt_path *clt_path; int err; - if (WARN_ON(!req->in_use)) + if (!req->in_use) return; if (WARN_ON(!req->con)) return; From patchwork Wed Nov 15 15:27:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haris Iqbal X-Patchwork-Id: 13456957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 344ECC5AE4A for ; Wed, 15 Nov 2023 15:28:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229861AbjKOP2J (ORCPT ); Wed, 15 Nov 2023 10:28:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230460AbjKOP2H (ORCPT ); Wed, 15 Nov 2023 10:28:07 -0500 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9060018A for ; Wed, 15 Nov 2023 07:28:03 -0800 (PST) Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-53df747cfe5so10727182a12.2 for ; Wed, 15 Nov 2023 07:28:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1700062082; x=1700666882; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RUAK1l/2ByJ754WFSE3HJ7EYyscP+GOYGl/ZbcBVs30=; b=VpBNsU5qIpvCS2eBLamXK4Ise2qjdL6k4gY/u+BzO6wZ5sap+cLysC9OG1b45fsQDh vthkTaYnxl/P1Dd3AeyyqC12kqjjBsKpASwYNSuZh7HoP/JEwFxjs06eEsj7fLu2Zxct lIMsPcBHg6K2VtHSwQxTIazfB3AVZpfwlqmOhrTcUXQiEHwVQkdIHou6JxlD0v9N+dSs fD6X04Cj3UjSk8dUiECYteq4OYmRefnnhgKfDIwCqlv++C03w51eH6RtZYFTdeL1CH4D bDAnQ58aPWqXKuPxvFJ9+jEDX9tTazxEQguFtQVLTm34xeOK88OB/nerL9shLeS7siNI F+lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700062082; x=1700666882; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RUAK1l/2ByJ754WFSE3HJ7EYyscP+GOYGl/ZbcBVs30=; b=WoeaK5o2RXl1UvPJZdrioqIHml7FbzyjAh4GWJirpiNVJPOaF2KeaUGL0pZIjII85T 8ykA0UFqoWA1d65T1gVU2GUf7hvHLPxkHnRVrRubN7rqUvu67YM2zKoaLqqIhw2xx1JW VB2/h1EXfUMtSfxpptiJRObDZiLJDK/OjbA8S1fO7eWVQaWzSaxhMcJDDMcXUTCNZSqt /qfZXbyY6xNqnetOne53imqXJ1wdbytvnvFF8Ptd/RXcoUkWoEKx2bvwiZQrIE2kuiPr ajSjiL5shSaSg16Owpo06vc29CCQdkiuBdWTDWSMf37yWHpuWiQhA+lWgLeUHNlh6EYt aFIw== X-Gm-Message-State: AOJu0YxU2hevDS1iF1u6GP3jbzIFHDpyY8iIArfTqypgioZK0bx6xQTx 4z8mtluXUVn7uWmYmRkW6n0/15YIk3Uyvl6u+q4= X-Google-Smtp-Source: AGHT+IGavqx6F7s3IrlO1gFVclBLask02NYLaRK2a22rUABoxrooZNbSd3K4RXn8ElBwTur1fn3bhA== X-Received: by 2002:aa7:df83:0:b0:543:fa0:b4b6 with SMTP id b3-20020aa7df83000000b005430fa0b4b6mr10598177edy.33.1700062081846; Wed, 15 Nov 2023 07:28:01 -0800 (PST) Received: from lb01533.fkb.profitbricks.net ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id u6-20020a056402064600b00542da55a716sm6577349edx.90.2023.11.15.07.28.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 07:28:01 -0800 (PST) From: Md Haris Iqbal To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Supriti Singh , Grzegorz Prajsner Subject: [PATCH for-next 9/9] RDMA/rtrs: use %pe to print errors Date: Wed, 15 Nov 2023 16:27:49 +0100 Message-Id: <20231115152749.424301-10-haris.iqbal@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231115152749.424301-1-haris.iqbal@ionos.com> References: <20231115152749.424301-1-haris.iqbal@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Supriti Singh While printing error, replace %ld by %pe. %pe prints a string whereas %ld would print an error code. Signed-off-by: Supriti Singh Signed-off-by: Jack Wang Signed-off-by: Grzegorz Prajsner Signed-off-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 6 +++--- drivers/infiniband/ulp/rtrs/rtrs.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 8c5054d18402..493efbba2fe3 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -1391,9 +1391,9 @@ static int alloc_path_reqs(struct rtrs_clt_path *clt_path) clt_path->max_pages_per_mr); if (IS_ERR(req->mr)) { err = PTR_ERR(req->mr); + pr_err("Failed to alloc clt_path->max_pages_per_mr %d: %pe\n", + clt_path->max_pages_per_mr, req->mr); req->mr = NULL; - pr_err("Failed to alloc clt_path->max_pages_per_mr %d\n", - clt_path->max_pages_per_mr); goto out; } @@ -2061,7 +2061,7 @@ static int create_cm(struct rtrs_clt_con *con) RDMA_PS_IB : RDMA_PS_TCP, IB_QPT_RC); if (IS_ERR(cm_id)) { err = PTR_ERR(cm_id); - rtrs_err(s, "Failed to create CM ID, err: %d\n", err); + rtrs_err(s, "Failed to create CM ID, err: %pe\n", cm_id); return err; } diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index d80edfffd2e4..4e17d546d4cc 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -242,8 +242,8 @@ static int create_cq(struct rtrs_con *con, int cq_vector, int nr_cqe, cq = ib_cq_pool_get(cm_id->device, nr_cqe, cq_vector, poll_ctx); if (IS_ERR(cq)) { - rtrs_err(con->path, "Creating completion queue failed, errno: %ld\n", - PTR_ERR(cq)); + rtrs_err(con->path, "Creating completion queue failed, errno: %pe\n", + cq); return PTR_ERR(cq); } con->cq = cq;