From patchwork Mon Jul 4 06:00:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Li Zhijian X-Patchwork-Id: 12904733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FCD6C433EF for ; Mon, 4 Jul 2022 06:01:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231294AbiGDGBN (ORCPT ); Mon, 4 Jul 2022 02:01:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229706AbiGDGBM (ORCPT ); Mon, 4 Jul 2022 02:01:12 -0400 Received: from esa20.fujitsucc.c3s2.iphmx.com (esa20.fujitsucc.c3s2.iphmx.com [216.71.158.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 206B765AB for ; Sun, 3 Jul 2022 23:01:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj1; t=1656914472; x=1688450472; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=Pgx26XhWVX/1szP8P3up81NTZQEpH4ZmyViVtqfQ8eY=; b=wh6S7Bks5nv9wtWHejfG87Q1tc3Ml9H0QJc2KTTCwADZBuf38XOlZEFI VNDNqOh6L/ggkvN33H8LjLNRtcKcOwKkkJ6sUF0qEttlR/7LzIRkBQYAt UVGMYCK4hU1Oezc6AEku2lj+FXIcnQN8E+LW5xp2MVThWPztDVonFPfUR gKNpNicSbSzQRIKoAkAAX+zpBaM9jR1mYXlc1P51SH2H/IRoKzzptsngP iRDDgFHNIfVfhCMfUxjbvjqmWlJaoHXgTdakRsdLVFCkA/NHDlBZZj0wQ Pg6e6YXM6tpCM6Nheo5PEvZINHNrLDtSccCEphlmMjP7aejHBU/3xmfrs w==; X-IronPort-AV: E=McAfee;i="6400,9594,10397"; a="59469711" X-IronPort-AV: E=Sophos;i="5.92,243,1650898800"; d="scan'208";a="59469711" Received: from mail-tycjpn01lp2171.outbound.protection.outlook.com (HELO JPN01-TYC-obe.outbound.protection.outlook.com) ([104.47.23.171]) by ob1.fujitsucc.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Jul 2022 15:01:09 +0900 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bF4Dq+u3LN9gA6DhDfCUSHMHoHpdwCtRqJD0vRTDeKoWJ7DUX/YsFp27L3A5rWd23dxkVoXwAe+0uJNolJ9EwwwPCYXASkUY/fZEQDIevbr/rYXaluaLdo0KCt/tmgxZtm18pDxVFtcRepFSWakSJCXWqwesyxSL0qdI+yRWPBg+NXeTRqXeJ4VqtTZkfNFA+9xk/WyhjtEDd47TOZhkAdj5aYMtPUdGaAEm0Jh7ndqw1ijQeLxA+GYUyvU1l2jsNQbhtUGfX3iNe3R+roF7DOiNeAMgIusEFhr76r26aE6WNbjnA+ek/KhfXhaCjZQ8ZowvyzmC8wLDvFytoYeC2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Pgx26XhWVX/1szP8P3up81NTZQEpH4ZmyViVtqfQ8eY=; b=KHt2W23XJ2G0gnWPP9PD77KORtAtB/xdw7ehs0j+x5Sw+u5SeKrXgkdegYQYJ3DxScUWAOhLd1wgFlZkJRDRd+Pq59du+71Yn3k8EcQyzjQqx3IjZAVzXxwwR1iSDq7tqWyoNgSgpIaJoNSI8/BfHOiGtuZM2FAhZgEGmvg+itXqfJY8bkQVcy4mK/R12cvPyI6sB+KunHbhHQ7vDAL4+fFFJT+PfirB9fS7AbgtVh2QnTGGbwMLqhZXh/ZqOS84X6lfqvPAajhgPdP41LqXGLWrAN3o6gDYaDclynhUMj7M1CFDgQNJJN3/Yh6pSlgBoCzcoo/Sn+PDH98zBGCJAw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fujitsu.com; dmarc=pass action=none header.from=fujitsu.com; dkim=pass header.d=fujitsu.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fujitsu.onmicrosoft.com; s=selector2-fujitsu-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pgx26XhWVX/1szP8P3up81NTZQEpH4ZmyViVtqfQ8eY=; b=Kfizh63dk3fTx4PsBoDI7lQaj2kTNWv0ZuDiGugPvFVOP3kBcyzoHeMyIs89iKbLIQJnXM7eP+kpzhr9JOqlreFfgAtOLC5tiOFRRsEpM0OSynlcoY2bEJlu6MOJh4nyXxsdiB7TgxArfYiId8bYQaZl3i2gYrlhg/YTcpqgY1k= Received: from TYCPR01MB9305.jpnprd01.prod.outlook.com (2603:1096:400:196::10) by TY1PR01MB1753.jpnprd01.prod.outlook.com (2603:1096:403:1::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5395.18; Mon, 4 Jul 2022 06:00:57 +0000 Received: from TYCPR01MB9305.jpnprd01.prod.outlook.com ([fe80::8d7a:baa8:3b18:cd93]) by TYCPR01MB9305.jpnprd01.prod.outlook.com ([fe80::8d7a:baa8:3b18:cd93%6]) with mapi id 15.20.5395.020; Mon, 4 Jul 2022 06:00:57 +0000 From: "lizhijian@fujitsu.com" To: Yanjun Zhu , Jason Gunthorpe , Haakon Bugge , "linux-rdma@vger.kernel.org" , Bob Pearson CC: Cheng Xu Subject: [PATCH v5 3/4] RDMA/rxe: Split qp state for requester and completer Thread-Topic: [PATCH v5 3/4] RDMA/rxe: Split qp state for requester and completer Thread-Index: AQHYj2tt9y4n+Wr7F0eaR79jUYL1uQ== Date: Mon, 4 Jul 2022 06:00:56 +0000 Message-ID: <20220704060806.1622849-4-lizhijian@fujitsu.com> References: <20220704060806.1622849-1-lizhijian@fujitsu.com> In-Reply-To: <20220704060806.1622849-1-lizhijian@fujitsu.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.31.1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=fujitsu.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a5f4077a-6cd2-48e1-041f-08da5d828ffd x-ms-traffictypediagnostic: TY1PR01MB1753:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: VtYyxYdVoU2WJGh6qFUsa7a+lA+WfuaROFlYlS5QCy72yoZIQqUixEfo+7Tpg9j9FWbWGZES8YjR/3O8nAXJgDrsrmqlEv+7wRWFB5Ly/SrUdOustTaoFUPkqr3DAlGE6EBVg+Y1hfPzvjgRO1Byr+N/q9YlU0nYDsp90r7IVJtMFDYmcqtEeyoD+dQWLtbSg2OPNncdbJjdUHeBfS/8D1l75RHQ0lq2+n/k5y4nMWF4IvQzVVKqJlzAdCs1+duHqRQ7XBzi16SSMKgfpX6OpUQrASQwc//qFF75wpKOiYV9B267cvNtSJHR1zNfEf5yvZpo5A1FNVjahUdYrtUIICbMl5x+RNj6VIvjqZOn1c9DEkeDiql92jUZXs5Jkcc3T/7baRlf0a9RF6W1yi7w1vuJhTql54kyDclz5DNEu+MtavRZJZrjZsXd6PSan66qsznjYGWZ8ekbBQ8BRK34w00yiYmF3fnGFUfj0QQQhEHPiPZPc7Hi5x5GG9vRRTzcd/6d44qykaJfroLRtqhJgv02B6xRS8V4xUsZSZt1J6NB++qIwHlkUybFsBESVkKiSIQppmN03vy+VFnU9NWMHurEOkOurEeO8wmt8XraVZv8txWMO0ptc+dWYa+e1S4ATWz5NoJKjwegg3GOZ3ZTbmzwiVXSdUR5S2pHl1x6c1zKwapRkIOd1hOt5sk1eoFXoAIl88iQkok5dB0G1ylZL8mcfC09ivnW8S0syscNAyeKxov/h0dn/HW/8oEeDIGtvibiP1y/Y3kNJFATPRy6/GtD1POdbJuR+5+vySNscLrXte5ST687gcML9QsDy8wY x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:TYCPR01MB9305.jpnprd01.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(39860400002)(346002)(136003)(376002)(366004)(396003)(2616005)(186003)(1076003)(38070700005)(38100700002)(64756008)(36756003)(71200400001)(8676002)(66446008)(316002)(66476007)(66946007)(76116006)(110136005)(66556008)(83380400001)(85182001)(91956017)(4326008)(5660300002)(6512007)(6506007)(478600001)(6486002)(8936002)(82960400001)(122000001)(86362001)(41300700001)(26005)(2906002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?eucgb2312_cn?b?MTdSUXZHd1ZXM2ZsRGxCMnlk?= =?eucgb2312_cn?b?NFV0dVBDUzViNWRRS0VmUWVnRGRsZWNCUi80YzFBbmhpNml1T3RxVERia3o3QVZt?= =?eucgb2312_cn?b?SGFsVDhTWGFrQ1Y1SS93ZGVSMGRxSVlzWHdIYm10aUZ4K0RWeVQxVjE1NHJDaHhM?= =?eucgb2312_cn?b?SU1vSjV4NGpJUnBzYTE1NVRST3Ryc0kycXpycWdKZjZiNUM4UFNzSXEvNGZjL21H?= =?eucgb2312_cn?b?ZjlSMCtUZG51R09CNDNjNi90YjlobHRrMTNrWkZtZ1VBdXRFN0tUN0VydG5XNGZJ?= =?eucgb2312_cn?b?VUFHZlJOT3RBWjVabGtrNTZhMzRmTTlqZjZEM1gwdmVLZXdtL1IzY1hhb3RhSUJE?= =?eucgb2312_cn?b?dzY1dEZzRmpONEV5TUpWeHFMQkNQV01UV1lXbVNpbW1kUXczMmtHckFFdUhkcmZm?= =?eucgb2312_cn?b?K3VPazFkS2h2QWw1WXdaaWlrVDg2OXl1WlQ1UTZoQldXZ1crcFVBNGRNaXlZZ3No?= =?eucgb2312_cn?b?Z3BjdFRSMWpXaDVENStxVkxRTG9XVkR1eGY3TTYxdzR6MEt6cmFwVDlvNnNMOHRO?= =?eucgb2312_cn?b?Y3Y1WG5xc1phbEtRQTRHcEduSDlaSHlwQmxPeCsvZ1VKVkRnTktMUG9XN2hRaHlO?= =?eucgb2312_cn?b?dWVhRDhYQzhidkY1TnpoMDVsQTRraXNXa1libTc4TU1TSUJZMmROYzhQVWRwVlo2?= =?eucgb2312_cn?b?Y0VEMVdVd3h2NUEzcmVjanNiZFMvQkhwVG81ZFFPQ1l6K0F6TE83Y3lYeXUvVG1R?= =?eucgb2312_cn?b?K3NyZUZOTjZwYjd4V3FwWkNHM0I3Uy9TSWhpWXhyTDROYTBLbTRodGN3MEtuc053?= =?eucgb2312_cn?b?aGNhUGw1YnJIYWc0dW9jZUlaM2tXM3pzOEN3NkZKSjc1OEFyWDdCVmc1eThGb0xX?= =?eucgb2312_cn?b?cHRONEZQMktPK3ROckYxT0IvNzVCR2tTdUIrSkhjWUhjbk9DdWxzNkcwekdkbXBl?= =?eucgb2312_cn?b?UTYxWFg0dXZCeDVXRXY4SHlJSXBmZGdKdkErNGFOcmFLU2szWVBqckViSUlnbDVR?= =?eucgb2312_cn?b?Zm5YVzcyUjJiRFVoemhFTU80aFBJSm5VUW4weVRqRTQvNUdLM1pqZFNPRlNtVXFC?= =?eucgb2312_cn?b?Rk1nM21TYjkxSW1vRytxVExuNnhLREJaTVFCUm9rZHI0bm5LRlRYbkpXbXNkNjFF?= =?eucgb2312_cn?b?RFF4aHhHODRYNThLTTBiSkdEbHNJNW5QbXVqU0czR3dnWHlaTmtVZzVPMXJJZkxp?= =?eucgb2312_cn?b?N3ppYUF6aWFuUDV2bStnMlRtbjBWUHcySTlLNHFacTBKOTBBS0Y4TVdkdzE1dFpP?= =?eucgb2312_cn?b?dzA0a1U0TDludFB3Y3lZUUhmbndHK01tNzZnaXNRNHVLVVFpZ09QWDZRU1p0UTZJ?= =?eucgb2312_cn?b?QWhXanZSVk1FcVh1UGpUaHBTaWh1S2dkWllRaFdzVFRVNDRaUU5TVFhGQk5vZDM4?= =?eucgb2312_cn?b?YXllQzN0VUd2dVliR1dDY3Y5UXZ4NGxieVB1Yy81MGNtc0lZc3N0RU5YTC9aL0pW?= =?eucgb2312_cn?b?RzdpZ05JTVN0VHhuMGVobXJpblAwb1FxU24vekZDY2tzWU1oRzVTcTdvdVFLTkxs?= =?eucgb2312_cn?b?NVVkNmZBRnFmZkN0YzNOanBTN0RzRTlWUG5pL2JpMTlQT2dXRmExQ2pPNWhYSHNs?= =?eucgb2312_cn?b?bFZlWVBQMnRjVldmdHdvMzJiZDZYVTdCazZwS1RTay9HZHVKWndRdThkTFg2a043?= =?eucgb2312_cn?b?cXgvanZaNkcvYnY5MngvUHRnNVlrY2xHOTNLRElDWk9PTGt6L2hia3dBWDlvWGRk?= =?eucgb2312_cn?b?TGVURFEzY1J4eE1JMEU2VGlIVlVlQ2VYRnhSNGFYU1pNTm5qd01EMU5wMWFEWlhW?= =?eucgb2312_cn?b?ak1ocFNQVElkRlpHZEZYRGUwVENWK1E0bDQ3TmpQT2lLTW9SdTd4bEdIVHowNmRm?= =?eucgb2312_cn?b?YlhhSzJhM2xnMkRPejNLSE9PcDkyK1lmZVFGVU45bVc3Y3JWcHdHa1UwZllDbGZJ?= =?eucgb2312_cn?b?WWV1Ym9rR0JuZUJoSGQzdm9PSG4zandkM1ZSU3FpUnROaWpVU2tQT1lUeFRRUkhm?= =?eucgb2312_cn?b?UUFJMmhDVTdGME1hZ2VnZ3BzOSt3eW1OTGk3SUxYTHBMRk03bnQyTGF3SnhtOUdZ?= =?eucgb2312_cn?b?ckdxVnQxODRpMGxmYkd6bCtwZkY4SE5sbnhtWUhmcjVjemUwZnI2eHYwb0xjZXRT?= =?eucgb2312_cn?b?MUczQlZEMElwZklDTHVUSTJCaFhDVDhTck5mT044MnJRV0V4WG8rT0FqQVpmS01z?= =?eucgb2312_cn?b?WklqNE50bStNMlJENTFONUhkeHc9PQ==?= MIME-Version: 1.0 X-OriginatorOrg: fujitsu.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: TYCPR01MB9305.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a5f4077a-6cd2-48e1-041f-08da5d828ffd X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jul 2022 06:00:56.8889 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a19f121d-81e1-4858-a9d8-736e267fd4c7 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: UO5DAOx729DHw0tBmKb6HBe6yjZ96i1pPuWpI4q0PLK3cxR+nW8MKXoDbLrxDlzKYkcmMBwwbGfGGJ5U79mdbg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TY1PR01MB1753 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Bob Pearson Currently the requester can continue to process send wqes after an local qp operation error is detected because the setting of the qp state to the error state is deferred until later. This patch splits the qp state for the completer and requester into two separate states and sets qp->req.state = QP_STATE_ERROR as soon as the error is detected before another wqe can be executed. Signed-off-by: Bob Pearson --- V4: new patch --- drivers/infiniband/sw/rxe/rxe_comp.c | 6 +++--- drivers/infiniband/sw/rxe/rxe_qp.c | 5 +++++ drivers/infiniband/sw/rxe/rxe_req.c | 1 + drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index da3a398053b8..0b68630a3e49 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -565,10 +565,10 @@ int rxe_completer(void *arg) if (!rxe_get(qp)) return -EAGAIN; - if (!qp->valid || qp->req.state == QP_STATE_ERROR || - qp->req.state == QP_STATE_RESET) { + if (!qp->valid || qp->comp.state == QP_STATE_ERROR || + qp->comp.state == QP_STATE_RESET) { rxe_drain_resp_pkts(qp, qp->valid && - qp->req.state == QP_STATE_ERROR); + qp->comp.state == QP_STATE_ERROR); ret = -EAGAIN; goto done; } diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 22e9b85344c3..a95d3b49ae20 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -230,6 +230,7 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, QUEUE_TYPE_FROM_CLIENT); qp->req.state = QP_STATE_RESET; + qp->comp.state = QP_STATE_RESET; qp->req.opcode = -1; qp->comp.opcode = -1; @@ -490,6 +491,7 @@ static void rxe_qp_reset(struct rxe_qp *qp) /* move qp to the reset state */ qp->req.state = QP_STATE_RESET; + qp->comp.state = QP_STATE_RESET; qp->resp.state = QP_STATE_RESET; /* let state machines reset themselves drain work and packet queues @@ -552,6 +554,7 @@ void rxe_qp_error(struct rxe_qp *qp) { qp->req.state = QP_STATE_ERROR; qp->resp.state = QP_STATE_ERROR; + qp->comp.state = QP_STATE_ERROR; qp->attr.qp_state = IB_QPS_ERR; /* drain work and packet queues */ @@ -689,6 +692,7 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, pr_debug("qp#%d state -> INIT\n", qp_num(qp)); qp->req.state = QP_STATE_INIT; qp->resp.state = QP_STATE_INIT; + qp->comp.state = QP_STATE_INIT; break; case IB_QPS_RTR: @@ -699,6 +703,7 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, case IB_QPS_RTS: pr_debug("qp#%d state -> RTS\n", qp_num(qp)); qp->req.state = QP_STATE_READY; + qp->comp.state = QP_STATE_READY; break; case IB_QPS_SQD: diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 6d2742997e1b..ad25290e393d 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -773,6 +773,7 @@ int rxe_requester(void *arg) /* update wqe_index for each wqe completion */ qp->req.wqe_index = queue_next_index(qp->sq.queue, qp->req.wqe_index); wqe->state = wqe_state_error; + qp->req.state = QP_STATE_ERROR; __rxe_do_task(&qp->comp.task); exit: diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index ac464e68c923..bbfffe243fd6 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -129,6 +129,7 @@ struct rxe_req_info { }; struct rxe_comp_info { + enum rxe_qp_state state; u32 psn; int opcode; int timeout;