From patchwork Tue Oct 22 03:18:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13845057 Received: from tiger.tulip.relay.mailchannels.net (tiger.tulip.relay.mailchannels.net [23.83.218.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2C9F13E88C; Tue, 22 Oct 2024 03:29:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=23.83.218.248 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567781; cv=pass; b=UKivaN4sSbaSEPQ592DZrorFhgz4rU3/jgBSDN8BZPFGePcmkPK8oBXKZMKkc2YCuTmVAGKJk4NHyrFyIaxyMJdfDRZNN2XzCorxbWi1fwvVWmYnanxQ4bXwyM7nIUpugZfskqas6CN0tOwObVbxSrZtzExp6b7LIY3kxxP7dmI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567781; c=relaxed/simple; bh=4eQKBL4lamxSiFpROmuWhyo4tQG6isTNpCHTeBH9oLU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n6wSkZT8ev9kWLdGJQW17n/iBrguayw/LHTndRoXtfzL9YtJlEdusbJLg7KHOBurg2t1Jys77cbRcyZr36pRMYcAKgJBEAjvru/4IlhwzLducw0C8IjJVKzWTTaGOkMmUr+8fa2EBA7sZ8BR+vkip02hb1btHyWtdNoWALAjqC0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=stgolabs.net; spf=pass smtp.mailfrom=stgolabs.net; dkim=pass (2048-bit key) header.d=stgolabs.net header.i=@stgolabs.net header.b=DPFsl2Ma; arc=pass smtp.client-ip=23.83.218.248 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=stgolabs.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=stgolabs.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=stgolabs.net header.i=@stgolabs.net header.b="DPFsl2Ma" X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 934CAC2F31; Tue, 22 Oct 2024 03:21:15 +0000 (UTC) Received: from pdx1-sub0-mail-a259.dreamhost.com (100-98-255-245.trex-nlb.outbound.svc.cluster.local [100.98.255.245]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 14394C2AEA; Tue, 22 Oct 2024 03:21:15 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1729567275; a=rsa-sha256; cv=none; b=KUdo40p3jNHxG6YFZ1+IEY/PeL4nytf3nxEqk46oEXE0ytk1uxyeMWs7XL9Esj0QLZCSM5 WwB+IFyVXSQ7gUSih5RcD+Fl8bY2YoYZALlumNZbBgnrJIOGvScTRCyKyPs9+6MHcm83bo tukGliaR6keK6Op40qQDj2QC4xL6BtQfB5urRiA5QTVNMGzFGi0AX/wp7tVl9B4g08ZXMf Yzt0KbX6awZcUBNP7E14dUAjL7EVpbQIFkVNi+q32jQIt9aeHqvn2bLxBuSpyQlNpVFTf1 tjIJ7kGWzAB33vewvM5dW65MKJFj1BPpzz2PFHhj+igbNpkWuLSxnZkoOvLANQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1729567275; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CCk2wco2OeAmTATlemc+OtbF0EDfORUEfM7CkmZnhEQ=; b=Ni3wAT6zV866cSi6uttkPTiWddFwM/b1y2t8TMbD5VfWOVy4bj7osdsDsIuT2DCcQVYQew EXweWBpXNKhUWvIRpbRkhk0ZkBC+DvsJSRXYMW3o8tQipGvMILozr6FXHHlMMN7bwWnBWi T3I64mwi3q6BnlDNT/RGu9mzsbLbvqWdPbLsJis+oxD9+ba2TWl8wQWkUMdsMPR8Wuvln3 BfAwyGjPGOWVVRnJf9alSiSDtDrQHTdF6o91tv1357zGAM5L68iaJujFEjJRYpUgvpLekK wIyoKMbR8uuLTewnfy0Tnia88vAATV6WV/az5S2O6rJ3N6HjgA4+zPvVqvztCw== ARC-Authentication-Results: i=1; rspamd-75d86777c9-lf85w; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Shoe-Grain: 0f648e5473dd4b9e_1729567275425_761448160 X-MC-Loop-Signature: 1729567275425:4151781414 X-MC-Ingress-Time: 1729567275424 Received: from pdx1-sub0-mail-a259.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.98.255.245 (trex/7.0.2); Tue, 22 Oct 2024 03:21:15 +0000 Received: from localhost.localdomain (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a259.dreamhost.com (Postfix) with ESMTPSA id 4XXcs56S12z1p; Mon, 21 Oct 2024 20:21:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1729567274; bh=CCk2wco2OeAmTATlemc+OtbF0EDfORUEfM7CkmZnhEQ=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=DPFsl2MacD34Rlsa/zY/EIA8ah8KLL4cd7PqWbwn5uRWvtqHWxBFtq7whRX8ktVEy hyRDxWkqz/ljk1crsD1bsLOsbj8Ow2doAKyXsLYC4kwFfVc9LtK/hRd5JwCO8qJB64 Jg6kkSATiArDhS23csbOjG7GQrRtHyp6QA6CvYDBRwionoF95Er2GsRIZXkNhlPrLR NtV1UbzuNYMfCzllR9GbwvI+VHrrAe1kyYiC2RyXT0eCY/fSbFXfXc/MiZLftUQ0v0 W36BBIn4P+I5gBpUZhcx0O6iy7ztWoH7Uq7piMbqhP44bikeoQuSe81PRNTm32u0rQ uxUieHO01KLtQ== From: Davidlohr Bueso To: dan.j.williams@intel.com, dave.jiang@intel.com Cc: jonathan.cameron@huawei.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, fan.ni@samsung.com, a.manzanares@samsung.com, sthanneeru.opensrc@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, Ravis.OpenSrc@micron.com, sthanneeru@micron.com, dave@stgolabs.net, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] cxl/pci: lockless background synchronous polling Date: Mon, 21 Oct 2024 20:18:07 -0700 Message-ID: <20241022031809.242591-2-dave@stgolabs.net> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20241022031809.242591-1-dave@stgolabs.net> References: <20241022031809.242591-1-dave@stgolabs.net> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For non-sanitize background commands we rely on holding the mbox_mutex throughout the duration of the operation. This causes other incoming commands to queue up behind, and interleaving executions while the background command is timesliced by the user. However, in order to support the mbox request cancel background operation command, the lock will need to be available to actually perform the request. Semantically this allows other commands to many times be at the mercy of hardware returning the respective error. Potentially users would be exposed to changes in the form of errors instead of commands taking longer to run - but this is not forseen as a problem. In order to not loose sync with the hardware, introduce a mailbox atomic that blocks any other incoming bg operations while the driver is still polling (synchronously) on the current one. Signed-off-by: Davidlohr Bueso --- drivers/cxl/core/mbox.c | 1 + drivers/cxl/cxlmem.h | 13 +++++ drivers/cxl/pci.c | 114 +++++++++++++++++++++++----------------- include/cxl/mailbox.h | 2 + 4 files changed, 82 insertions(+), 48 deletions(-) diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index 5175138c4fb7..9af3cd16d23d 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1429,6 +1429,7 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host) cxl_mbox->host = host; mutex_init(&cxl_mbox->mbox_mutex); + atomic_set(&cxl_mbox->poll_bgop, 0); rcuwait_init(&cxl_mbox->mbox_wait); return 0; diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 2a25d1957ddb..b933fb73ef8a 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -557,6 +557,19 @@ enum cxl_opcode { CXL_MBOX_OP_MAX = 0x10000 }; +static inline bool cxl_is_background_cmd(u16 opcode) +{ + switch (opcode) { + case CXL_MBOX_OP_TRANSFER_FW: + case CXL_MBOX_OP_ACTIVATE_FW: + case CXL_MBOX_OP_SCAN_MEDIA: + case CXL_MBOX_OP_SANITIZE: + return true; + default: + return false; + } +} + #define DEFINE_CXL_CEL_UUID \ UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96, 0xb1, 0x62, \ 0x3b, 0x3f, 0x17) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 188412d45e0d..f2378604669b 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -278,29 +278,15 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_mailbox *cxl_mbox, mbox_cmd->return_code = FIELD_GET(CXLDEV_MBOX_STATUS_RET_CODE_MASK, status_reg); - /* - * Handle the background command in a synchronous manner. - * - * All other mailbox commands will serialize/queue on the mbox_mutex, - * which we currently hold. Furthermore this also guarantees that - * cxl_mbox_background_complete() checks are safe amongst each other, - * in that no new bg operation can occur in between. - * - * Background operations are timesliced in accordance with the nature - * of the command. In the event of timeout, the mailbox state is - * indeterminate until the next successful command submission and the - * driver can get back in sync with the hardware state. - */ if (mbox_cmd->return_code == CXL_MBOX_CMD_RC_BACKGROUND) { - u64 bg_status_reg; - int i, timeout; - /* * Sanitization is a special case which monopolizes the device * and cannot be timesliced. Handle asynchronously instead, * and allow userspace to poll(2) for completion. */ if (mbox_cmd->opcode == CXL_MBOX_OP_SANITIZE) { + int timeout; + if (mds->security.sanitize_active) return -EBUSY; @@ -311,44 +297,19 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_mailbox *cxl_mbox, schedule_delayed_work(&mds->security.poll_dwork, timeout * HZ); dev_dbg(dev, "Sanitization operation started\n"); - goto success; - } - - dev_dbg(dev, "Mailbox background operation (0x%04x) started\n", - mbox_cmd->opcode); - - timeout = mbox_cmd->poll_interval_ms; - for (i = 0; i < mbox_cmd->poll_count; i++) { - if (rcuwait_wait_event_timeout(&cxl_mbox->mbox_wait, - cxl_mbox_background_complete(cxlds), - TASK_UNINTERRUPTIBLE, - msecs_to_jiffies(timeout)) > 0) - break; - } - - if (!cxl_mbox_background_complete(cxlds)) { - dev_err(dev, "timeout waiting for background (%d ms)\n", - timeout * mbox_cmd->poll_count); - return -ETIMEDOUT; + } else { + /* pairs with release/acquire semantics */ + WARN_ON_ONCE(atomic_xchg(&cxl_mbox->poll_bgop, + mbox_cmd->opcode)); + dev_dbg(dev, "Mailbox background operation (0x%04x) started\n", + mbox_cmd->opcode); } - - bg_status_reg = readq(cxlds->regs.mbox + - CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); - mbox_cmd->return_code = - FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_RC_MASK, - bg_status_reg); - dev_dbg(dev, - "Mailbox background operation (0x%04x) completed\n", - mbox_cmd->opcode); - } - - if (mbox_cmd->return_code != CXL_MBOX_CMD_RC_SUCCESS) { + } else if (mbox_cmd->return_code != CXL_MBOX_CMD_RC_SUCCESS) { dev_dbg(dev, "Mailbox operation had an error: %s\n", cxl_mbox_cmd_rc2str(mbox_cmd)); return 0; /* completed but caller must check return_code */ } -success: /* #7 */ cmd_reg = readq(cxlds->regs.mbox + CXLDEV_MBOX_CMD_OFFSET); out_len = FIELD_GET(CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK, cmd_reg); @@ -378,11 +339,68 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd) { int rc; + struct cxl_dev_state *cxlds = mbox_to_cxlds(cxl_mbox); + struct device *dev = cxlds->dev; mutex_lock_io(&cxl_mbox->mbox_mutex); + /* + * Ensure cxl_mbox_background_complete() checks are safe amongst + * each other: no new bg operation can occur in between while polling. + */ + if (cxl_is_background_cmd(cmd->opcode)) { + if (atomic_read_acquire(&cxl_mbox->poll_bgop)) { + mutex_unlock(&cxl_mbox->mbox_mutex); + return -EBUSY; + } + } + rc = __cxl_pci_mbox_send_cmd(cxl_mbox, cmd); mutex_unlock(&cxl_mbox->mbox_mutex); + if (cmd->return_code != CXL_MBOX_CMD_RC_BACKGROUND) + return rc; + + /* + * Handle the background command in a synchronous manner. Background + * operations are timesliced in accordance with the nature of the + * command. + */ + if (cmd->opcode != CXL_MBOX_OP_SANITIZE) { + int i, timeout; + u64 bg_status_reg; + + timeout = cmd->poll_interval_ms; + for (i = 0; i < cmd->poll_count; i++) { + if (rcuwait_wait_event_timeout(&cxl_mbox->mbox_wait, + cxl_mbox_background_complete(cxlds), + TASK_UNINTERRUPTIBLE, + msecs_to_jiffies(timeout)) > 0) + break; + } + + /* + * In the event of timeout, the mailbox state is indeterminate + * until the next successful command submission and the driver + * can get back in sync with the hardware state. + */ + if (!cxl_mbox_background_complete(cxlds)) { + dev_err(dev, "timeout waiting for background (%d ms)\n", + timeout * cmd->poll_count); + rc = -ETIMEDOUT; + goto done; + } + + bg_status_reg = readq(cxlds->regs.mbox + + CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); + cmd->return_code = FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_RC_MASK, + bg_status_reg); + dev_dbg(dev, + "Mailbox background operation (0x%04x) completed\n", + cmd->opcode); +done: + atomic_set_release(&cxl_mbox->poll_bgop, 0); + } + return rc; } diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h index bacd111e75f1..23282a3c44f1 100644 --- a/include/cxl/mailbox.h +++ b/include/cxl/mailbox.h @@ -13,6 +13,7 @@ struct cxl_mbox_cmd; * (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register) * @mbox_mutex: mutex protects device mailbox and firmware * @mbox_wait: rcuwait for mailbox + * @poll_bgop: current background operation being polled on * @mbox_send: @dev specific transport for transmitting mailbox commands */ struct cxl_mailbox { @@ -20,6 +21,7 @@ struct cxl_mailbox { size_t payload_size; struct mutex mbox_mutex; /* lock to protect mailbox context */ struct rcuwait mbox_wait; + atomic_t poll_bgop; int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd); }; From patchwork Tue Oct 22 03:18:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13845045 Received: from bumble.maple.relay.mailchannels.net (bumble.maple.relay.mailchannels.net [23.83.214.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0397C198A22; Tue, 22 Oct 2024 03:21:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=23.83.214.25 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567286; cv=pass; b=RjmqcvFFTiByopCU2yUtMXIcz7S/6vhSAStEGpHkKL6s0zVr698RHaWhB9J7Mp5wuyPJvDfNfVvAWYV8UoDYDWypuWuyORXGCFFkIFFTITwT1qK6Qf1tfk4pa6GPWOceCoDJIxha21ySDadlSAkloKArq23mH3WGMIBFmgKiGNs= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567286; c=relaxed/simple; bh=bo68H4faO1bWI30btJNP3hJMZkRXPCXCvTz/JcusREQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WPmKlZh6OqCEuZNAy54QYtuDDvFb122VtsNLVBoDq2jPAArAV1TbIdPDtHPbFfyvKVSF6uuQJg4opjnezhHhfTUooMSV9cJ0MkUsLwvTrE3dVXyIt1VvPdPH3A0cqkYgsAMUnSRfChUuGazOc90v24tLMPIvOHN7JjbiTqDMevg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=stgolabs.net; spf=pass smtp.mailfrom=stgolabs.net; dkim=pass (2048-bit key) header.d=stgolabs.net header.i=@stgolabs.net header.b=JRymzZpZ; arc=pass smtp.client-ip=23.83.214.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=stgolabs.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=stgolabs.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=stgolabs.net header.i=@stgolabs.net header.b="JRymzZpZ" X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id F216D3220B1; Tue, 22 Oct 2024 03:21:16 +0000 (UTC) Received: from pdx1-sub0-mail-a259.dreamhost.com (trex-9.trex.outbound.svc.cluster.local [100.99.180.193]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 53225321FD8; Tue, 22 Oct 2024 03:21:16 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1729567276; a=rsa-sha256; cv=none; b=x5MYd7IflK0gA1sN+/fI2dIpgpV9FWonZdfVgrfbEJ0ZH2L66cJmqjCrqMGfvi0VDKrtCP Uv/nzf0HX7gdFXcrCmJkhll79bLe+yVPLrA01eNx0JxAdAhpfQv+4TGNTsUOxEAzr81c32 kA/LFE8c5U7A+Ox3h3Hc9RE19TaYB19Wvvi0Se0CHp6S6wDtu8PZwuWe1Hrsx4EwGCxq7v nb7NSeXKPc1KnXt3D6i8M//WqBu7t8ozZTcqdVF2Uk3fz6YxJYi+Y8ggPIIpiuDR+1QAcK cou/W+oSSp5/h0hS0lCoL9jN4jPacoRInCuPV3PV47UXt460GIh43iXqd/DyZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1729567276; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xWuXI1STFWVB6JAkyrFpy2OWqAiJ2oDnJCL4Dlt8unk=; b=OFoaxKFyvrq+ReC9+/o3l5qBF+Tb4fSyoTT2jpx5th1pDwIdP0NfCUleeK1LJPnzhe4FxH 9fn04CGD5IasN5+XrORrTF1jHTZBIm3YKPUTVcygblsWwIqURBipnZOco2K3Xi+4pWYZqc r1xwtSihXbJpXfTpc2O0yep+DkwoEjEMcXyXkbBOwKXPAznXy0tdoUaRsE+/M2CtY4x539 z1BXZxARhBIQCdTbTIgBOgVltk78Tdw+Rup9SqM3CUqWn4pBSywa1rTOETt42jep6ZbXgj RD6NeLjesdTFY/FB2q46ArPBIRsjWTxDlyKgi1eRIS6mAuqZ8CkK+vX9uA1dVA== ARC-Authentication-Results: i=1; rspamd-75d86777c9-m4jlz; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Oafish-Stupid: 1e699014540633d4_1729567276827_3440509185 X-MC-Loop-Signature: 1729567276827:160516643 X-MC-Ingress-Time: 1729567276826 Received: from pdx1-sub0-mail-a259.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.99.180.193 (trex/7.0.2); Tue, 22 Oct 2024 03:21:16 +0000 Received: from localhost.localdomain (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a259.dreamhost.com (Postfix) with ESMTPSA id 4XXcs70rrkzC9; Mon, 21 Oct 2024 20:21:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1729567276; bh=xWuXI1STFWVB6JAkyrFpy2OWqAiJ2oDnJCL4Dlt8unk=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=JRymzZpZt8j0qA4FrbQ8+cClG2+G0e7BtsNnJd7cVOu+BH/B3npfqY1RVzkEEe4ox k0fCFjbXUgBZpixaDYOejC4ugfQGsXPqsL6j8kiZa7kYZrb2Uekx2Y1q8TlN+PTUfF eHQl+/nj+NL464XM9cgVrk5dLRJ2tAw9JlNZqnMl3BvDBnZ4UDf/rSN3sbycWcUC8w 9Xeyuq1CMWruUgBOm//i9sJbLlYw6IztIV5qcbCjC9N9arlf2NG6q2+CFJ4Oscq1qG s4V7qU1FxUZp6k6T0rbRWG8DDP/rBlUL9beGhnj4e88pBOs7a+9xZMAS5DDCh3UABA gmzIBXcEbrEIQ== From: Davidlohr Bueso To: dan.j.williams@intel.com, dave.jiang@intel.com Cc: jonathan.cameron@huawei.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, fan.ni@samsung.com, a.manzanares@samsung.com, sthanneeru.opensrc@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, Ravis.OpenSrc@micron.com, sthanneeru@micron.com, dave@stgolabs.net, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] cxl/mbox: support aborting the current background operation Date: Mon, 21 Oct 2024 20:18:08 -0700 Message-ID: <20241022031809.242591-3-dave@stgolabs.net> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20241022031809.242591-1-dave@stgolabs.net> References: <20241022031809.242591-1-dave@stgolabs.net> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 CXL 3.1 introduced the ability to request that the current on-going background command be aborted. Add support for this, where the current policy is for the request to occur whenever a new incoming bg command wants to run. As such everything is left to user discretion and it becomes impossible to hog the device/mailbox. The context of doing the cancellation request is the same as the new incoming command, and will always hold the mbox_mutex, guaranteeing that any successful cancel does not race with a third thread coming in and stealing the effort. - For Sanitize, the thread doing the will cancel the work, and clean on behalf of the respective wq callback that will never come. - For the other bg commands, the sleeping thread is kicked and we busy-wait until the polling flag is cleared. In both scenarios, we guarantee that the aborted op's thread is no longer around, giving the new bg op full authority to submit the command. Semantics for devices that do not support such functionality are left unchanged, and hence, with this, the driver benefits in both scenarios. Signed-off-by: Davidlohr Bueso --- drivers/cxl/cxlmem.h | 1 + drivers/cxl/pci.c | 81 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 73 insertions(+), 9 deletions(-) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index b933fb73ef8a..e843ffc3c23a 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -516,6 +516,7 @@ to_cxl_memdev_state(struct cxl_dev_state *cxlds) enum cxl_opcode { CXL_MBOX_OP_INVALID = 0x0000, CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID, + CXL_MBOX_OP_REQUEST_ABORT_BG_OP = 0x0005, CXL_MBOX_OP_GET_EVENT_RECORD = 0x0100, CXL_MBOX_OP_CLEAR_EVENT_RECORD = 0x0101, CXL_MBOX_OP_GET_EVT_INT_POLICY = 0x0102, diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index f2378604669b..5da50e26e4c4 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -115,8 +115,8 @@ static bool cxl_mbox_background_complete(struct cxl_dev_state *cxlds) { u64 reg; - reg = readq(cxlds->regs.mbox + CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); - return FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_PCT_MASK, reg) == 100; + reg = readq(cxlds->regs.mbox + CXLDEV_MBOX_STATUS_OFFSET); + return FIELD_GET(CXLDEV_MBOX_STATUS_BG_CMD, reg) == 0; } static irqreturn_t cxl_pci_mbox_irq(int irq, void *id) @@ -241,7 +241,8 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_mailbox *cxl_mbox, * hardware semantics and only allow device health status. */ if (mds->security.poll_tmo_secs > 0) { - if (mbox_cmd->opcode != CXL_MBOX_OP_GET_HEALTH_INFO) + if (mbox_cmd->opcode != CXL_MBOX_OP_GET_HEALTH_INFO && + mbox_cmd->opcode != CXL_MBOX_OP_REQUEST_ABORT_BG_OP) return -EBUSY; } @@ -335,11 +336,64 @@ static int __cxl_pci_mbox_send_cmd(struct cxl_mailbox *cxl_mbox, return 0; } +/* + * Return true implies that the request was successful and the on-going + * background operation was in fact aborted. This also guarantees that + * the respective thread is done. + */ +static bool cxl_try_to_cancel_background(struct cxl_mailbox *cxl_mbox) +{ + int rc; + struct cxl_dev_state *cxlds = mbox_to_cxlds(cxl_mbox); + struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds); + struct device *dev = cxlds->dev; + struct cxl_mbox_cmd cmd = { + .opcode = CXL_MBOX_OP_REQUEST_ABORT_BG_OP + }; + + lockdep_assert_held(&cxl_mbox->mbox_mutex); + + rc = __cxl_pci_mbox_send_cmd(cxl_mbox, &cmd); + if (rc) { + dev_dbg(dev, "Failed to send abort request : %d\n", rc); + return false; + } + + if (!cxl_mbox_background_complete(cxlds)) + return false; + + if (mds->security.sanitize_active) { + /* + * Cancel the work and cleanup on its behalf - we hold + * the mbox_mutex, cannot race with cxl_mbox_sanitize_work(). + */ + cancel_delayed_work_sync(&mds->security.poll_dwork); + mds->security.poll_tmo_secs = 0; + if (mds->security.sanitize_node) + sysfs_notify_dirent(mds->security.sanitize_node); + mds->security.sanitize_active = false; + + dev_dbg(cxlds->dev, "Sanitization operation aborted\n"); + } else { + /* + * Kick the poller and wait for it to be done - no one else + * can touch mbox regs. rcuwait_wake_up() provides full + * barriers such that wake up occurs before waiting on the + * bgpoll atomic to be cleared. + */ + rcuwait_wake_up(&cxl_mbox->mbox_wait); + atomic_cond_read_acquire(&cxl_mbox->poll_bgop, !VAL); + } + + return true; +} + static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd) { int rc; struct cxl_dev_state *cxlds = mbox_to_cxlds(cxl_mbox); + struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds); struct device *dev = cxlds->dev; mutex_lock_io(&cxl_mbox->mbox_mutex); @@ -348,10 +402,18 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, * each other: no new bg operation can occur in between while polling. */ if (cxl_is_background_cmd(cmd->opcode)) { - if (atomic_read_acquire(&cxl_mbox->poll_bgop)) { - mutex_unlock(&cxl_mbox->mbox_mutex); - return -EBUSY; + if (mds->security.sanitize_active || + atomic_read_acquire(&cxl_mbox->poll_bgop)) { + if (!cxl_try_to_cancel_background(cxl_mbox)) { + mutex_unlock(&cxl_mbox->mbox_mutex); + return -EBUSY; + } } + /* + * ... at this point we know that the canceled + * bgop context is gone, and we are the _only_ + * background command in town. Proceed to send it. + */ } rc = __cxl_pci_mbox_send_cmd(cxl_mbox, cmd); @@ -394,10 +456,11 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); cmd->return_code = FIELD_GET(CXLDEV_MBOX_BG_CMD_COMMAND_RC_MASK, bg_status_reg); - dev_dbg(dev, - "Mailbox background operation (0x%04x) completed\n", - cmd->opcode); + + dev_dbg(dev, "Mailbox background operation (0x%04x) %s\n", + cmd->opcode, !cmd->return_code ? "completed":"aborted"); done: + /* ensure clearing poll_bop is the last operation */ atomic_set_release(&cxl_mbox->poll_bgop, 0); } From patchwork Tue Oct 22 03:18:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13845062 Received: from tiger.tulip.relay.mailchannels.net (tiger.tulip.relay.mailchannels.net [23.83.218.248]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DD15256D; Tue, 22 Oct 2024 03:38:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=23.83.218.248 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729568297; cv=pass; b=ufvcYtFpU4PbC7mHspn3nLLdA6Z3jHNwup8fxYagfk+pX0YbBOahs/M5PbRHK8xvG8/NAENvkZaQLJRXd48nXUicdn3AAVw+6N7717sD9mfYt0kuD7sEhPaxYHC+JcobRmV3u0Rbww63vpw+RERrJYM9g/zXFAFfqpIi1rkV6H4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729568297; c=relaxed/simple; bh=7D0XF6hx0jiTrgMrVL0CzhYSCpD/QNzpapqQvKUh9ms=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ISHTP7moJrlit+/AImoWqSsJi750Z4lKTvmAE1+G5K0ssIv6R+A9jvsPqAF0q2V5stD7QF1ZDR9mAkZYL8K3FujCUTVE1bnKY3GwpDeCe5ebjekPY9s2rEhvLDssimeOe2LlAw0zhZqJTq0eaiji+eBhsbOiPT7VMNjAstWM9PI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=stgolabs.net; spf=pass smtp.mailfrom=stgolabs.net; dkim=pass (2048-bit key) header.d=stgolabs.net header.i=@stgolabs.net header.b=j9vNN2wN; arc=pass smtp.client-ip=23.83.218.248 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=stgolabs.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=stgolabs.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=stgolabs.net header.i=@stgolabs.net header.b="j9vNN2wN" X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id F084C900472; Tue, 22 Oct 2024 03:21:17 +0000 (UTC) Received: from pdx1-sub0-mail-a259.dreamhost.com (100-99-177-19.trex-nlb.outbound.svc.cluster.local [100.99.177.19]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 71D019034C2; Tue, 22 Oct 2024 03:21:17 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1729567277; a=rsa-sha256; cv=none; b=8rgEcjUfnraCI5l8I+H2X6sMfWcIYbGfNkE9ohDKd5S8tKF2nfzH0oBG6+a6b8Ndzi+0Ck QgOv/4/3GLstMKgn5d4RZynhR4l1iyKjI+HF3oQ3hE6Kqb8OuvhXSu3Agz4USQIDiTj6MM xCTTAD9wYo4kRFXwyg+sDZgvYSjign9x+UZo2IpF/CmiWqHd4nDiwEmWG74uEof3m6hpWa ZzTFj0TfMWEFbcXoVPDj0y9nno9XzH7au2o/g/iDRO2c/rJ6jpOeGL/7Xh0FuXVpW+SWY0 L0dw+7nzq/7SXVZ8mwenr3JfWPVHlZRG21m3pnxenuj2bOG8xRfBrV9GU/setA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1729567277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=byorMJl2bYxBjm5ML9d5+2YL1dznnN5t3iQgO/ppNDI=; b=LW3p4T7aLdhBJZ5SOpjl2xz195FeASkAn/2wDUPEqXJIfpadA6n83dCrErKzp2Fxwnu1F0 2srQCWn4DN3udkEjbIC4yr4yLjnHn6+vARMIM2AJIA+Sbit7PrbPFdQPMLb5Xt28hb0beW AlKFb9BM2t+SILnMIMk74Ui1SXzNZTZCbzCVF/HVJyTZFF5JaKm7xsX9XTkz81wIh8fGg3 +W6LoQOKZJ22fREMve2G6dpF56hEDCRDfiXGJQyey+lLTqycXlt5MKLnDZeO3w1pobOTdX fZFF807DPZIpOIUE711RAWomt0Iwkh5i7QnjLpRdwcvGy3LXQ0nln+LJP/gA9Q== ARC-Authentication-Results: i=1; rspamd-75d86777c9-lf85w; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Power-Harbor: 098162ca14c9e48e_1729567277808_3789994943 X-MC-Loop-Signature: 1729567277808:3442003941 X-MC-Ingress-Time: 1729567277808 Received: from pdx1-sub0-mail-a259.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.99.177.19 (trex/7.0.2); Tue, 22 Oct 2024 03:21:17 +0000 Received: from localhost.localdomain (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a259.dreamhost.com (Postfix) with ESMTPSA id 4XXcs82XkKz9g; Mon, 21 Oct 2024 20:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1729567277; bh=byorMJl2bYxBjm5ML9d5+2YL1dznnN5t3iQgO/ppNDI=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=j9vNN2wNIIUe4sS9dVdUsjbyIrZNCt7WDTrgFtgk66oBcJPapjHp+SkcAtaZe86RV VmzE2T+n8SH/amMxElSUNevx0lvYxav7HAyJGQ+FpYOnC+1ywOSgJ3GBWm+gkSKOHA 0UeJuAPnQf1tLUqVrulo2t/YLF8qu/jSJQE0Ep5bKMzduoEtIe8r2OdbE/wtYclbbA wkVZ4oYQTQTUEG99xw55LhLtLr7VgGTberUNP5K3XY5zeyZm9IIf0m+4hSGHaApsfG wGsD0XDOdc/S7d5TvFWgfCVqDgjnU3tXCucJg2uR0AT/IpwhkhFLeYm8UDZ1O+Sm+A R8Mu8f1LK/srg== From: Davidlohr Bueso To: dan.j.williams@intel.com, dave.jiang@intel.com Cc: jonathan.cameron@huawei.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, fan.ni@samsung.com, a.manzanares@samsung.com, sthanneeru.opensrc@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, Ravis.OpenSrc@micron.com, sthanneeru@micron.com, dave@stgolabs.net, linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] cxl/pci: rename cxl_mbox_background_complete() Date: Mon, 21 Oct 2024 20:18:09 -0700 Message-ID: <20241022031809.242591-4-dave@stgolabs.net> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20241022031809.242591-1-dave@stgolabs.net> References: <20241022031809.242591-1-dave@stgolabs.net> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 With the abort functionality, this is a misleading name, rename to cxl_mbox_background_done() instead. Signed-off-by: Davidlohr Bueso --- drivers/cxl/pci.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 5da50e26e4c4..7b0fad7f6c4d 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -111,7 +111,7 @@ static int cxl_request_irq(struct cxl_dev_state *cxlds, int irq, dev_id); } -static bool cxl_mbox_background_complete(struct cxl_dev_state *cxlds) +static bool cxl_mbox_background_done(struct cxl_dev_state *cxlds) { u64 reg; @@ -128,7 +128,7 @@ static irqreturn_t cxl_pci_mbox_irq(int irq, void *id) struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox; struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlds); - if (!cxl_mbox_background_complete(cxlds)) + if (!cxl_mbox_background_done(cxlds)) return IRQ_NONE; reg = readq(cxlds->regs.mbox + CXLDEV_MBOX_BG_CMD_STATUS_OFFSET); @@ -157,7 +157,7 @@ static void cxl_mbox_sanitize_work(struct work_struct *work) struct cxl_mailbox *cxl_mbox = &cxlds->cxl_mbox; mutex_lock(&cxl_mbox->mbox_mutex); - if (cxl_mbox_background_complete(cxlds)) { + if (cxl_mbox_background_done(cxlds)) { mds->security.poll_tmo_secs = 0; if (mds->security.sanitize_node) sysfs_notify_dirent(mds->security.sanitize_node); @@ -359,7 +359,7 @@ static bool cxl_try_to_cancel_background(struct cxl_mailbox *cxl_mbox) return false; } - if (!cxl_mbox_background_complete(cxlds)) + if (!cxl_mbox_background_done(cxlds)) return false; if (mds->security.sanitize_active) { @@ -398,7 +398,7 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, mutex_lock_io(&cxl_mbox->mbox_mutex); /* - * Ensure cxl_mbox_background_complete() checks are safe amongst + * Ensure cxl_mbox_background_done() checks are safe amongst * each other: no new bg operation can occur in between while polling. */ if (cxl_is_background_cmd(cmd->opcode)) { @@ -434,7 +434,7 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, timeout = cmd->poll_interval_ms; for (i = 0; i < cmd->poll_count; i++) { if (rcuwait_wait_event_timeout(&cxl_mbox->mbox_wait, - cxl_mbox_background_complete(cxlds), + cxl_mbox_background_done(cxlds), TASK_UNINTERRUPTIBLE, msecs_to_jiffies(timeout)) > 0) break; @@ -445,7 +445,7 @@ static int cxl_pci_mbox_send(struct cxl_mailbox *cxl_mbox, * until the next successful command submission and the driver * can get back in sync with the hardware state. */ - if (!cxl_mbox_background_complete(cxlds)) { + if (!cxl_mbox_background_done(cxlds)) { dev_err(dev, "timeout waiting for background (%d ms)\n", timeout * cmd->poll_count); rc = -ETIMEDOUT;