From patchwork Fri Oct 14 19:49:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13007286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08D2DC4332F for ; Fri, 14 Oct 2022 19:50:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230161AbiJNTug (ORCPT ); Fri, 14 Oct 2022 15:50:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231358AbiJNTuf (ORCPT ); Fri, 14 Oct 2022 15:50:35 -0400 Received: from beige.elm.relay.mailchannels.net (beige.elm.relay.mailchannels.net [23.83.212.16]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13E548C026; Fri, 14 Oct 2022 12:50:31 -0700 (PDT) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 17A7014167E; Fri, 14 Oct 2022 19:50:18 +0000 (UTC) Received: from pdx1-sub0-mail-a219.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 5E9CE1414BE; Fri, 14 Oct 2022 19:50:17 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1665777017; a=rsa-sha256; cv=none; b=OknIobnNwbXerNu7HApdSiBuNkKHkue4exW7DUJB04etTnpPBprN1wgDVDZGpa48VecPBK UUVdiryevbTUnwb+90Ukhw8ii1O0tyzh8wX+mnSBVKHkKdZCZxjJIuQwFpEYUslyCuAmx3 R03JwmvgvYF9c6rgJAZJWoCNWRdipRYJsyXC/9EJpXMTwZaSCxNtrJV3rtm4YMmEknNz5n saHoRltbuO+fSGe6PYxB4/jumhU5/oPjJAzH1a/k/4+epLWQ38kwob9gzlt26RH4pNyjkS ndIBnMZtmBB8LmryeuAQ2PIpm6HYjYUh06TpbNAX1Oce9JH1dty1r33Bue3MHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1665777017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4McCZ56zCh20Vp80Kwo95PLFd0Mi47/KFhOpUHARCVY=; b=ty2Ga96fOUXFea57d7iTJSqCBowIFxytGFsVJ02u5nY8VeqBNkTE5WTGF6kBjlfLkum5P1 SCiQgfs838M0JXXmPoCxsmuKbPZYRad7HNpX82IMf49g/iU+LE7OUjFrFmaGT/E12Ve7uu CVIyAf5UK6wUGFgCiBVgai5Iluci0sZW69WCqSopnKubhDUlE6ukfC3A/WzTVwwIwb3ZUC 28pbB4bHqBtgAchADVLVpaHsAVsKORXu/IZApCxK2LvUuLcb2tpCweY44QQ26aUvF3oKmK qLH2UNVB3n33kTuN1wNBNiSQ5VVMWrXeBuR/7NmQyd1zxt4qCvPdc/Nydd5Gsg== ARC-Authentication-Results: i=1; rspamd-5798657bcf-rqdj4; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Bottle-Imminent: 1761f7291737c9b3_1665777017763_295819113 X-MC-Loop-Signature: 1665777017763:3860012546 X-MC-Ingress-Time: 1665777017762 Received: from pdx1-sub0-mail-a219.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.106.246.144 (trex/6.7.1); Fri, 14 Oct 2022 19:50:17 +0000 Received: from offworld.. (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a219.dreamhost.com (Postfix) with ESMTPSA id 4MpxnN2X3zz4s; Fri, 14 Oct 2022 12:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1665777017; bh=4McCZ56zCh20Vp80Kwo95PLFd0Mi47/KFhOpUHARCVY=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=cUfSgyHcaW+V/7/sMFTBQFNCgQha8e7XSX3MO7xtHfDI/jNKNkhMjoutp+hb0Aidp f1t9JxxsaKdksa1WfTGPgIzf5qKC/cNoFn1WfsX1FPiUTb0+h3Rc1YNa1iyCUA2uoe 73ivN3VRTm9+AkPP5fGfVcthaUuJzzNqv9FNGM0q63/InalDExvRBg9OybbHk8s8Fx uEnMs0oX0tUeOA2j7NZvOqnSakSUxlYMm5Ujd1Sr6cK5PifPW18SedEr5wlfsEcKPi kljT5F80uVJUvWm1OyYJRPwoETWP7YjGcDXBtS54K26/I7WyRMQ9EXOtuvGJyZOpMu YRSJ6vBuvOpSQ== From: Davidlohr Bueso To: dan.j.williams@intel.com Cc: ira.weiny@intel.com, Jonathan.Cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, a.manzanares@samsung.com, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 1/2] cxl/pci: Add generic MSI/MSI-X interrupt support Date: Fri, 14 Oct 2022 12:49:29 -0700 Message-Id: <20221014194930.2630416-2-dave@stgolabs.net> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221014194930.2630416-1-dave@stgolabs.net> References: <20221014194930.2630416-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org Introduce a generic irq table for CXL components that can have standard irq support - DOE requires dynamic vector sizing and is as such is not considered here. Create an infrastructure to query the max vectors required for the CXL device. Users can check the irq_type in the device state to figure if they want to attempt to register a handler for it's specific irq and deal with it accordingly. Reviewed-by: Dave Jiang Signed-off-by: Davidlohr Bueso --- drivers/cxl/cxl.h | 5 ++++ drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/pci.c | 66 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index f680450f0b16..879661702054 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -119,6 +119,11 @@ static inline int ways_to_cxl(unsigned int ways, u8 *iw) return 0; } +enum { + CXL_IRQ_NONE, + CXL_IRQ_MSI, +}; + /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ #define CXLDEV_CAP_ARRAY_OFFSET 0x0 #define CXLDEV_CAP_ARRAY_CAP_ID 0 diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 88e3a8e54b6a..ca020767f7fc 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -247,6 +247,8 @@ struct cxl_dev_state { struct xarray doe_mbs; + int irq_type; + int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd); }; diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index faeb5d9d7a7a..942c4449d30f 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -428,6 +428,67 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } } +/** + * struct cxl_irq_cap - CXL feature that is capable of receiving MSI/MSI-X irqs. + * + * @name: Name of the device generating this interrupt. + * @get_max_msgnum: Get the feature's largest interrupt message number. If the + * feature does not have the Interrupt Supported bit set, then + * return -1. + */ +struct cxl_irq_cap { + const char *name; + int (*get_max_msgnum)(struct cxl_dev_state *cxlds); +}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { NULL }; + +static void cxl_pci_free_irq_vectors(void *data) +{ + pci_free_irq_vectors(data); +} + +/* + * Attempt to allocate the largest amount of necessary vectors. + * + * Returns 0 upon a successful allocation of *all* vectors, or a + * negative value otherwise. + */ +static int cxl_pci_alloc_irq_vectors(struct cxl_dev_state *cxlds) +{ + struct device *dev = cxlds->dev; + struct pci_dev *pdev = to_pci_dev(dev); + int rc, i, vectors = -1; + + for (i = 0; i < ARRAY_SIZE(cxl_irq_cap_table); i++) { + int irq; + + if (!cxl_irq_cap_table[i].get_max_msgnum) + continue; + + irq = cxl_irq_cap_table[i].get_max_msgnum(cxlds); + vectors = max_t(int, irq, vectors); + } + + if (vectors == -1) + return -1; + + vectors++; + rc = pci_alloc_irq_vectors(pdev, vectors, vectors, + PCI_IRQ_MSIX | PCI_IRQ_MSI); + if (rc < 0) + return rc; + + if (rc != vectors) { + dev_err(dev, "Not enough interrupts; use polling instead.\n"); + /* some got allocated, clean them up */ + cxl_pci_free_irq_vectors(pdev); + return -ENOSPC; + } + + return devm_add_action_or_reset(dev, cxl_pci_free_irq_vectors, pdev); +} + static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct cxl_register_map map; @@ -478,6 +539,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) devm_cxl_pci_create_doe(cxlds); + if (!cxl_pci_alloc_irq_vectors(cxlds)) + cxlds->irq_type = CXL_IRQ_MSI; + else + cxlds->irq_type = CXL_IRQ_NONE; + rc = cxl_pci_setup_mailbox(cxlds); if (rc) return rc; From patchwork Fri Oct 14 19:49:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Davidlohr Bueso X-Patchwork-Id: 13007287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59359C433FE for ; Fri, 14 Oct 2022 19:50:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231222AbiJNTur (ORCPT ); Fri, 14 Oct 2022 15:50:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231368AbiJNTuq (ORCPT ); Fri, 14 Oct 2022 15:50:46 -0400 Received: from crane.ash.relay.mailchannels.net (crane.ash.relay.mailchannels.net [23.83.222.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DC73A8CE2; Fri, 14 Oct 2022 12:50:34 -0700 (PDT) X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 89FDE6C2207; Fri, 14 Oct 2022 19:50:20 +0000 (UTC) Received: from pdx1-sub0-mail-a219.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id A13EB6C21FA; Fri, 14 Oct 2022 19:50:19 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1665777019; a=rsa-sha256; cv=none; b=bCRoiEOR2noCWwHM4YQ9ty2djvpvNZ/PUmmkLfGcDoRXixygs4WX7W3mBSwj1DifEgSNul 0W2PyMU85luQKuVIpZh3o9/LPVZ/9mKEfSujoo62jgPS6ESIVdU9I6/kEoA4hcQeOws5dx TrYPZbawr5f0Bxg79wyDNTuquhbWtN7tRzOjVdKFj36TH6oVFRJAOBnDiLyPZ/wl8OnFbL /dMoeX3FjhGujch2wnC00BN4zQT4KeR3LEeCFybVUvYfRiR26Mm2NHPd9kth309g5Amb2X eunYaA/s4Zh8vv9LCLV7jnX2wY9k3R/u3Fw8DfY2UQHwRLVU8nmUEMj3j3B+NA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1665777019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hfzbsr9FI/rzLOLEFBaAvUZAbZ6DVfHSQA0uzUjx5qs=; b=HFVwuu2aewx890rozxPt5Fa3tJjbVFk3Syd2hXKUDpCsmcJAkw3oCPSrJOjtwlYMI8nNjS dZNn1xqcJES5PuIQ0u8sgOZZ17TqH1/3LUsgt3bFLFybgbKMruVn8CIAp+T9W6gfCJ0uyH 1ffUYs7tD8nUV7DZFYwpwo+Q4m50bH58g1UC+LYthMj6DafBZJQDGX8bYMKAcOMuvtKXsl PJWC2TzXgsVMfvjW8j28N2t3eJuXsDaU3bcL97klFvSx4wuqd8YKhzly7vJ/KpTwqC+93r 3PIz70t+/DDHQpf9ZBqYXD7XfFxo7o2mvUL5xhu79l4/3vwGY5CBNAZzey81jg== ARC-Authentication-Results: i=1; rspamd-5798657bcf-8zx8p; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Spicy-Occur: 419aa3984982d9c9_1665777020009_1395546131 X-MC-Loop-Signature: 1665777020009:465102095 X-MC-Ingress-Time: 1665777020009 Received: from pdx1-sub0-mail-a219.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.122.146.22 (trex/6.7.1); Fri, 14 Oct 2022 19:50:20 +0000 Received: from offworld.. (ip72-199-50-187.sd.sd.cox.net [72.199.50.187]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a219.dreamhost.com (Postfix) with ESMTPSA id 4MpxnP22gyz3p; Fri, 14 Oct 2022 12:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1665777017; bh=hfzbsr9FI/rzLOLEFBaAvUZAbZ6DVfHSQA0uzUjx5qs=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=LBXQNzKkFNGvlcw2g1l6ZBy61RbR2Sq48CUW7lnbxvi9mkhq80iMvQvEM9Gr3dUz6 uNfxvIjmhPJsuSiwTXEQXnzCGhkhypi0F33WrZmWd+uySBsGU0RAFVRJgoKdBKJt+C uDMhD7FO2meN8emPwIgofxkGpVfBIJaWk8HHwtqEv3MHSx0InXLR8LPalJLOSJa9cS gHKDxNmkr8xDjfNzIj9Ze4fJNYC6FoMartV1ZNy2dp/gf7JLX3YYWHBSgOyfUb4hLZ s5ZgOQT+d86QLIjrIx743vhEYF0IEncUcxn1lPjHszSRuiLuEMWmnotLQkyexUJq4/ FJDxMxYJylw3Q== From: Davidlohr Bueso To: dan.j.williams@intel.com Cc: ira.weiny@intel.com, Jonathan.Cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, bwidawsk@kernel.org, a.manzanares@samsung.com, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, dave@stgolabs.net Subject: [PATCH 2/2] cxl/mbox: Wire up basic irq support Date: Fri, 14 Oct 2022 12:49:30 -0700 Message-Id: <20221014194930.2630416-3-dave@stgolabs.net> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221014194930.2630416-1-dave@stgolabs.net> References: <20221014194930.2630416-1-dave@stgolabs.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-cxl@vger.kernel.org This adds support for mailbox interrupts, which are needed, for example, for background completion handling. Signed-off-by: Davidlohr Bueso --- Note: We could also handle doorbell irq, but not sure this is actually needed. drivers/cxl/cxl.h | 1 + drivers/cxl/pci.c | 27 ++++++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 879661702054..d15a743bfc9e 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -140,6 +140,7 @@ enum { /* CXL 2.0 8.2.8.4 Mailbox Registers */ #define CXLDEV_MBOX_CAPS_OFFSET 0x00 #define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) +#define CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK GENMASK(10, 7) #define CXLDEV_MBOX_CTRL_OFFSET 0x04 #define CXLDEV_MBOX_CTRL_DOORBELL BIT(0) #define CXLDEV_MBOX_CMD_OFFSET 0x08 diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 942c4449d30f..6e18ca3e551f 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -51,6 +51,20 @@ static unsigned short mbox_ready_timeout = 60; module_param(mbox_ready_timeout, ushort, 0644); MODULE_PARM_DESC(mbox_ready_timeout, "seconds to wait for mailbox ready"); +static int cxl_pci_mbox_get_max_msgnum(struct cxl_dev_state *cxlds) +{ + int cap; + + cap = readl(cxlds->regs.mbox + CXLDEV_MBOX_CAPS_OFFSET); + return FIELD_GET(CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK, cap); +} + +static irqreturn_t cxl_pci_mbox_irq(int irq, void *id) +{ + /* TODO: handle completion of background commands */ + return IRQ_HANDLED; +} + static int cxl_pci_mbox_wait_for_doorbell(struct cxl_dev_state *cxlds) { const unsigned long start = jiffies; @@ -271,6 +285,15 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) dev_dbg(cxlds->dev, "Mailbox payload sized %zu", cxlds->payload_size); + if (cxlds->irq_type == CXL_IRQ_MSI) { + struct device *dev = cxlds->dev; + int irq = cxl_pci_mbox_get_max_msgnum(cxlds); + + if (devm_request_irq(dev, irq, cxl_pci_mbox_irq, + IRQF_SHARED, "mailbox", cxlds)) + dev_dbg(dev, "Mailbox irq (%d) supported", irq); + } + return 0; } @@ -441,7 +464,9 @@ struct cxl_irq_cap { int (*get_max_msgnum)(struct cxl_dev_state *cxlds); }; -static const struct cxl_irq_cap cxl_irq_cap_table[] = { NULL }; +static const struct cxl_irq_cap cxl_irq_cap_table[] = { + { "mailbox", cxl_pci_mbox_get_max_msgnum } +}; static void cxl_pci_free_irq_vectors(void *data) {