From patchwork Tue Sep 4 16:28:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sunil Kovvuri X-Patchwork-Id: 10587647 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCAD0112B for ; Tue, 4 Sep 2018 16:33:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C0A6291DE for ; Tue, 4 Sep 2018 16:33:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F9BF294AB; Tue, 4 Sep 2018 16:33:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B408A291DE for ; Tue, 4 Sep 2018 16:33:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BYcKtfIpu/K14A/13IysluxhS8sMT4S1Z2KpFXla+Ls=; b=k75YKc9h/D8lawhRyc9xhiZLZk GLh/QvzaGI0eEwi6yrnwbkXvNnSK4kQdhNbgPZNdv+wIkS+fDsw4M9ClUvX2WG9lItHSTUCiXJ2D2 h3H4ilxvyZDhrxt//jdv0MLvVQPwfLxky+CpVY0uG8vHp//vOhAt+nmyv9rw68I9McnRc7RZ7HfZs MJFDYwBI/jKjqAMp3oMre6U7pTeph6Dx6bYaUZV+9EGej1qfCm1/FfAGUX3fu440InHC1XYlwqUA/ Ab/92+amtstKQk3rPRdRc0VzrKAKbtnrukCcUXRCzoLsiLrW9gZr/27sSO0+ron/02GoSECXGDWEK bblKXxyg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fxEGb-0007To-Lr; Tue, 04 Sep 2018 16:33:25 +0000 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fxECi-0004XU-N9 for linux-arm-kernel@lists.infradead.org; Tue, 04 Sep 2018 16:29:27 +0000 Received: by mail-pg1-x544.google.com with SMTP id r1-v6so1906652pgp.11 for ; Tue, 04 Sep 2018 09:29:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=280EP+QgaX/l+BzeVZuKBc4Hv62//0y2K3uoqdg3hqA=; b=lGRzLpCMULpDkzIlCOKgM20bG7jzEH0DYPLGhnU3pWzXLvbiYdTeMp5ASIqWeQP7ci lnudN0jh4uvspH5bzRcZH3/q8JgKuRMkAMkGdyGdtkGuOQUT3QhG1il0v+2ePJekzSld tvoUsp+DukeM+CGtii9vffn7LQvrep9lR2uKaGGIoDBrPhGu/hnCVSxLSJJmqT74fEML ISVbgl+zx45QaPxA1/u4RqLNb+UxmOVuNPgWsQd889bz792DuGQ2WGiCCR5qOxwhlIhA 6FbLAtoJ2KQDm4l4Q1wo7wqmIIZOYuDng5tj6CCa/Zn1uWEdlJHuWwbFKIq05xaoEGQU eeRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=280EP+QgaX/l+BzeVZuKBc4Hv62//0y2K3uoqdg3hqA=; b=LT7Drdtng8/RX2dukv5L6IPLoNuoARfifehgbDCVKtiHJ0H+CoLyCj5GcWJZaAwW1K S5gUR1RavFq8ErA761KjJfIMQmJ5JbagCTaOHRhX2BMLkp06ODs3gVHOCbwriwC7hJIA sFPl1At2yIuIf8D8BiVAOAoMtFe1pS32y74tLFbR0Xs1dOIwvPPGhyc82vOgbFouPO8a K3u7Cm8/cDPNLeM5dXoMO/BoossRuKaaFJcbOhaY+Ny4FayLOTDfz8kHfzEU+sYJ97ya Kv0aASAgrlqZ5IjIABAucJr2pmpk2twwwJJv0Dce0VLQXNy9tLQhxnTTjs85eDsZZCZW n/8g== X-Gm-Message-State: APzg51DXWmhUAIXP/GGjhb12ron91udK6k5lUmb4wGpyAYuc1F6FkpOx kdsmyOIuvWrxwcgH3e9pokk= X-Google-Smtp-Source: ANB0Vda2YfT8gX3dW3jmRQwdZCw7Nf1E9nbaQ7Fkdh1J0aOdjQXU/jpZq1zDN95PJ88A5D0/UeDsVA== X-Received: by 2002:a62:8186:: with SMTP id t128-v6mr35237488pfd.192.1536078553760; Tue, 04 Sep 2018 09:29:13 -0700 (PDT) Received: from machine421.caveonetworks.com ([115.113.156.2]) by smtp.googlemail.com with ESMTPSA id j191-v6sm32243827pfc.136.2018.09.04.09.29.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 04 Sep 2018 09:29:12 -0700 (PDT) From: sunil.kovvuri@gmail.com To: linux-kernel@vger.kernel.org, arnd@arndb.de, olof@lixom.net Subject: [PATCH v3 05/15] soc: octeontx2: Add mailbox IRQ and msg handlers Date: Tue, 4 Sep 2018 21:58:35 +0530 Message-Id: <1536078525-31534-6-git-send-email-sunil.kovvuri@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536078525-31534-1-git-send-email-sunil.kovvuri@gmail.com> References: <1536078525-31534-1-git-send-email-sunil.kovvuri@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180904_092924_791711_0BF60100 X-CRM114-Status: GOOD ( 23.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: andrew@lunn.ch, netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-soc@vger.kernel.org, davem@davemloft.net, Sunil Goutham MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Sunil Goutham This patch adds support for mailbox interrupt and message handling. Mapped mailbox region and registered a workqueue for message handling. Enabled mailbox IRQ of RVU PFs and registered a interrupt handler. When IRQ is triggered work is added to the mbox workqueue for msgs to get processed. Signed-off-by: Sunil Goutham --- drivers/soc/marvell/octeontx2/mbox.h | 14 +- drivers/soc/marvell/octeontx2/rvu.c | 254 +++++++++++++++++++++++++++++ drivers/soc/marvell/octeontx2/rvu.h | 22 +++ drivers/soc/marvell/octeontx2/rvu_struct.h | 22 +++ 4 files changed, 309 insertions(+), 3 deletions(-) diff --git a/drivers/soc/marvell/octeontx2/mbox.h b/drivers/soc/marvell/octeontx2/mbox.h index 8e205fd..fc593f0 100644 --- a/drivers/soc/marvell/octeontx2/mbox.h +++ b/drivers/soc/marvell/octeontx2/mbox.h @@ -33,6 +33,8 @@ # error "incorrect mailbox area sizes" #endif +#define INTR_MASK(pfvfs) ((pfvfs < 64) ? (BIT_ULL(pfvfs) - 1) : (~0ull)) + #define MBOX_RSP_TIMEOUT 1000 /* in ms, Time to wait for mbox response */ #define MBOX_MSG_ALIGN 16 /* Align mbox msg start to 16bytes */ @@ -90,8 +92,9 @@ struct mbox_msghdr { void otx2_mbox_reset(struct otx2_mbox *mbox, int devid); void otx2_mbox_destroy(struct otx2_mbox *mbox); -int otx2_mbox_init(struct otx2_mbox *mbox, void *hwbase, struct pci_dev *pdev, - void *reg_base, int direction, int ndevs); +int otx2_mbox_init(struct otx2_mbox *mbox, void __force *hwbase, + struct pci_dev *pdev, void __force *reg_base, + int direction, int ndevs); void otx2_mbox_msg_send(struct otx2_mbox *mbox, int devid); int otx2_mbox_wait_for_rsp(struct otx2_mbox *mbox, int devid); int otx2_mbox_busy_poll_for_rsp(struct otx2_mbox *mbox, int devid); @@ -115,7 +118,7 @@ static inline struct mbox_msghdr *otx2_mbox_alloc_msg(struct otx2_mbox *mbox, #define MBOX_MSG_MAX 0xFFFF #define MBOX_MESSAGES \ -M(READY, 0x001, msg_req, msg_rsp) +M(READY, 0x001, msg_req, ready_msg_rsp) enum { #define M(_name, _id, _1, _2) MBOX_MSG_ ## _name = _id, @@ -139,4 +142,9 @@ struct msg_rsp { struct mbox_msghdr hdr; }; +struct ready_msg_rsp { + struct mbox_msghdr hdr; + u16 sclk_feq; /* SCLK frequency */ +}; + #endif /* MBOX_H */ diff --git a/drivers/soc/marvell/octeontx2/rvu.c b/drivers/soc/marvell/octeontx2/rvu.c index fa5f40b..e795c2f 100644 --- a/drivers/soc/marvell/octeontx2/rvu.c +++ b/drivers/soc/marvell/octeontx2/rvu.c @@ -258,6 +258,245 @@ static int rvu_setup_hw_resources(struct rvu *rvu) return 0; } +static int rvu_process_mbox_msg(struct rvu *rvu, int devid, + struct mbox_msghdr *req) +{ + /* Check if valid, if not reply with a invalid msg */ + if (req->sig != OTX2_MBOX_REQ_SIG) + goto bad_message; + + if (req->id == MBOX_MSG_READY) + return 0; + +bad_message: + otx2_reply_invalid_msg(&rvu->mbox, devid, req->pcifunc, + req->id); + return -ENODEV; +} + +static void rvu_mbox_handler(struct work_struct *work) +{ + struct rvu_work *mwork = container_of(work, struct rvu_work, work); + struct rvu *rvu = mwork->rvu; + struct otx2_mbox_dev *mdev; + struct mbox_hdr *req_hdr; + struct mbox_msghdr *msg; + struct otx2_mbox *mbox; + int offset, id, err; + u16 pf; + + mbox = &rvu->mbox; + pf = mwork - rvu->mbox_wrk; + mdev = &mbox->dev[pf]; + + /* Process received mbox messages */ + req_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + if (req_hdr->num_msgs == 0) + return; + + offset = mbox->rx_start + ALIGN(sizeof(*req_hdr), MBOX_MSG_ALIGN); + + for (id = 0; id < req_hdr->num_msgs; id++) { + msg = (struct mbox_msghdr *)(mdev->mbase + offset); + + /* Set which PF sent this message based on mbox IRQ */ + msg->pcifunc &= ~(RVU_PFVF_PF_MASK << RVU_PFVF_PF_SHIFT); + msg->pcifunc |= (pf << RVU_PFVF_PF_SHIFT); + err = rvu_process_mbox_msg(rvu, pf, msg); + if (!err) { + offset = mbox->rx_start + msg->next_msgoff; + continue; + } + + if (msg->pcifunc & RVU_PFVF_FUNC_MASK) + dev_warn(rvu->dev, "Error %d when processing message %s (0x%x) from PF%d:VF%d\n", + err, otx2_mbox_id2name(msg->id), msg->id, pf, + (msg->pcifunc & RVU_PFVF_FUNC_MASK) - 1); + else + dev_warn(rvu->dev, "Error %d when processing message %s (0x%x) from PF%d\n", + err, otx2_mbox_id2name(msg->id), msg->id, pf); + } + + /* Send mbox responses to PF */ + otx2_mbox_msg_send(mbox, pf); +} + +static int rvu_mbox_init(struct rvu *rvu) +{ + struct rvu_hwinfo *hw = rvu->hw; + struct rvu_work *mwork; + void __iomem *hwbase = NULL; + u64 bar4_addr; + int err, pf; + + rvu->mbox_wq = alloc_workqueue("rvu_afpf_mailbox", + WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM, + hw->total_pfs); + if (!rvu->mbox_wq) + return -ENOMEM; + + rvu->mbox_wrk = devm_kcalloc(rvu->dev, hw->total_pfs, + sizeof(struct rvu_work), GFP_KERNEL); + if (!rvu->mbox_wrk) { + err = -ENOMEM; + goto exit; + } + + /* Map mbox region shared with PFs */ + bar4_addr = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_PF_BAR4_ADDR); + /* Mailbox is a reserved memory (in RAM) region shared between + * RVU devices, shouldn't be mapped as device memory to allow + * unaligned accesses. + */ + hwbase = ioremap_wc(bar4_addr, MBOX_SIZE * hw->total_pfs); + if (!hwbase) { + dev_err(rvu->dev, "Unable to map mailbox region\n"); + err = -ENOMEM; + goto exit; + } + + err = otx2_mbox_init(&rvu->mbox, hwbase, rvu->pdev, rvu->afreg_base, + MBOX_DIR_AFPF, hw->total_pfs); + if (err) + goto exit; + + for (pf = 0; pf < hw->total_pfs; pf++) { + mwork = &rvu->mbox_wrk[pf]; + mwork->rvu = rvu; + INIT_WORK(&mwork->work, rvu_mbox_handler); + } + + return 0; +exit: + if (hwbase) + iounmap((void __iomem *)hwbase); + destroy_workqueue(rvu->mbox_wq); + return err; +} + +static void rvu_mbox_destroy(struct rvu *rvu) +{ + if (rvu->mbox_wq) { + flush_workqueue(rvu->mbox_wq); + destroy_workqueue(rvu->mbox_wq); + rvu->mbox_wq = NULL; + } + + if (rvu->mbox.hwbase) + iounmap((void __iomem *)rvu->mbox.hwbase); + + otx2_mbox_destroy(&rvu->mbox); +} + +static irqreturn_t rvu_mbox_intr_handler(int irq, void *rvu_irq) +{ + struct rvu *rvu = (struct rvu *)rvu_irq; + struct otx2_mbox_dev *mdev; + struct otx2_mbox *mbox; + struct mbox_hdr *hdr; + u64 intr; + u8 pf; + + intr = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_PFAF_MBOX_INT); + /* Clear interrupts */ + rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_PFAF_MBOX_INT, intr); + + /* Sync with mbox memory region */ + smp_wmb(); + + for (pf = 0; pf < rvu->hw->total_pfs; pf++) { + if (intr & (1ULL << pf)) { + mbox = &rvu->mbox; + mdev = &mbox->dev[pf]; + hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); + if (hdr->num_msgs) + queue_work(rvu->mbox_wq, + &rvu->mbox_wrk[pf].work); + } + } + + return IRQ_HANDLED; +} + +static void rvu_enable_mbox_intr(struct rvu *rvu) +{ + struct rvu_hwinfo *hw = rvu->hw; + + /* Clear spurious irqs, if any */ + rvu_write64(rvu, BLKADDR_RVUM, + RVU_AF_PFAF_MBOX_INT, INTR_MASK(hw->total_pfs)); + + /* Enable mailbox interrupt for all PFs except PF0 i.e AF itself */ + rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_PFAF_MBOX_INT_ENA_W1S, + INTR_MASK(hw->total_pfs) & ~1ULL); +} + +static void rvu_unregister_interrupts(struct rvu *rvu) +{ + int irq; + + /* Disable the Mbox interrupt */ + rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_PFAF_MBOX_INT_ENA_W1C, + INTR_MASK(rvu->hw->total_pfs) & ~1ULL); + + for (irq = 0; irq < rvu->num_vec; irq++) { + if (rvu->irq_allocated[irq]) + free_irq(pci_irq_vector(rvu->pdev, irq), rvu); + } + + pci_free_irq_vectors(rvu->pdev); + rvu->num_vec = 0; +} + +static int rvu_register_interrupts(struct rvu *rvu) +{ + int ret; + + rvu->num_vec = pci_msix_vec_count(rvu->pdev); + + rvu->irq_name = devm_kmalloc_array(rvu->dev, rvu->num_vec, + NAME_SIZE, GFP_KERNEL); + if (!rvu->irq_name) + return -ENOMEM; + + rvu->irq_allocated = devm_kcalloc(rvu->dev, rvu->num_vec, + sizeof(bool), GFP_KERNEL); + if (!rvu->irq_allocated) + return -ENOMEM; + + /* Enable MSI-X */ + ret = pci_alloc_irq_vectors(rvu->pdev, rvu->num_vec, + rvu->num_vec, PCI_IRQ_MSIX); + if (ret < 0) { + dev_err(rvu->dev, + "RVUAF: Request for %d msix vectors failed, ret %d\n", + rvu->num_vec, ret); + return ret; + } + + /* Register mailbox interrupt handler */ + sprintf(&rvu->irq_name[RVU_AF_INT_VEC_MBOX * NAME_SIZE], "RVUAF Mbox"); + ret = request_irq(pci_irq_vector(rvu->pdev, RVU_AF_INT_VEC_MBOX), + rvu_mbox_intr_handler, 0, + &rvu->irq_name[RVU_AF_INT_VEC_MBOX * NAME_SIZE], rvu); + if (ret) { + dev_err(rvu->dev, + "RVUAF: IRQ registration failed for mbox irq\n"); + goto fail; + } + + rvu->irq_allocated[RVU_AF_INT_VEC_MBOX] = true; + + /* Enable mailbox interrupts from all PFs */ + rvu_enable_mbox_intr(rvu); + + return 0; + +fail: + pci_free_irq_vectors(rvu->pdev); + return ret; +} + static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct device *dev = &pdev->dev; @@ -320,8 +559,21 @@ static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (err) goto err_release_regions; + err = rvu_mbox_init(rvu); + if (err) + goto err_hwsetup; + + err = rvu_register_interrupts(rvu); + if (err) + goto err_mbox; + return 0; +err_mbox: + rvu_mbox_destroy(rvu); +err_hwsetup: + rvu_reset_all_blocks(rvu); + rvu_free_hw_resources(rvu); err_release_regions: pci_release_regions(pdev); err_disable_device: @@ -337,6 +589,8 @@ static void rvu_remove(struct pci_dev *pdev) { struct rvu *rvu = pci_get_drvdata(pdev); + rvu_unregister_interrupts(rvu); + rvu_mbox_destroy(rvu); rvu_reset_all_blocks(rvu); rvu_free_hw_resources(rvu); diff --git a/drivers/soc/marvell/octeontx2/rvu.h b/drivers/soc/marvell/octeontx2/rvu.h index 592b820..2fb9407 100644 --- a/drivers/soc/marvell/octeontx2/rvu.h +++ b/drivers/soc/marvell/octeontx2/rvu.h @@ -12,6 +12,7 @@ #define RVU_H #include "rvu_struct.h" +#include "mbox.h" /* PCI device IDs */ #define PCI_DEVID_OCTEONTX2_RVU_AF 0xA065 @@ -23,6 +24,17 @@ #define NAME_SIZE 32 +/* PF_FUNC */ +#define RVU_PFVF_PF_SHIFT 10 +#define RVU_PFVF_PF_MASK 0x3F +#define RVU_PFVF_FUNC_SHIFT 0 +#define RVU_PFVF_FUNC_MASK 0x3FF + +struct rvu_work { + struct work_struct work; + struct rvu *rvu; +}; + struct rsrc_bmap { unsigned long *bmap; /* Pointer to resource bitmap */ u16 max; /* Max resource id or count */ @@ -57,6 +69,16 @@ struct rvu { struct pci_dev *pdev; struct device *dev; struct rvu_hwinfo *hw; + + /* Mbox */ + struct otx2_mbox mbox; + struct rvu_work *mbox_wrk; + struct workqueue_struct *mbox_wq; + + /* MSI-X */ + u16 num_vec; + char *irq_name; + bool *irq_allocated; }; static inline void rvu_write64(struct rvu *rvu, u64 block, u64 offset, u64 val) diff --git a/drivers/soc/marvell/octeontx2/rvu_struct.h b/drivers/soc/marvell/octeontx2/rvu_struct.h index 4122559..1dc1518 100644 --- a/drivers/soc/marvell/octeontx2/rvu_struct.h +++ b/drivers/soc/marvell/octeontx2/rvu_struct.h @@ -33,4 +33,26 @@ enum rvu_block_addr_e { BLK_COUNT = 0xfULL, }; +/* RVU Admin function Interrupt Vector Enumeration */ +enum rvu_af_int_vec_e { + RVU_AF_INT_VEC_POISON = 0x0, + RVU_AF_INT_VEC_PFFLR = 0x1, + RVU_AF_INT_VEC_PFME = 0x2, + RVU_AF_INT_VEC_GEN = 0x3, + RVU_AF_INT_VEC_MBOX = 0x4, +}; + +/** + * RVU PF Interrupt Vector Enumeration + */ +enum rvu_pf_int_vec_e { + RVU_PF_INT_VEC_VFFLR0 = 0x0, + RVU_PF_INT_VEC_VFFLR1 = 0x1, + RVU_PF_INT_VEC_VFME0 = 0x2, + RVU_PF_INT_VEC_VFME1 = 0x3, + RVU_PF_INT_VEC_VFPF_MBOX0 = 0x4, + RVU_PF_INT_VEC_VFPF_MBOX1 = 0x5, + RVU_PF_INT_VEC_AFPF_MBOX = 0x6, +}; + #endif /* RVU_STRUCT_H */