From patchwork Wed Apr 12 19:15:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uma Krishnan X-Patchwork-Id: 9678265 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8191760382 for ; Wed, 12 Apr 2017 19:16:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D92928633 for ; Wed, 12 Apr 2017 19:16:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 627B128637; Wed, 12 Apr 2017 19:16:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A411728633 for ; Wed, 12 Apr 2017 19:16:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754640AbdDLTQK (ORCPT ); Wed, 12 Apr 2017 15:16:10 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:43429 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752299AbdDLTQJ (ORCPT ); Wed, 12 Apr 2017 15:16:09 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v3CJ98ER079682 for ; Wed, 12 Apr 2017 15:16:09 -0400 Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.149]) by mx0b-001b2d01.pphosted.com with ESMTP id 29snvj5ah9-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 12 Apr 2017 15:16:08 -0400 Received: from localhost by e31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Apr 2017 13:16:07 -0600 Received: from b03cxnp08026.gho.boulder.ibm.com (9.17.130.18) by e31.co.us.ibm.com (192.168.1.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 12 Apr 2017 13:16:04 -0600 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v3CJFtRm11338110; Wed, 12 Apr 2017 12:15:55 -0700 Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6F98ABE038; Wed, 12 Apr 2017 13:15:55 -0600 (MDT) Received: from p8tul1-build.aus.stglabs.ibm.com (unknown [9.3.141.206]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP id CA7C5BE040; Wed, 12 Apr 2017 13:15:54 -0600 (MDT) From: Uma Krishnan To: linux-scsi@vger.kernel.org, James Bottomley , "Martin K. Petersen" , "Matthew R. Ochs" , "Manoj N. Kumar" Cc: linuxppc-dev@lists.ozlabs.org, Ian Munsie , Andrew Donnellan , Frederic Barrat , Christophe Lombard Subject: [PATCH 16/17] cxlflash: Add hardware queues attribute Date: Wed, 12 Apr 2017 14:15:53 -0500 X-Mailer: git-send-email 2.1.0 In-Reply-To: <1492024215-55579-1-git-send-email-ukrishn@linux.vnet.ibm.com> References: <1492024215-55579-1-git-send-email-ukrishn@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17041219-8235-0000-0000-00000B4D42EB X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006924; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00846635; UDB=6.00417627; IPR=6.00625073; BA=6.00005286; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015025; XFM=3.00000013; UTC=2017-04-12 19:16:05 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17041219-8236-0000-0000-00003B172A3F Message-Id: <1492024553-56230-1-git-send-email-ukrishn@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-04-12_15:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1704120157 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: "Matthew R. Ochs" As staging for supporting multiple hardware queues, add an attribute to show and set the current number of hardware queues for the host. Support specifying a hard limit or a CPU affinitized value. This will allow the number of hardware queues to be tuned by a system administrator. Signed-off-by: Matthew R. Ochs Signed-off-by: Uma Krishnan --- drivers/scsi/cxlflash/common.h | 10 ++-- drivers/scsi/cxlflash/main.c | 112 ++++++++++++++++++++++++++++++++++++----- 2 files changed, 106 insertions(+), 16 deletions(-) diff --git a/drivers/scsi/cxlflash/common.h b/drivers/scsi/cxlflash/common.h index b5858ae..8fd7a1f 100644 --- a/drivers/scsi/cxlflash/common.h +++ b/drivers/scsi/cxlflash/common.h @@ -60,7 +60,9 @@ extern const struct file_operations cxlflash_cxl_fops; /* SQ for master issued cmds */ #define NUM_SQ_ENTRY CXLFLASH_MAX_CMDS -#define CXLFLASH_NUM_HWQS 1 +/* Hardware queue definitions */ +#define CXLFLASH_DEF_HWQS 1 +#define CXLFLASH_MAX_HWQS 8 #define PRIMARY_HWQ 0 @@ -201,7 +203,7 @@ struct hwq { } __aligned(cache_line_size()); struct afu { - struct hwq hwqs[CXLFLASH_NUM_HWQS]; + struct hwq hwqs[CXLFLASH_MAX_HWQS]; int (*send_cmd)(struct afu *, struct afu_cmd *); void (*context_reset)(struct afu_cmd *); @@ -211,6 +213,8 @@ struct afu { atomic_t cmds_active; /* Number of currently active AFU commands */ u64 hb; u32 internal_lun; /* User-desired LUN mode for this AFU */ + u32 num_hwqs; /* Number of hardware queues */ + u32 desired_hwqs; /* Desired h/w queues, effective on AFU reset */ char version[16]; u64 interface_version; @@ -221,7 +225,7 @@ struct afu { static inline struct hwq *get_hwq(struct afu *afu, u32 index) { - WARN_ON(index >= CXLFLASH_NUM_HWQS); + WARN_ON(index >= CXLFLASH_MAX_HWQS); return &afu->hwqs[index]; } diff --git a/drivers/scsi/cxlflash/main.c b/drivers/scsi/cxlflash/main.c index 5d06869..113797a 100644 --- a/drivers/scsi/cxlflash/main.c +++ b/drivers/scsi/cxlflash/main.c @@ -566,7 +566,7 @@ static void stop_afu(struct cxlflash_cfg *cfg) ssleep(1); if (afu_is_irqpoll_enabled(afu)) { - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); irq_poll_disable(&hwq->irqpoll); @@ -676,13 +676,13 @@ static void term_afu(struct cxlflash_cfg *cfg) * 2) Unmap the problem state area * 3) Stop each master context */ - for (k = CXLFLASH_NUM_HWQS - 1; k >= 0; k--) + for (k = cfg->afu->num_hwqs - 1; k >= 0; k--) term_intr(cfg, UNMAP_THREE, k); if (cfg->afu) stop_afu(cfg); - for (k = CXLFLASH_NUM_HWQS - 1; k >= 0; k--) + for (k = cfg->afu->num_hwqs - 1; k >= 0; k--) term_mc(cfg, k); dev_dbg(dev, "%s: returning\n", __func__); @@ -823,6 +823,7 @@ static int alloc_mem(struct cxlflash_cfg *cfg) goto out; } cfg->afu->parent = cfg; + cfg->afu->desired_hwqs = CXLFLASH_DEF_HWQS; cfg->afu->afu_map = NULL; out: return rc; @@ -1116,7 +1117,7 @@ static void afu_err_intr_init(struct afu *afu) /* IOARRIN yet), so there is nothing to clear. */ /* set LISN#, it is always sent to the context that wrote IOARRIN */ - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); writeq_be(SISL_MSI_SYNC_ERROR, &hwq->host_map->ctx_ctrl); @@ -1551,7 +1552,7 @@ static void init_pcr(struct cxlflash_cfg *cfg) } /* Copy frequently used fields into hwq */ - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); hwq->ctx_hndl = (u16) cxl_process_element(hwq->ctx); @@ -1586,7 +1587,7 @@ static int init_global(struct cxlflash_cfg *cfg) } /* Set up RRQ and SQ in HWQ for master issued cmds */ - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); hmap = hwq->host_map; @@ -1640,7 +1641,7 @@ static int init_global(struct cxlflash_cfg *cfg) /* Set up master's own CTX_CAP to allow real mode, host translation */ /* tables, afu cmds and read/write GSCSI cmds. */ /* First, unlock ctx_cap write by reading mbox */ - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); (void)readq_be(&hwq->ctrl_map->mbox_r); /* unlock ctx_cap */ @@ -1670,7 +1671,7 @@ static int start_afu(struct cxlflash_cfg *cfg) init_pcr(cfg); /* Initialize each HWQ */ - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); /* After an AFU reset, RRQ entries are stale, clear them */ @@ -1888,7 +1889,8 @@ static int init_afu(struct cxlflash_cfg *cfg) cxl_perst_reloads_same_image(cfg->cxl_afu, true); - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + afu->num_hwqs = afu->desired_hwqs; + for (i = 0; i < afu->num_hwqs; i++) { rc = init_mc(cfg, i); if (rc) { dev_err(dev, "%s: init_mc failed rc=%d index=%d\n", @@ -1939,7 +1941,7 @@ static int init_afu(struct cxlflash_cfg *cfg) } afu_err_intr_init(cfg->afu); - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); spin_lock_init(&hwq->rrin_slock); @@ -1953,7 +1955,7 @@ static int init_afu(struct cxlflash_cfg *cfg) return rc; err1: - for (i = CXLFLASH_NUM_HWQS - 1; i >= 0; i--) { + for (i = afu->num_hwqs - 1; i >= 0; i--) { term_intr(cfg, UNMAP_THREE, i); term_mc(cfg, i); } @@ -2550,7 +2552,7 @@ static ssize_t irqpoll_weight_store(struct device *dev, } if (afu_is_irqpoll_enabled(afu)) { - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); irq_poll_disable(&hwq->irqpoll); @@ -2560,7 +2562,7 @@ static ssize_t irqpoll_weight_store(struct device *dev, afu->irqpoll_weight = weight; if (weight > 0) { - for (i = 0; i < CXLFLASH_NUM_HWQS; i++) { + for (i = 0; i < afu->num_hwqs; i++) { hwq = get_hwq(afu, i); irq_poll_init(&hwq->irqpoll, weight, cxlflash_irqpoll); @@ -2571,6 +2573,88 @@ static ssize_t irqpoll_weight_store(struct device *dev, } /** + * num_hwqs_show() - presents the number of hardware queues for the host + * @dev: Generic device associated with the host. + * @attr: Device attribute representing the number of hardware queues. + * @buf: Buffer of length PAGE_SIZE to report back the number of hardware + * queues in ASCII. + * + * Return: The size of the ASCII string returned in @buf. + */ +static ssize_t num_hwqs_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxlflash_cfg *cfg = shost_priv(class_to_shost(dev)); + struct afu *afu = cfg->afu; + + return scnprintf(buf, PAGE_SIZE, "%u\n", afu->num_hwqs); +} + +/** + * num_hwqs_store() - sets the number of hardware queues for the host + * @dev: Generic device associated with the host. + * @attr: Device attribute representing the number of hardware queues. + * @buf: Buffer of length PAGE_SIZE containing the number of hardware + * queues in ASCII. + * @count: Length of data resizing in @buf. + * + * n > 0: num_hwqs = n + * n = 0: num_hwqs = num_online_cpus() + * n < 0: num_online_cpus() / abs(n) + * + * Return: The size of the ASCII string returned in @buf. + */ +static ssize_t num_hwqs_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct cxlflash_cfg *cfg = shost_priv(class_to_shost(dev)); + struct afu *afu = cfg->afu; + int rc; + int nhwqs, num_hwqs; + + rc = kstrtoint(buf, 10, &nhwqs); + if (rc) + return -EINVAL; + + if (nhwqs >= 1) + num_hwqs = nhwqs; + else if (nhwqs == 0) + num_hwqs = num_online_cpus(); + else + num_hwqs = num_online_cpus() / abs(nhwqs); + + afu->desired_hwqs = min(num_hwqs, CXLFLASH_MAX_HWQS); + WARN_ON_ONCE(afu->desired_hwqs == 0); + +retry: + switch (cfg->state) { + case STATE_NORMAL: + cfg->state = STATE_RESET; + drain_ioctls(cfg); + cxlflash_mark_contexts_error(cfg); + rc = afu_reset(cfg); + if (rc) + cfg->state = STATE_FAILTERM; + else + cfg->state = STATE_NORMAL; + wake_up_all(&cfg->reset_waitq); + break; + case STATE_RESET: + wait_event(cfg->reset_waitq, cfg->state != STATE_RESET); + if (cfg->state == STATE_NORMAL) + goto retry; + default: + /* Ideally should not happen */ + dev_err(dev, "%s: Device is not ready, state=%d\n", + __func__, cfg->state); + break; + } + + return count; +} + +/** * mode_show() - presents the current mode of the device * @dev: Generic device associated with the device. * @attr: Device attribute representing the device mode. @@ -2601,6 +2685,7 @@ static DEVICE_ATTR_RO(port1_lun_table); static DEVICE_ATTR_RO(port2_lun_table); static DEVICE_ATTR_RO(port3_lun_table); static DEVICE_ATTR_RW(irqpoll_weight); +static DEVICE_ATTR_RW(num_hwqs); static struct device_attribute *cxlflash_host_attrs[] = { &dev_attr_port0, @@ -2614,6 +2699,7 @@ static struct device_attribute *cxlflash_host_attrs[] = { &dev_attr_port2_lun_table, &dev_attr_port3_lun_table, &dev_attr_irqpoll_weight, + &dev_attr_num_hwqs, NULL };