From patchwork Mon Jun 19 17:05:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 9796929 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 74505603F5 for ; Mon, 19 Jun 2017 17:05:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 598DE1FF20 for ; Mon, 19 Jun 2017 17:05:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4E4F727F8F; Mon, 19 Jun 2017 17:05:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3EB1126E97 for ; Mon, 19 Jun 2017 17:05:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752665AbdFSRFY (ORCPT ); Mon, 19 Jun 2017 13:05:24 -0400 Received: from mail-io0-f181.google.com ([209.85.223.181]:33878 "EHLO mail-io0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752640AbdFSRFS (ORCPT ); Mon, 19 Jun 2017 13:05:18 -0400 Received: by mail-io0-f181.google.com with SMTP id i7so68377332ioe.1 for ; Mon, 19 Jun 2017 10:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sDCbiVxSRegwl7nGp1FdQHg3dgoZHGom0ao0TBjXF08=; b=JeMHNvjhQmPHxmRhbUr76XuifKOlxPYL1z1pT4NB6Dx+NGB0GgLfqMFLXLMfKsmSAR 7k20QQvYXrUTbtK4mpFhDCWWjw1PsadmyuDnQEtkvA6RYBw0V3LQp/Cfs/obz4n+DU/k s/OA1ox9Tl5MjXw7d/RH1FmEM4ZOQh2NCKfQlKlViE7Q5g87j1yfGl5w4ATBQpvwi+5k mEm+aROLkGC4qyWWOXNLyaPpYgtFaAsj6LrPiXDiDzidUjA8FBJxGxINFj7Mcq8cuhOP gat2aBajjZI81DERWizz0HIit+GKIE15ucRh/k1w2xnjMBpajSwhDexs1eCeL4qexDWK 0JGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sDCbiVxSRegwl7nGp1FdQHg3dgoZHGom0ao0TBjXF08=; b=ey0/9NeqL52ozcM4AR8jUnHnhQ300xtkcKkF06epo4evOm4r2MqbHRoWN532zCHwL3 X/VsHEDwX5GQld3WGdUEA8tJFkHZ22VRzbyZJ2jW3CuHhFNFq5s+/ZH88OCTuScXV/U6 dXRw/XueWrvT7xatMFokTf1FPz36O2DW9RsRKUYnjjk4PMj4rpXgn3aDa3MIMc1/PHUY xJG54FMFrK4ZtCn8gcMExbZ6mrmuicK771YEC/xlvRl4/VHoRCbBIsMr15qGKk2hFqJi BgXHJfLXqaH3NtDzQr04CXIea+lGCWzyUQ2YJpt5K2Qv5fsTrDLfUoEXdlCJiWTgYmDL kysQ== X-Gm-Message-State: AKS2vOw2IT3Ia3lp/Bh3XwX8jwoOw8gB5khEjBGLwpEeeXVzgeyN9wa9 nk8+phSs2HmNxGjQ X-Received: by 10.107.142.132 with SMTP id q126mr21437227iod.151.1497891917038; Mon, 19 Jun 2017 10:05:17 -0700 (PDT) Received: from localhost.localdomain ([216.160.245.98]) by smtp.gmail.com with ESMTPSA id 3sm6770522ith.7.2017.06.19.10.05.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 19 Jun 2017 10:05:16 -0700 (PDT) From: Jens Axboe To: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org Cc: adilger@dilger.ca, hch@infradead.org, martin.petersen@oracle.com, linux-nvme@lists.infradead.org, Jens Axboe Subject: [PATCH 9/9] nvme: add support for streams and directives Date: Mon, 19 Jun 2017 11:05:02 -0600 Message-Id: <1497891902-25737-10-git-send-email-axboe@kernel.dk> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1497891902-25737-1-git-send-email-axboe@kernel.dk> References: <1497891902-25737-1-git-send-email-axboe@kernel.dk> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This adds support for Directives in NVMe, particular for the Streams directive. Support for Directives is a new feature in NVMe 1.3. It allows a user to pass in information about where to store the data, so that it the device can do so most effiently. If an application is managing and writing data with different life times, mixing differently retentioned data onto the same locations on flash can cause write amplification to grow. This, in turn, will reduce performance and life time of the device. We default to allocating 4 streams, controller wide, so we can use them on all name spaces. This is configurable with the 'streams' module parameter. If a write stream is set in a write, flag is as such before sending it to the device. Signed-off-by: Jens Axboe --- drivers/nvme/host/core.c | 189 +++++++++++++++++++++++++++++++++++++++++++++-- drivers/nvme/host/nvme.h | 4 + include/linux/nvme.h | 48 ++++++++++++ 3 files changed, 236 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index aee37b73231d..fa1acfa27fa4 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -68,6 +68,10 @@ MODULE_PARM_DESC(force_apst, "allow APST for newly enumerated devices even if qu struct workqueue_struct *nvme_wq; EXPORT_SYMBOL_GPL(nvme_wq); +static bool streams = true; +module_param(streams, bool, 0644); +MODULE_PARM_DESC(streams, "use streams, if available"); + static LIST_HEAD(nvme_ctrl_list); static DEFINE_SPINLOCK(dev_list_lock); @@ -297,6 +301,141 @@ struct request *nvme_alloc_request(struct request_queue *q, } EXPORT_SYMBOL_GPL(nvme_alloc_request); +/* + + * Returns number of streams allocated for use by, or -1 on error. + */ +static int nvme_streams_allocate(struct nvme_ctrl *ctrl, unsigned int nstreams) +{ + struct nvme_command c; + union nvme_result res; + int ret; + + memset(&c, 0, sizeof(c)); + + c.directive.opcode = nvme_admin_directive_recv; + c.directive.nsid = cpu_to_le32(0xffffffff); + c.directive.doper = NVME_DIR_RCV_ST_OP_RESOURCE; + c.directive.dtype = NVME_DIR_STREAMS; + c.directive.endir = nstreams; + + ret = __nvme_submit_sync_cmd(ctrl->admin_q, &c, &res, NULL, 0, 0, + NVME_QID_ANY, 0, 0); + if (ret) + return -1; + + return le32_to_cpu(res.u32) & 0xffff; +} + +static int nvme_enable_streams(struct nvme_ctrl *ctrl) +{ + struct nvme_command c; + + memset(&c, 0, sizeof(c)); + + c.directive.opcode = nvme_admin_directive_send; + c.directive.nsid = cpu_to_le32(0xffffffff); + c.directive.doper = NVME_DIR_SND_ID_OP_ENABLE; + c.directive.dtype = NVME_DIR_IDENTIFY; + c.directive.tdtype = NVME_DIR_STREAMS; + c.directive.endir = NVME_DIR_ENDIR; + + return nvme_submit_sync_cmd(ctrl->admin_q, &c, NULL, 0); +} + +static int nvme_get_stream_params(struct nvme_ctrl *ctrl, + struct streams_directive_params *s, u32 nsid) +{ + struct nvme_command c; + + memset(&c, 0, sizeof(c)); + memset(s, 0, sizeof(*s)); + + c.directive.opcode = nvme_admin_directive_recv; + c.directive.nsid = cpu_to_le32(nsid); + c.directive.numd = sizeof(*s); + c.directive.doper = NVME_DIR_RCV_ST_OP_PARAM; + c.directive.dtype = NVME_DIR_STREAMS; + + return nvme_submit_sync_cmd(ctrl->admin_q, &c, s, sizeof(*s)); +} + +static int nvme_setup_directives(struct nvme_ctrl *ctrl) +{ + struct streams_directive_params s; + unsigned int nstreams; + int ret; + + if (!(ctrl->oacs & NVME_CTRL_OACS_DIRECTIVES)) + return 0; + if (!streams) + return 0; + + ret = nvme_enable_streams(ctrl); + if (ret) + return ret; + + ret = nvme_get_stream_params(ctrl, &s, 0xffffffff); + if (ret) + return ret; + + ctrl->nssa = le16_to_cpu(s.nssa); + + nstreams = min_t(unsigned int, ctrl->nssa, BLK_MAX_WRITE_HINTS - 1); + ret = nvme_streams_allocate(ctrl, nstreams); + if (ret < 0) + return ret; + + /* require at least 2 streams to use them effectively */ + if (ret > 1) { + ret = min(ret, BLK_MAX_WRITE_HINTS - 1); + ctrl->nr_streams = ret; + dev_info(ctrl->device, "successfully enabled %d streams\n", ret); + } + + return 0; +} + +/* + * Write hint number to stream mappings + */ +static const unsigned int stream_mappings[BLK_MAX_WRITE_HINTS][BLK_MAX_WRITE_HINTS] = { + /* 0 or 1 stream, we don't use streams */ + { 0, }, + { 0, }, + /* collapse short+medium to short, and long+extreme to medium */ + { WRITE_LIFE_NONE, WRITE_LIFE_SHORT, WRITE_LIFE_SHORT, + WRITE_LIFE_MEDIUM, WRITE_LIFE_MEDIUM }, + /* collapse long+extreme to long */ + { WRITE_LIFE_NONE, WRITE_LIFE_SHORT, WRITE_LIFE_MEDIUM, + WRITE_LIFE_LONG, WRITE_LIFE_LONG }, + /* 4 streams, no collapsing needed */ + { WRITE_LIFE_NONE, WRITE_LIFE_SHORT, WRITE_LIFE_MEDIUM, + WRITE_LIFE_LONG, WRITE_LIFE_EXTREME }, +}; + +/* + * Check if 'req' has a write hint associated with it. If it does, assign + * a valid namespace stream to the write. If we haven't setup streams yet, + * kick off configuration and ignore the hints until that has completed. + */ +static void nvme_assign_write_stream(struct nvme_ctrl *ctrl, + struct request *req, u16 *control, + u32 *dsmgmt) +{ + enum rw_hint streamid; + + streamid = opf_to_write_hint(req->cmd_flags); + if (streamid != WRITE_LIFE_NONE) { + streamid = stream_mappings[ctrl->nr_streams][streamid]; + *control |= NVME_RW_DTYPE_STREAMS; + *dsmgmt |= streamid << 16; + } + + if (streamid < ARRAY_SIZE(req->q->write_hints)) + req->q->write_hints[streamid] += blk_rq_bytes(req) >> 9; +} + static inline void nvme_setup_flush(struct nvme_ns *ns, struct nvme_command *cmnd) { @@ -348,6 +487,7 @@ static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { + struct nvme_ctrl *ctrl = ns->ctrl; u16 control = 0; u32 dsmgmt = 0; @@ -375,6 +515,9 @@ static inline blk_status_t nvme_setup_rw(struct nvme_ns *ns, cmnd->rw.slba = cpu_to_le64(nvme_block_nr(ns, blk_rq_pos(req))); cmnd->rw.length = cpu_to_le16((blk_rq_bytes(req) >> ns->lba_shift) - 1); + if (req_op(req) == REQ_OP_WRITE && ctrl->nr_streams) + nvme_assign_write_stream(ctrl, req, &control, &dsmgmt); + if (ns->ms) { switch (ns->pi_type) { case NVME_NS_DPS_PI_TYPE3: @@ -1088,14 +1231,21 @@ static void nvme_set_chunk_size(struct nvme_ns *ns) static void nvme_config_discard(struct nvme_ns *ns) { - struct nvme_ctrl *ctrl = ns->ctrl; u32 logical_block_size = queue_logical_block_size(ns->queue); + struct nvme_ctrl *ctrl = ns->ctrl; BUILD_BUG_ON(PAGE_SIZE / sizeof(struct nvme_dsm_range) < NVME_DSM_MAX_RANGES); - ns->queue->limits.discard_alignment = logical_block_size; - ns->queue->limits.discard_granularity = logical_block_size; + if (ctrl->nr_streams && ns->sws && ns->sgs) { + unsigned int sz = logical_block_size * ns->sws * ns->sgs; + + ns->queue->limits.discard_alignment = sz; + ns->queue->limits.discard_granularity = sz; + } else { + ns->queue->limits.discard_alignment = logical_block_size; + ns->queue->limits.discard_granularity = logical_block_size; + } blk_queue_max_discard_sectors(ns->queue, UINT_MAX); blk_queue_max_discard_segments(ns->queue, NVME_DSM_MAX_RANGES); queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, ns->queue); @@ -1135,6 +1285,7 @@ static int nvme_revalidate_ns(struct nvme_ns *ns, struct nvme_id_ns **id) static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id) { struct nvme_ns *ns = disk->private_data; + struct nvme_ctrl *ctrl = ns->ctrl; u16 bs; /* @@ -1149,7 +1300,7 @@ static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id) blk_mq_freeze_queue(disk->queue); - if (ns->ctrl->ops->flags & NVME_F_METADATA_SUPPORTED) + if (ctrl->ops->flags & NVME_F_METADATA_SUPPORTED) nvme_prep_integrity(disk, id, bs); blk_queue_logical_block_size(ns->queue, bs); if (ns->noiob) @@ -1161,7 +1312,7 @@ static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id) else set_capacity(disk, le64_to_cpup(&id->nsze) << (ns->lba_shift - 9)); - if (ns->ctrl->oncs & NVME_CTRL_ONCS_DSM) + if (ctrl->oncs & NVME_CTRL_ONCS_DSM) nvme_config_discard(ns); blk_mq_unfreeze_queue(disk->queue); } @@ -1766,6 +1917,7 @@ int nvme_init_identify(struct nvme_ctrl *ctrl) dev_pm_qos_hide_latency_tolerance(ctrl->device); nvme_configure_apst(ctrl); + nvme_setup_directives(ctrl); ctrl->identified = true; @@ -2158,6 +2310,32 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid) return ret; } +static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns) +{ + struct streams_directive_params s; + int ret; + + if (!ctrl->nr_streams) + return 0; + + ret = nvme_get_stream_params(ctrl, &s, ns->ns_id); + if (ret) + return ret; + + ns->sws = le32_to_cpu(s.sws); + ns->sgs = le16_to_cpu(s.sgs); + + if (ns->sws) { + unsigned int bs = 1 << ns->lba_shift; + + blk_queue_io_min(ns->queue, bs * ns->sws); + if (ns->sgs) + blk_queue_io_opt(ns->queue, bs * ns->sws * ns->sgs); + } + + return 0; +} + static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) { struct nvme_ns *ns; @@ -2187,6 +2365,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) blk_queue_logical_block_size(ns->queue, 1 << ns->lba_shift); nvme_set_queue_limits(ctrl, ns->queue); + nvme_setup_streams_ns(ctrl, ns); sprintf(disk_name, "nvme%dn%d", ctrl->instance, ns->instance); diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index ec8c7363934d..f616835afc4c 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -147,6 +147,8 @@ struct nvme_ctrl { u16 oncs; u16 vid; u16 oacs; + u16 nssa; + u16 nr_streams; atomic_t abort_limit; u8 event_limit; u8 vwc; @@ -199,6 +201,8 @@ struct nvme_ns { unsigned ns_id; int lba_shift; u16 ms; + u16 sgs; + u32 sws; bool ext; u8 pi_type; unsigned long flags; diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 291587a0743f..f516a975bb21 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -253,6 +253,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, + NVME_CTRL_OACS_DIRECTIVES = 1 << 5, NVME_CTRL_OACS_DBBUF_SUPP = 1 << 7, }; @@ -304,6 +305,19 @@ enum { }; enum { + NVME_DIR_IDENTIFY = 0x00, + NVME_DIR_STREAMS = 0x01, + NVME_DIR_SND_ID_OP_ENABLE = 0x01, + NVME_DIR_SND_ST_OP_REL_ID = 0x01, + NVME_DIR_SND_ST_OP_REL_RSC = 0x02, + NVME_DIR_RCV_ID_OP_PARAM = 0x01, + NVME_DIR_RCV_ST_OP_PARAM = 0x01, + NVME_DIR_RCV_ST_OP_STATUS = 0x02, + NVME_DIR_RCV_ST_OP_RESOURCE = 0x03, + NVME_DIR_ENDIR = 0x01, +}; + +enum { NVME_NS_FEAT_THIN = 1 << 0, NVME_NS_FLBAS_LBA_MASK = 0xf, NVME_NS_FLBAS_META_EXT = 0x10, @@ -560,6 +574,7 @@ enum { NVME_RW_PRINFO_PRCHK_APP = 1 << 11, NVME_RW_PRINFO_PRCHK_GUARD = 1 << 12, NVME_RW_PRINFO_PRACT = 1 << 13, + NVME_RW_DTYPE_STREAMS = 1 << 4, }; struct nvme_dsm_cmd { @@ -634,6 +649,8 @@ enum nvme_admin_opcode { nvme_admin_download_fw = 0x11, nvme_admin_ns_attach = 0x15, nvme_admin_keep_alive = 0x18, + nvme_admin_directive_send = 0x19, + nvme_admin_directive_recv = 0x1a, nvme_admin_dbbuf = 0x7C, nvme_admin_format_nvm = 0x80, nvme_admin_security_send = 0x81, @@ -797,6 +814,24 @@ struct nvme_get_log_page_command { __u32 rsvd14[2]; }; +struct nvme_directive_cmd { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2[2]; + union nvme_data_ptr dptr; + __le32 numd; + __u8 doper; + __u8 dtype; + __le16 dspec; + __u8 endir; + __u8 tdtype; + __u16 rsvd15; + + __u32 rsvd16[3]; +}; + /* * Fabrics subcommands. */ @@ -927,6 +962,18 @@ struct nvme_dbbuf { __u32 rsvd12[6]; }; +struct streams_directive_params { + __u16 msl; + __u16 nssa; + __u16 nsso; + __u8 rsvd[10]; + __u32 sws; + __u16 sgs; + __u16 nsa; + __u16 nso; + __u8 rsvd2[6]; +}; + struct nvme_command { union { struct nvme_common_command common; @@ -947,6 +994,7 @@ struct nvme_command { struct nvmf_property_set_command prop_set; struct nvmf_property_get_command prop_get; struct nvme_dbbuf dbbuf; + struct nvme_directive_cmd directive; }; };