From patchwork Sun Mar 31 19:30:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minwoo Im X-Patchwork-Id: 13612261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6BC14CD1284 for ; Sun, 31 Mar 2024 19:32:11 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rr0tH-0000IX-RE; Sun, 31 Mar 2024 15:30:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rr0tG-0000IG-AZ; Sun, 31 Mar 2024 15:30:50 -0400 Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rr0tE-0001Pv-OP; Sun, 31 Mar 2024 15:30:50 -0400 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-5d42e7ab8a9so1813959a12.3; Sun, 31 Mar 2024 12:30:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711913446; x=1712518246; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=js9MPuxE6HDFGb/bm1Qi8CQe6wvYU0HraTNqQ6YK0o4=; b=akEv1Ton/31SWx/CfmZwlLOCVrUGPWvM9I0/EGbR5N9gItN5XwLLSacCg3lqNVGpuK 4j/MTpX4D05EwrnSBzQb+ZfPDn17C5fOADE0JoTJIEVO4H8+05YKpKNJ3wFpMuU/M/+G FNNXm2aPQTav24E2QGytH87QgJgd7pTKUIs1Qpq4vX6a48QFEkQhEm5z19oPcAU1B8eq hH2+PT3YVFpjC78Rd0AtLhUaGry2nTMb8SiY8D6wY5/xWtzd4dvdDTQgqLlyLT/olK0Z F8xnRxegrQ/rPPUJSZHRmIniDCDW0TSjtl57wlZ3V8BzJYM/3b2LIcnLfw8RrvX8QLfr nXhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711913446; x=1712518246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=js9MPuxE6HDFGb/bm1Qi8CQe6wvYU0HraTNqQ6YK0o4=; b=ouLSxUM86S5lt3UwqKyNGZKlO5b+vbHEnuIy20CjMzPSujiFbly6B86z/8tQdLrioo FmviPjEVT5ewTduSUn0SoCuvod+w/eeOIZGnltTCNlxb3m60FEAkqJZVW5yfT/ps74hO w0juXFgjfNkg6IMbJHq/0qDl/k4xanOUtRQL5BB6aFsDad/AzK5flakLZ3xUauhEsNJZ aJFNhBKZY+AXFQLuxy9fVKMr2O0+gDof3DqTILuDurFvgRdDAT1E0qecIq820LcCVuhR 4g/lBKoI+6hhFjS//EhLKqhS6D6cMwGK53jTDkxSP1AnNAt9mKd43cT+yjesp6ZFi6Ya lsrQ== X-Forwarded-Encrypted: i=1; AJvYcCU9+Hmwqx6FknrMeew7z2IdFrYf8O35sBfFWaHmDqyjF4B/2D0OxPCIc5eo6VAa0hE6HFRg7MvUuDl8q47pmEQItDWZ9GU= X-Gm-Message-State: AOJu0YwPRQwPsOz4m2/1lZ0sbbw8RFp920LV1yhTHaCYB9tshJvxuecO e2nu+El4Ifvm+GqNiy2OqEKFjz2CybUcwWZ6oTHXRcabxuObE9Hu X-Google-Smtp-Source: AGHT+IFhf0fx9aXtbewMkV+ijIhcG6WT5qJeTWTELqHJyNpgSsNuDN1yJYSdtKa8UdQqlZ/bANbIqA== X-Received: by 2002:a05:6a20:6a1b:b0:1a5:6f67:3f63 with SMTP id p27-20020a056a206a1b00b001a56f673f63mr7724117pzk.0.1711913446195; Sun, 31 Mar 2024 12:30:46 -0700 (PDT) Received: from localhost ([116.121.76.56]) by smtp.gmail.com with ESMTPSA id s16-20020a170902ea1000b001e0afbb6d99sm7232394plg.47.2024.03.31.12.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Mar 2024 12:30:45 -0700 (PDT) From: Minwoo Im To: Klaus Jensen , Keith Busch Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Minwoo Im , Klaus Jensen Subject: [PATCH v2 1/4] hw/nvme: add Identify Endurance Group List Date: Mon, 1 Apr 2024 04:30:29 +0900 Message-Id: <20240331193032.5186-2-minwoo.im.dev@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240331193032.5186-1-minwoo.im.dev@gmail.com> References: <20240331193032.5186-1-minwoo.im.dev@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::533; envelope-from=minwoo.im.dev@gmail.com; helo=mail-pg1-x533.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Minwoo Im Commit 73064edfb864 ("hw/nvme: flexible data placement emulation") intorudced NVMe FDP feature to nvme-subsys and nvme-ctrl with a single endurance group #1 supported. This means that controller should return proper identify data to host with Identify Endurance Group List (CNS 19h). But, yes, only just for the endurance group #1. This patch allows host applications to ask for which endurance group is available and utilize FDP through that endurance group. Signed-off-by: Minwoo Im Reviewed-by: Klaus Jensen --- hw/nvme/ctrl.c | 22 ++++++++++++++++++++++ include/block/nvme.h | 1 + 2 files changed, 23 insertions(+) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index f026245d1e9e..cfe53a358871 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -5629,6 +5629,26 @@ static uint16_t nvme_identify_nslist_csi(NvmeCtrl *n, NvmeRequest *req, return nvme_c2h(n, list, data_len, req); } +static uint16_t nvme_endurance_group_list(NvmeCtrl *n, NvmeRequest *req) +{ + uint16_t list[NVME_CONTROLLER_LIST_SIZE] = {}; + uint16_t *nr_ids = &list[0]; + uint16_t *ids = &list[1]; + uint16_t endgid = le32_to_cpu(req->cmd.cdw11) & 0xffff; + + /* + * The current nvme-subsys only supports Endurance Group #1. + */ + if (!endgid) { + *nr_ids = 1; + ids[0] = 1; + } else { + *nr_ids = 0; + } + + return nvme_c2h(n, list, sizeof(list), req); +} + static uint16_t nvme_identify_ns_descr_list(NvmeCtrl *n, NvmeRequest *req) { NvmeNamespace *ns; @@ -5732,6 +5752,8 @@ static uint16_t nvme_identify(NvmeCtrl *n, NvmeRequest *req) return nvme_identify_nslist(n, req, false); case NVME_ID_CNS_CS_NS_ACTIVE_LIST: return nvme_identify_nslist_csi(n, req, true); + case NVME_ID_CNS_ENDURANCE_GROUP_LIST: + return nvme_endurance_group_list(n, req); case NVME_ID_CNS_CS_NS_PRESENT_LIST: return nvme_identify_nslist_csi(n, req, false); case NVME_ID_CNS_NS_DESCR_LIST: diff --git a/include/block/nvme.h b/include/block/nvme.h index bb231d0b9ad0..7c77d38174a7 100644 --- a/include/block/nvme.h +++ b/include/block/nvme.h @@ -1074,6 +1074,7 @@ enum NvmeIdCns { NVME_ID_CNS_CTRL_LIST = 0x13, NVME_ID_CNS_PRIMARY_CTRL_CAP = 0x14, NVME_ID_CNS_SECONDARY_CTRL_LIST = 0x15, + NVME_ID_CNS_ENDURANCE_GROUP_LIST = 0x19, NVME_ID_CNS_CS_NS_PRESENT_LIST = 0x1a, NVME_ID_CNS_CS_NS_PRESENT = 0x1b, NVME_ID_CNS_IO_COMMAND_SET = 0x1c, From patchwork Sun Mar 31 19:30:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minwoo Im X-Patchwork-Id: 13612260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3670FCD128A for ; Sun, 31 Mar 2024 19:31:56 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rr0tK-0000JO-UG; Sun, 31 Mar 2024 15:30:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rr0tJ-0000J3-1m; Sun, 31 Mar 2024 15:30:53 -0400 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rr0tH-0001QE-5J; Sun, 31 Mar 2024 15:30:52 -0400 Received: by mail-pg1-x52f.google.com with SMTP id 41be03b00d2f7-5e42b4bbfa4so2024589a12.1; Sun, 31 Mar 2024 12:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711913449; x=1712518249; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P/Oz4nAb4eNMYZyk5vmRGZoOtdjU0VylSe0dYHONRL4=; b=OqWl/1u2clk8ud8gIBINY6l1/ZzXPusrfirPe/LFGitR9/QzR31sdwsI9QiFNFxfJI PVY4DwpCzZLzwv6SebjEYWox5jdjpdO70hfeqR4AVNZrIq+d4+iMplZo/vQ7pHDB4FK/ DVq4E7xIFH8g5iC2CE1CSMi9IqNnEfFnDb3c1tJ7XamAbWZjxuVwV6natnu1FiR5xqv4 5vdQoNVsNQuWCv4YNK7WgTqz9yNW/ZE/Nlv8X6m+u9IkE353FPFsYatot5gt3HnzMyhb /CEIuCtipwYO6C/lWyS0AuvKKIpzQasU3iYxoApEN90JuASWTOeblKvzdSGpuwZehLsB sCcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711913449; x=1712518249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P/Oz4nAb4eNMYZyk5vmRGZoOtdjU0VylSe0dYHONRL4=; b=oz2WUUvD4eGEET45lKkjCIBovETopvS2Otl1Zf/I8PRLb71YNjY6/Xs7G25Awy4HDi bzciKOFWZxLNlgQWOXNHP4OlNOC14gm66ruoKspII6JntSef93Zpp1mAgNkzVZZ/ZKTA wqc/bxYhCQOEjog/NTBDBIauK+AJGQqoBxnnVNB0oOUVJK8NKcTvM9LtNeQpRfUONwzS lWmUADXmQ5EFxlUHGyMNL0eax3KHKmjYfP0ErKDsnPlqtHNotxLLjh9WiXHmiC37lZnn 4hTOtSiDt00nOF0f+wrAQIOuaVJWZGqhinuHub4Vsj0fV6lfcT8V4fanViIKap3BMaKQ 8VHw== X-Forwarded-Encrypted: i=1; AJvYcCUyAB6yS+MEMMAjg6tbigVGE+rPvh+hpell8D6pJQ1Phk3e1O/BWF+IDRMqOzqKjiEy8ImZEMv4P/Lfq4usmq0+ecGuwIw= X-Gm-Message-State: AOJu0Yy4jDjRGxfeDD1GfoSykdwZ3GalICsVqKSfL4rYUpm85+gRUIyJ 7BzZHTSdTBxrvYsOzGSt7qarYTMNi557LCe1UqoBjVHcY2vVg+YP X-Google-Smtp-Source: AGHT+IG2j1TIVw0Vgmf4x6RolE+0PsvCpr3ZNbaC/aL/IT5s+NDmMlzdmpUnJl10HcLJLa0QppRY5g== X-Received: by 2002:a17:903:32cb:b0:1e0:7bbf:bef4 with SMTP id i11-20020a17090332cb00b001e07bbfbef4mr9192272plr.41.1711913449363; Sun, 31 Mar 2024 12:30:49 -0700 (PDT) Received: from localhost ([116.121.76.56]) by smtp.gmail.com with ESMTPSA id h15-20020a170902680f00b001dd6f305a81sm7339566plk.293.2024.03.31.12.30.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Mar 2024 12:30:49 -0700 (PDT) From: Minwoo Im To: Klaus Jensen , Keith Busch Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Minwoo Im Subject: [PATCH v2 2/4] hw/nvme: separate identify data for sec. ctrl list Date: Mon, 1 Apr 2024 04:30:30 +0900 Message-Id: <20240331193032.5186-3-minwoo.im.dev@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240331193032.5186-1-minwoo.im.dev@gmail.com> References: <20240331193032.5186-1-minwoo.im.dev@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::52f; envelope-from=minwoo.im.dev@gmail.com; helo=mail-pg1-x52f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Minwoo Im Secondary controller list for virtualization has been managed by Identify Secondary Controller List data structure with NvmeSecCtrlList where up to 127 secondary controller entries can be managed. The problem hasn't arisen so far because NVME_MAX_VFS has been 127. This patch separated identify data itself from the actual secondary controller list managed by controller to support more than 127 secondary controllers with the following patch. This patch reused NvmeSecCtrlEntry structure to manage all the possible secondary controllers, and copy entries to identify data structure when the command comes in. Signed-off-by: Minwoo Im --- hw/nvme/ctrl.c | 21 ++++++++++----------- hw/nvme/nvme.h | 14 ++++++++------ hw/nvme/subsys.c | 8 ++++---- 3 files changed, 22 insertions(+), 21 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index cfe53a358871..7e60bc9f2075 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -219,7 +219,6 @@ #define NVME_TEMPERATURE_CRITICAL 0x175 #define NVME_NUM_FW_SLOTS 1 #define NVME_DEFAULT_MAX_ZA_SIZE (128 * KiB) -#define NVME_MAX_VFS 127 #define NVME_VF_RES_GRANULARITY 1 #define NVME_VF_OFFSET 0x1 #define NVME_VF_STRIDE 1 @@ -5480,14 +5479,14 @@ static uint16_t nvme_identify_sec_ctrl_list(NvmeCtrl *n, NvmeRequest *req) NvmeIdentify *c = (NvmeIdentify *)&req->cmd; uint16_t pri_ctrl_id = le16_to_cpu(n->pri_ctrl_cap.cntlid); uint16_t min_id = le16_to_cpu(c->ctrlid); - uint8_t num_sec_ctrl = n->sec_ctrl_list.numcntl; + uint8_t num_sec_ctrl = n->nr_sec_ctrls; NvmeSecCtrlList list = {0}; uint8_t i; for (i = 0; i < num_sec_ctrl; i++) { - if (n->sec_ctrl_list.sec[i].scid >= min_id) { - list.numcntl = num_sec_ctrl - i; - memcpy(&list.sec, n->sec_ctrl_list.sec + i, + if (n->sec_ctrl_list[i].scid >= min_id) { + list.numcntl = MIN(num_sec_ctrl - i, 127); + memcpy(&list.sec, n->sec_ctrl_list + i, list.numcntl * sizeof(NvmeSecCtrlEntry)); break; } @@ -7132,8 +7131,8 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetType rst) if (n->params.sriov_max_vfs) { if (!pci_is_vf(pci_dev)) { - for (i = 0; i < n->sec_ctrl_list.numcntl; i++) { - sctrl = &n->sec_ctrl_list.sec[i]; + for (i = 0; i < n->nr_sec_ctrls; i++) { + sctrl = &n->sec_ctrl_list[i]; nvme_virt_set_state(n, le16_to_cpu(sctrl->scid), false); } @@ -7921,7 +7920,7 @@ static bool nvme_check_params(NvmeCtrl *n, Error **errp) static void nvme_init_state(NvmeCtrl *n) { NvmePriCtrlCap *cap = &n->pri_ctrl_cap; - NvmeSecCtrlList *list = &n->sec_ctrl_list; + NvmeSecCtrlEntry *list = n->sec_ctrl_list; NvmeSecCtrlEntry *sctrl; PCIDevice *pci = PCI_DEVICE(n); uint8_t max_vfs; @@ -7946,9 +7945,9 @@ static void nvme_init_state(NvmeCtrl *n) n->aer_reqs = g_new0(NvmeRequest *, n->params.aerl + 1); QTAILQ_INIT(&n->aer_queue); - list->numcntl = cpu_to_le16(max_vfs); + n->nr_sec_ctrls = max_vfs; for (i = 0; i < max_vfs; i++) { - sctrl = &list->sec[i]; + sctrl = &list[i]; sctrl->pcid = cpu_to_le16(n->cntlid); sctrl->vfn = cpu_to_le16(i + 1); } @@ -8505,7 +8504,7 @@ static void nvme_sriov_pre_write_ctrl(PCIDevice *dev, uint32_t address, if (!(val & PCI_SRIOV_CTRL_VFE)) { num_vfs = pci_get_word(dev->config + sriov_cap + PCI_SRIOV_NUM_VF); for (i = 0; i < num_vfs; i++) { - sctrl = &n->sec_ctrl_list.sec[i]; + sctrl = &n->sec_ctrl_list[i]; nvme_virt_set_state(n, le16_to_cpu(sctrl->scid), false); } } diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 5f2ae7b28b9c..02c11d909cd1 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -26,6 +26,7 @@ #define NVME_MAX_CONTROLLERS 256 #define NVME_MAX_NAMESPACES 256 +#define NVME_MAX_VFS 127 #define NVME_EUI64_DEFAULT ((uint64_t)0x5254000000000000) #define NVME_FDP_MAX_EVENTS 63 #define NVME_FDP_MAXPIDS 128 @@ -597,7 +598,8 @@ typedef struct NvmeCtrl { } features; NvmePriCtrlCap pri_ctrl_cap; - NvmeSecCtrlList sec_ctrl_list; + uint32_t nr_sec_ctrls; + NvmeSecCtrlEntry sec_ctrl_list[NVME_MAX_VFS]; struct { uint16_t vqrfap; uint16_t virfap; @@ -647,7 +649,7 @@ static inline NvmeSecCtrlEntry *nvme_sctrl(NvmeCtrl *n) NvmeCtrl *pf = NVME(pcie_sriov_get_pf(pci_dev)); if (pci_is_vf(pci_dev)) { - return &pf->sec_ctrl_list.sec[pcie_sriov_vf_number(pci_dev)]; + return &pf->sec_ctrl_list[pcie_sriov_vf_number(pci_dev)]; } return NULL; @@ -656,12 +658,12 @@ static inline NvmeSecCtrlEntry *nvme_sctrl(NvmeCtrl *n) static inline NvmeSecCtrlEntry *nvme_sctrl_for_cntlid(NvmeCtrl *n, uint16_t cntlid) { - NvmeSecCtrlList *list = &n->sec_ctrl_list; + NvmeSecCtrlEntry *list = n->sec_ctrl_list; uint8_t i; - for (i = 0; i < list->numcntl; i++) { - if (le16_to_cpu(list->sec[i].scid) == cntlid) { - return &list->sec[i]; + for (i = 0; i < n->nr_sec_ctrls; i++) { + if (le16_to_cpu(list[i].scid) == cntlid) { + return &list[i]; } } diff --git a/hw/nvme/subsys.c b/hw/nvme/subsys.c index d30bb8bfd5b4..561ed04a5317 100644 --- a/hw/nvme/subsys.c +++ b/hw/nvme/subsys.c @@ -17,13 +17,13 @@ static int nvme_subsys_reserve_cntlids(NvmeCtrl *n, int start, int num) { NvmeSubsystem *subsys = n->subsys; - NvmeSecCtrlList *list = &n->sec_ctrl_list; + NvmeSecCtrlEntry *list = n->sec_ctrl_list; NvmeSecCtrlEntry *sctrl; int i, cnt = 0; for (i = start; i < ARRAY_SIZE(subsys->ctrls) && cnt < num; i++) { if (!subsys->ctrls[i]) { - sctrl = &list->sec[cnt]; + sctrl = &list[cnt]; sctrl->scid = cpu_to_le16(i); subsys->ctrls[i] = SUBSYS_SLOT_RSVD; cnt++; @@ -36,12 +36,12 @@ static int nvme_subsys_reserve_cntlids(NvmeCtrl *n, int start, int num) static void nvme_subsys_unreserve_cntlids(NvmeCtrl *n) { NvmeSubsystem *subsys = n->subsys; - NvmeSecCtrlList *list = &n->sec_ctrl_list; + NvmeSecCtrlEntry *list = n->sec_ctrl_list; NvmeSecCtrlEntry *sctrl; int i, cntlid; for (i = 0; i < n->params.sriov_max_vfs; i++) { - sctrl = &list->sec[i]; + sctrl = &list[i]; cntlid = le16_to_cpu(sctrl->scid); if (cntlid) { From patchwork Sun Mar 31 19:30:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minwoo Im X-Patchwork-Id: 13612259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 982EBCD1284 for ; Sun, 31 Mar 2024 19:31:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rr0tM-0000KT-K5; Sun, 31 Mar 2024 15:30:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rr0tL-0000Jc-6D; Sun, 31 Mar 2024 15:30:55 -0400 Received: from mail-il1-x12f.google.com ([2607:f8b0:4864:20::12f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rr0tJ-0001QY-Io; Sun, 31 Mar 2024 15:30:54 -0400 Received: by mail-il1-x12f.google.com with SMTP id e9e14a558f8ab-368cd5ddb65so9149975ab.0; Sun, 31 Mar 2024 12:30:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711913452; x=1712518252; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wq1YmLBqyP4iUMA93VUo8jDUZAGxmCsbTe8wz4eZ4Wg=; b=DLZd3sm2yQRwYrJiHMzunx1lNLYS9QGAaPVA5kafOKxpbNs+Cak5743EZcvk0P3cuC CpzEGuI4j5xsfKojCkjXbURR5K3K6r0C25ZUQ/QcYVLUPxQ0x4O+Y9eOFX90MT/58K4M O3us3fNoxKwbaOoRrDYu6uPax5S2KpeLSzTrfHW6zSBGeuuPeUycCKrmrsPXPSlB5MM3 Gptux26L4ZNV9MhOXQwE1y2bmAlisH2eRnT9cHTsoqYbeak75xhWZ5O2SR0xgh9E80nu JBSkyJpZ69CJ7VSCyzAGscToV5h5dpRStlendk24GJa1rS8Gfy5AtknKdMXrxMdNut/y kSkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711913452; x=1712518252; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wq1YmLBqyP4iUMA93VUo8jDUZAGxmCsbTe8wz4eZ4Wg=; b=vHORQQd9a+q/gMtQI4zpWBK2YAqrNpDfPZEgNkMxbl5jFuHieKghlY0IqnA0nHaOOb KOcFpaq4X8+E7k8pUUN6dz08FNLaN813Dh6DHPjDM6OQ6G+bX5IJ3jzk5YcUDw+smH1l isyQSnS63WfL1Bibn9Si34OiIcQecbd7fWcPzrj9mZ7I0G6dQJktndD67CkRtRQzVViH HE0FI5Tt0/QWrlqHy0GmehYKqFo1f/HiLi1U36tIIDjb4+Xi4UtVMqMYzkuwEtLm1msD peNTBznHldmUpj0Zt+ZzHg54y4csglUesWp7DDHA5uYTrlcfqhAN6tf0QdrwljXV7KtQ jmlA== X-Forwarded-Encrypted: i=1; AJvYcCVSi0PsIvr8cdH+S560Tz+3Mt1RtO5wnoOmyKXd7SqXxUf0cyPkjJCLp1DsDmKowPxmDK/G4ZXooiTVztPXuFyNtg8oKPY= X-Gm-Message-State: AOJu0YxYsTINuMGRrsQsv1+ODOZhjy4POT0XnpJYmC3zt73MG5ktWmqJ btwBcLKUKZ8OgLri0bcLbXURDiZVNmZVg4hwfhcSF0Colw2eDA20 X-Google-Smtp-Source: AGHT+IHH5M22Oi7Hzooz+5f2zp/yM3nTBUnNRnJIIMCL/sitO5lSHE9HOzYkw6VRXMbJtaUr53DcCA== X-Received: by 2002:a05:6e02:1383:b0:369:9421:89bd with SMTP id d3-20020a056e02138300b00369942189bdmr5790416ilo.29.1711913452082; Sun, 31 Mar 2024 12:30:52 -0700 (PDT) Received: from localhost ([116.121.76.56]) by smtp.gmail.com with ESMTPSA id d9-20020a170902b70900b001db37fd26bcsm7235043pls.116.2024.03.31.12.30.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Mar 2024 12:30:51 -0700 (PDT) From: Minwoo Im To: Klaus Jensen , Keith Busch Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Minwoo Im , Klaus Jensen Subject: [PATCH v2 3/4] hw/nvme: Support SR-IOV VFs more than 127 Date: Mon, 1 Apr 2024 04:30:31 +0900 Message-Id: <20240331193032.5186-4-minwoo.im.dev@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240331193032.5186-1-minwoo.im.dev@gmail.com> References: <20240331193032.5186-1-minwoo.im.dev@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::12f; envelope-from=minwoo.im.dev@gmail.com; helo=mail-il1-x12f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Minwoo Im The number of virtual functions(VFs) supported in SR-IOV is 64k as per spec. To test a large number of MSI-X vectors mapping to CPU matrix in the QEMU system, we need much more than 127 VFs. This patch made support for 256 VFs per a physical function(PF). Signed-off-by: Minwoo Im Reviewed-by: Klaus Jensen --- hw/nvme/ctrl.c | 2 +- hw/nvme/nvme.h | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 7e60bc9f2075..893d4e96656b 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -8424,7 +8424,7 @@ static Property nvme_props[] = { DEFINE_PROP_UINT8("zoned.zasl", NvmeCtrl, params.zasl, 0), DEFINE_PROP_BOOL("zoned.auto_transition", NvmeCtrl, params.auto_transition_zones, true), - DEFINE_PROP_UINT8("sriov_max_vfs", NvmeCtrl, params.sriov_max_vfs, 0), + DEFINE_PROP_UINT16("sriov_max_vfs", NvmeCtrl, params.sriov_max_vfs, 0), DEFINE_PROP_UINT16("sriov_vq_flexible", NvmeCtrl, params.sriov_vq_flexible, 0), DEFINE_PROP_UINT16("sriov_vi_flexible", NvmeCtrl, diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 02c11d909cd1..ad928c28f2c5 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -26,7 +26,7 @@ #define NVME_MAX_CONTROLLERS 256 #define NVME_MAX_NAMESPACES 256 -#define NVME_MAX_VFS 127 +#define NVME_MAX_VFS 256 #define NVME_EUI64_DEFAULT ((uint64_t)0x5254000000000000) #define NVME_FDP_MAX_EVENTS 63 #define NVME_FDP_MAXPIDS 128 @@ -518,7 +518,7 @@ typedef struct NvmeParams { bool auto_transition_zones; bool legacy_cmb; bool ioeventfd; - uint8_t sriov_max_vfs; + uint16_t sriov_max_vfs; uint16_t sriov_vq_flexible; uint16_t sriov_vi_flexible; uint8_t sriov_max_vq_per_vf; From patchwork Sun Mar 31 19:30:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minwoo Im X-Patchwork-Id: 13612257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0B7D2CD128D for ; Sun, 31 Mar 2024 19:31:33 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rr0tP-0000LL-Nr; Sun, 31 Mar 2024 15:30:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rr0tO-0000Ky-Ia; Sun, 31 Mar 2024 15:30:58 -0400 Received: from mail-oa1-x2c.google.com ([2001:4860:4864:20::2c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rr0tM-0001R0-Vt; Sun, 31 Mar 2024 15:30:58 -0400 Received: by mail-oa1-x2c.google.com with SMTP id 586e51a60fabf-22a8df1df8fso2198229fac.2; Sun, 31 Mar 2024 12:30:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711913455; x=1712518255; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wSDYsWlELwKSPvJjZM7xr4TTz+NGl+wsAw++yWtE5cM=; b=lfoi8HWfWLwpVZCjf+VtwFyvgKWiB99MZBDAGtdKskPioE51KBhwUMxGZLiuuzj5q7 GbXI0WHScjTZJ2jFsDf/qv3Whiq2bPqRImUOzAHsTOz3yPuUWDXVscUbCNbtOKQFt+UX Ok3Zvlzsc9Urvx+DcbJ/SAzTP4nvD8y2xbRgeckAsdto0d2ssVOhKtrTEVy3nVKRFgQM LR/x7fu9aI57MkQntd9lbKnLnYXr8SHqnAZun6y8Ad947DhHfMm1zxsIvSDKvx4kTS/w ai78hEYf3NsOcerXvSr0yZmokVO7zJPaDAMYqjKJZlaRWwzEY4meon+YirXa2jlV4Lfs Fqrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711913455; x=1712518255; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wSDYsWlELwKSPvJjZM7xr4TTz+NGl+wsAw++yWtE5cM=; b=rUAwfZPcE1V0AhIf3jzZo9yNuSR9NV36WU3ssWkiC6UPHtt/2J7bfo/huk1KhlkdsU u1ZevSJods7OU5rv8bP1sMzAumG+6EIay4BE9WXtcpcgBLYMLuOq9em5btZWoOnMuzBA Z+mkzbNBjdp4W52W1Ppe5/3/Eno9w8BUs8PzPB2uWdReQ4vqIw0ZrGRQx4kBtCUyuT9/ lFFKk8fCulR3GVo/AnHrIOkVjrmYFcyiRB4ahEQg8v2icKGZWKnH5dVLPbrXYtCx3PbD oLZJgwNm5jcFmVpyvawFqQRrtEym1EmWFWVrQbLru6Y0txYHwdXdYRBdAn6xz3zOp63+ /OPg== X-Forwarded-Encrypted: i=1; AJvYcCWEcCiWtmyTuL571xpDde7gVbyVKFVXSu1X9hXm7JqTan8JnLMn0cfNJO4fYuDxoF9h3I3yMzZzrQ08dsMvPc5xMODA8H4= X-Gm-Message-State: AOJu0YzCN5W5kiEx2V/gaFQynDOyPOZZ+YLcTBxwZTvAd1crNOQBJJg9 xqzL0cADDHJOovwdBYs6QDyb5+1zrrl7uIVjjR+KQHTCVy+x1Mkw X-Google-Smtp-Source: AGHT+IFCIZUy1J7FasRzcv53Mh006SqS7TTWVq1M8gG6jSnZI9IRE2oGlQoyXzPGD6sklAQcCM4w4A== X-Received: by 2002:a05:6870:a548:b0:22a:55bd:a048 with SMTP id p8-20020a056870a54800b0022a55bda048mr8794638oal.5.1711913454994; Sun, 31 Mar 2024 12:30:54 -0700 (PDT) Received: from localhost ([116.121.76.56]) by smtp.gmail.com with ESMTPSA id j5-20020aa783c5000000b006eac81fa1fbsm6452490pfn.66.2024.03.31.12.30.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Mar 2024 12:30:54 -0700 (PDT) From: Minwoo Im To: Klaus Jensen , Keith Busch Cc: qemu-devel@nongnu.org, qemu-block@nongnu.org, Minwoo Im , Klaus Jensen Subject: [PATCH v2 4/4] hw/nvme: Expand VI/VQ resource to uint32 Date: Mon, 1 Apr 2024 04:30:32 +0900 Message-Id: <20240331193032.5186-5-minwoo.im.dev@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240331193032.5186-1-minwoo.im.dev@gmail.com> References: <20240331193032.5186-1-minwoo.im.dev@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2001:4860:4864:20::2c; envelope-from=minwoo.im.dev@gmail.com; helo=mail-oa1-x2c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Minwoo Im VI and VQ resources cover queue resources in each VFs in SR-IOV. Current maximum I/O queue pair size is 0xffff, we can expand them to cover the full number of I/O queue pairs. This patch also fixed Identify Secondary Controller List overflow due to expand of number of secondary controllers. Signed-off-by: Minwoo Im Reviewed-by: Klaus Jensen --- hw/nvme/ctrl.c | 8 ++++---- hw/nvme/nvme.h | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 893d4e96656b..893afae29336 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -8429,10 +8429,10 @@ static Property nvme_props[] = { params.sriov_vq_flexible, 0), DEFINE_PROP_UINT16("sriov_vi_flexible", NvmeCtrl, params.sriov_vi_flexible, 0), - DEFINE_PROP_UINT8("sriov_max_vi_per_vf", NvmeCtrl, - params.sriov_max_vi_per_vf, 0), - DEFINE_PROP_UINT8("sriov_max_vq_per_vf", NvmeCtrl, - params.sriov_max_vq_per_vf, 0), + DEFINE_PROP_UINT32("sriov_max_vi_per_vf", NvmeCtrl, + params.sriov_max_vi_per_vf, 0), + DEFINE_PROP_UINT32("sriov_max_vq_per_vf", NvmeCtrl, + params.sriov_max_vq_per_vf, 0), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index ad928c28f2c5..492617f19515 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -521,8 +521,8 @@ typedef struct NvmeParams { uint16_t sriov_max_vfs; uint16_t sriov_vq_flexible; uint16_t sriov_vi_flexible; - uint8_t sriov_max_vq_per_vf; - uint8_t sriov_max_vi_per_vf; + uint32_t sriov_max_vq_per_vf; + uint32_t sriov_max_vi_per_vf; } NvmeParams; typedef struct NvmeCtrl {