mbox series

[RFC,V3,0/8] hw/block/nvme: support multi-path for ctrl/ns

Message ID 20210119170147.19657-1-minwoo.im.dev@gmail.com (mailing list archive)
Headers show
Series hw/block/nvme: support multi-path for ctrl/ns | expand

Message

Minwoo Im Jan. 19, 2021, 5:01 p.m. UTC
Hello,

This patch series is third one to support multi-controller and namespace
sharing in multi-path.  This series introduced subsystem scheme to
manage controller(s) and namespace(s) in the subsystem.

This series has new patches from the V2:  'detached' parameter has been
added to the nvme-ns device.  This will decide whether to attach the
namespace to controller(s) in the current subsystem or not.  If it's
given with true, then it will be just allocated in the subsystem, but
not attaching to any controllers in the subsystem.  Otherwise, it will
automatically attach to all the controllers in the subsystem.  The other
t hing is that the last patch implemented Identify Active Namespace ID
List command handler apart from the Allocated Namespace ID List.

Run with:
  -device nvme,serial=qux,id=nvme3
  -device nvme-ns,id=ns3,drive=drv12,nsid=3,bus=nvme3

  -device nvme-subsys,id=subsys0
  -device nvme,serial=foo,id=nvme0,subsys=subsys0
  -device nvme,serial=bar,id=nvme1,subsys=subsys0
  -device nvme,serial=baz,id=nvme2,subsys=subsys0
  -device nvme-ns,id=ns1,drive=drv10,nsid=1,subsys=subsys0,detached=true
  -device nvme-ns,id=ns2,drive=drv11,nsid=2,bus=nvme2

nvme-cli:
  root@vm:~/work# nvme list -v                                                                                                      
  NVM Express Subsystems                                                                                                 
                                                                                                                                     
  Subsystem        Subsystem-NQN                                                                                    Controllers
  ---------------- ------------------------------------------------------------------------------------------------ ----------------
  nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
  nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
                                                                                                                                   
  NVM Express Controllers                                                                                           
                                                                                                                  
  Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
  -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
  nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0
  nvme1    foo                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:07.0   nvme-subsys1
  nvme2    bar                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:08.0   nvme-subsys1
  nvme3    baz                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:09.0   nvme-subsys1 nvme1n1
                                                                                  
  NVM Express Namespaces                                                 
                                                                       
  Device       NSID     Usage                      Format           Controllers
  ------------ -------- -------------------------- ---------------- ----------------
  nvme0n1      3        268.44  MB / 268.44  MB    512   B +  0 B   nvme0
  nvme1n1      2        268.44  MB / 268.44  MB    512   B +  0 B   nvme3

Now we have [allocated|not-allocated]/[attached/detached] scheme for
namespaces and controllers.  The next series is going to be to support
namespace management and attachment with few Identify command handlers.

Please review.

Thanks!

Since RFC V2:
  - Rebased on nvme-next branch with trivial patches from the previous
    version(V2) applied. (Klaus)
  - Fix enumeration type name convention with NvmeIdNs prefix. (Klaus)
  - Put 'cntlid' to NvmeCtrl instance in nvme_init_ctrl() which was
    missed in V2.
  - Added 'detached' parameter to nvme-ns device to decide whether to
    attach or not to controller(s) in the subsystem. (Klaus)
  - Implemented Identify Active Namespace ID List aprt from Identify
    Allocated Namespace ID List by removing fall-thru statement.

Since RFC V1:
  - Updated namespace sharing scheme to be based on nvme-subsys
    hierarchy.

Minwoo Im (8):
  hw/block/nvme: introduce nvme-subsys device
  hw/block/nvme: support to map controller to a subsystem
  hw/block/nvme: add CMIC enum value for Identify Controller
  hw/block/nvme: support for multi-controller in subsystem
  hw/block/nvme: add NMIC enum value for Identify Namespace
  hw/block/nvme: support for shared namespace in subsystem
  hw/block/nvme: add 'detached' param not to attach namespace
  hw/block/nvme: Add Identify Active Namespace ID List

 hw/block/meson.build   |   2 +-
 hw/block/nvme-ns.c     |  32 ++++++++--
 hw/block/nvme-ns.h     |  13 ++++
 hw/block/nvme-subsys.c | 111 +++++++++++++++++++++++++++++++++
 hw/block/nvme-subsys.h |  32 ++++++++++
 hw/block/nvme.c        | 137 ++++++++++++++++++++++++++++++++++++++---
 hw/block/nvme.h        |  19 ++++++
 include/block/nvme.h   |   8 +++
 8 files changed, 340 insertions(+), 14 deletions(-)
 create mode 100644 hw/block/nvme-subsys.c
 create mode 100644 hw/block/nvme-subsys.h

Comments

Klaus Jensen Jan. 19, 2021, 6:18 p.m. UTC | #1
On Jan 20 02:01, Minwoo Im wrote:
> Hello,
> 
> This patch series is third one to support multi-controller and namespace
> sharing in multi-path.  This series introduced subsystem scheme to
> manage controller(s) and namespace(s) in the subsystem.
> 
> This series has new patches from the V2:  'detached' parameter has been
> added to the nvme-ns device.  This will decide whether to attach the
> namespace to controller(s) in the current subsystem or not.  If it's
> given with true, then it will be just allocated in the subsystem, but
> not attaching to any controllers in the subsystem.  Otherwise, it will
> automatically attach to all the controllers in the subsystem.  The other
> t hing is that the last patch implemented Identify Active Namespace ID
> List command handler apart from the Allocated Namespace ID List.
> 
> Run with:
>   -device nvme,serial=qux,id=nvme3
>   -device nvme-ns,id=ns3,drive=drv12,nsid=3,bus=nvme3
> 
>   -device nvme-subsys,id=subsys0
>   -device nvme,serial=foo,id=nvme0,subsys=subsys0
>   -device nvme,serial=bar,id=nvme1,subsys=subsys0
>   -device nvme,serial=baz,id=nvme2,subsys=subsys0
>   -device nvme-ns,id=ns1,drive=drv10,nsid=1,subsys=subsys0,detached=true
>   -device nvme-ns,id=ns2,drive=drv11,nsid=2,bus=nvme2
> 
> nvme-cli:
>   root@vm:~/work# nvme list -v                                                                                                      
>   NVM Express Subsystems                                                                                                 
>                                                                                                                                      
>   Subsystem        Subsystem-NQN                                                                                    Controllers
>   ---------------- ------------------------------------------------------------------------------------------------ ----------------
>   nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
>   nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
>                                                                                                                                    
>   NVM Express Controllers                                                                                           
>                                                                                                                   
>   Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
>   -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
>   nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0

Shouldn't nvme0n1 be listed under Namespaces for nvme0?

>   nvme1    foo                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:07.0   nvme-subsys1
>   nvme2    bar                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:08.0   nvme-subsys1
>   nvme3    baz                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:09.0   nvme-subsys1 nvme1n1
>                                                                                   
>   NVM Express Namespaces                                                 
>                                                                        
>   Device       NSID     Usage                      Format           Controllers
>   ------------ -------- -------------------------- ---------------- ----------------
>   nvme0n1      3        268.44  MB / 268.44  MB    512   B +  0 B   nvme0
>   nvme1n1      2        268.44  MB / 268.44  MB    512   B +  0 B   nvme3
> 
> Now we have [allocated|not-allocated]/[attached/detached] scheme for
> namespaces and controllers.  The next series is going to be to support
> namespace management and attachment with few Identify command handlers.
> 
> Please review.
> 
> Thanks!
> 
> Since RFC V2:
>   - Rebased on nvme-next branch with trivial patches from the previous
>     version(V2) applied. (Klaus)
>   - Fix enumeration type name convention with NvmeIdNs prefix. (Klaus)
>   - Put 'cntlid' to NvmeCtrl instance in nvme_init_ctrl() which was
>     missed in V2.
>   - Added 'detached' parameter to nvme-ns device to decide whether to
>     attach or not to controller(s) in the subsystem. (Klaus)
>   - Implemented Identify Active Namespace ID List aprt from Identify
>     Allocated Namespace ID List by removing fall-thru statement.
> 
> Since RFC V1:
>   - Updated namespace sharing scheme to be based on nvme-subsys
>     hierarchy.
> 
> Minwoo Im (8):
>   hw/block/nvme: introduce nvme-subsys device
>   hw/block/nvme: support to map controller to a subsystem
>   hw/block/nvme: add CMIC enum value for Identify Controller
>   hw/block/nvme: support for multi-controller in subsystem
>   hw/block/nvme: add NMIC enum value for Identify Namespace
>   hw/block/nvme: support for shared namespace in subsystem
>   hw/block/nvme: add 'detached' param not to attach namespace
>   hw/block/nvme: Add Identify Active Namespace ID List
> 
>  hw/block/meson.build   |   2 +-
>  hw/block/nvme-ns.c     |  32 ++++++++--
>  hw/block/nvme-ns.h     |  13 ++++
>  hw/block/nvme-subsys.c | 111 +++++++++++++++++++++++++++++++++
>  hw/block/nvme-subsys.h |  32 ++++++++++
>  hw/block/nvme.c        | 137 ++++++++++++++++++++++++++++++++++++++---
>  hw/block/nvme.h        |  19 ++++++
>  include/block/nvme.h   |   8 +++
>  8 files changed, 340 insertions(+), 14 deletions(-)
>  create mode 100644 hw/block/nvme-subsys.c
>  create mode 100644 hw/block/nvme-subsys.h
> 
> -- 
> 2.17.1
> 
>
Klaus Jensen Jan. 19, 2021, 6:25 p.m. UTC | #2
On Jan 20 02:01, Minwoo Im wrote:
> Introduced 'detached' parameter to nvme-ns device.  If given, the
> namespace will not be attached to controller(s) in the subsystem.  If
> 'subsys' is not given with this option, it should be provided with 'bus'
> which is for private namespace.
> 
> This patch also introduced 'ctrls_bitmap' in NvmeNamespace instance to
> represent which controler id(cntlid) is attached to this namespace
> device.  A namespace can be attached to multiple controllers in a
> subsystem so that this bitmap maps those two relationships.
> 
> The ctrls_bitmap bitmap should not be accessed directly, but through the
> helpers introduced in this patch: nvme_ns_is_attached(),
> nvme_ns_attach(), nvme_ns_detach().
> 
> Note that this patch made identify namespace list data not hold
> non-attached namespace ID in nvme_identify_nslist.  Currently, this
> command handler is for CNS 0x2(Active) and 0x10(Allocated) both.  The
> next patch will introduce a handler for later on.
> 
> Signed-off-by: Minwoo Im <minwoo.im.dev@gmail.com>
> ---
>  hw/block/nvme-ns.c     |  9 +++++++++
>  hw/block/nvme-ns.h     |  6 ++++++
>  hw/block/nvme-subsys.c |  2 ++
>  hw/block/nvme.c        | 31 ++++++++++++++++++++++++++++++-
>  hw/block/nvme.h        | 15 +++++++++++++++
>  5 files changed, 62 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/block/nvme-ns.c b/hw/block/nvme-ns.c
> index 073f65e49cac..70d42c24065c 100644
> --- a/hw/block/nvme-ns.c
> +++ b/hw/block/nvme-ns.c
> @@ -17,6 +17,7 @@
>  #include "qemu/cutils.h"
>  #include "qemu/log.h"
>  #include "qemu/error-report.h"
> +#include "qemu/hbitmap.h"

Isn't the HBitmap slightly overkill? Can qemu/bitmap.h suffice?

>  #include "hw/block/block.h"
>  #include "hw/pci/pci.h"
>  #include "sysemu/sysemu.h"
Keith Busch Jan. 19, 2021, 7:26 p.m. UTC | #3
On Tue, Jan 19, 2021 at 07:18:16PM +0100, Klaus Jensen wrote:
> On Jan 20 02:01, Minwoo Im wrote:
> > Run with:
> >   -device nvme,serial=qux,id=nvme3
> >   -device nvme-ns,id=ns3,drive=drv12,nsid=3,bus=nvme3
> > 
> >   -device nvme-subsys,id=subsys0
> >   -device nvme,serial=foo,id=nvme0,subsys=subsys0
> >   -device nvme,serial=bar,id=nvme1,subsys=subsys0
> >   -device nvme,serial=baz,id=nvme2,subsys=subsys0
> >   -device nvme-ns,id=ns1,drive=drv10,nsid=1,subsys=subsys0,detached=true
> >   -device nvme-ns,id=ns2,drive=drv11,nsid=2,bus=nvme2
> > 
> > nvme-cli:
> >   root@vm:~/work# nvme list -v                                                                                                      
> >   NVM Express Subsystems                                                                                                 
> >                                                                                                                                      
> >   Subsystem        Subsystem-NQN                                                                                    Controllers
> >   ---------------- ------------------------------------------------------------------------------------------------ ----------------
> >   nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
> >   nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
> >                                                                                                                                    
> >   NVM Express Controllers                                                                                           
> >                                                                                                                   
> >   Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
> >   -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
> >   nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0
> 
> Shouldn't nvme0n1 be listed under Namespaces for nvme0?

Minwoo, try pulling the most current nvme-cli. There was a sysfs
scanning bug for non-mpath drives that should be fixed now.
Minwoo Im Jan. 20, 2021, 12:44 a.m. UTC | #4
On 21-01-19 19:18:16, Klaus Jensen wrote:
> On Jan 20 02:01, Minwoo Im wrote:
> > Hello,
> > 
> > This patch series is third one to support multi-controller and namespace
> > sharing in multi-path.  This series introduced subsystem scheme to
> > manage controller(s) and namespace(s) in the subsystem.
> > 
> > This series has new patches from the V2:  'detached' parameter has been
> > added to the nvme-ns device.  This will decide whether to attach the
> > namespace to controller(s) in the current subsystem or not.  If it's
> > given with true, then it will be just allocated in the subsystem, but
> > not attaching to any controllers in the subsystem.  Otherwise, it will
> > automatically attach to all the controllers in the subsystem.  The other
> > t hing is that the last patch implemented Identify Active Namespace ID
> > List command handler apart from the Allocated Namespace ID List.
> > 
> > Run with:
> >   -device nvme,serial=qux,id=nvme3
> >   -device nvme-ns,id=ns3,drive=drv12,nsid=3,bus=nvme3
> > 
> >   -device nvme-subsys,id=subsys0
> >   -device nvme,serial=foo,id=nvme0,subsys=subsys0
> >   -device nvme,serial=bar,id=nvme1,subsys=subsys0
> >   -device nvme,serial=baz,id=nvme2,subsys=subsys0
> >   -device nvme-ns,id=ns1,drive=drv10,nsid=1,subsys=subsys0,detached=true
> >   -device nvme-ns,id=ns2,drive=drv11,nsid=2,bus=nvme2
> > 
> > nvme-cli:
> >   root@vm:~/work# nvme list -v                                                                                                      
> >   NVM Express Subsystems                                                                                                 
> >                                                                                                                                      
> >   Subsystem        Subsystem-NQN                                                                                    Controllers
> >   ---------------- ------------------------------------------------------------------------------------------------ ----------------
> >   nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
> >   nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
> >                                                                                                                                    
> >   NVM Express Controllers                                                                                           
> >                                                                                                                   
> >   Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
> >   -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
> >   nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0
> 
> Shouldn't nvme0n1 be listed under Namespaces for nvme0?

Oh, I missed that one from the output.  As Keith mentioned, I ran the
list command again based on the latest nvme-cli.git:

Please refer the following result.  I think it's okay not to send the
cover letter again :)

# nvme --version
nvme version 1.13.48.g33c6

# nvme list -v
NVM Express Subsystems

Subsystem        Subsystem-NQN                                                                                    Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3

NVM Express Controllers

Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces      
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0 nvme0n1
nvme1    foo                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:07.0   nvme-subsys1 
nvme2    bar                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:08.0   nvme-subsys1 
nvme3    baz                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:09.0   nvme-subsys1 nvme1c3n1

NVM Express Namespaces

Device       NSID     Usage                      Format           Controllers     
------------ -------- -------------------------- ---------------- ----------------
nvme0n1      3        268.44  MB / 268.44  MB    512   B +  0 B   nvme0
nvme1n1      2        268.44  MB / 268.44  MB    512   B +  0 B   nvme3
Minwoo Im Jan. 20, 2021, 12:45 a.m. UTC | #5
> Minwoo, try pulling the most current nvme-cli. There was a sysfs
> scanning bug for non-mpath drives that should be fixed now.

Thank you, Keith!  I've posted list result based on the latest one :)
Minwoo Im Jan. 20, 2021, 12:47 a.m. UTC | #6
> Isn't the HBitmap slightly overkill? Can qemu/bitmap.h suffice?

Definitely, yes, I think.  Current patch series supoprt up to 32
controllers so I think qemu/bitmap.h is enough for us.

Will update the bitmap operations in the next series.
Klaus Jensen Jan. 20, 2021, 7:52 a.m. UTC | #7
On Jan 20 09:44, Minwoo Im wrote:
> On 21-01-19 19:18:16, Klaus Jensen wrote:
> > On Jan 20 02:01, Minwoo Im wrote:
> > > Hello,
> > > 
> > > This patch series is third one to support multi-controller and namespace
> > > sharing in multi-path.  This series introduced subsystem scheme to
> > > manage controller(s) and namespace(s) in the subsystem.
> > > 
> > > This series has new patches from the V2:  'detached' parameter has been
> > > added to the nvme-ns device.  This will decide whether to attach the
> > > namespace to controller(s) in the current subsystem or not.  If it's
> > > given with true, then it will be just allocated in the subsystem, but
> > > not attaching to any controllers in the subsystem.  Otherwise, it will
> > > automatically attach to all the controllers in the subsystem.  The other
> > > t hing is that the last patch implemented Identify Active Namespace ID
> > > List command handler apart from the Allocated Namespace ID List.
> > > 
> > > Run with:
> > >   -device nvme,serial=qux,id=nvme3
> > >   -device nvme-ns,id=ns3,drive=drv12,nsid=3,bus=nvme3
> > > 
> > >   -device nvme-subsys,id=subsys0
> > >   -device nvme,serial=foo,id=nvme0,subsys=subsys0
> > >   -device nvme,serial=bar,id=nvme1,subsys=subsys0
> > >   -device nvme,serial=baz,id=nvme2,subsys=subsys0
> > >   -device nvme-ns,id=ns1,drive=drv10,nsid=1,subsys=subsys0,detached=true
> > >   -device nvme-ns,id=ns2,drive=drv11,nsid=2,bus=nvme2
> > > 
> > > nvme-cli:
> > >   root@vm:~/work# nvme list -v                                                                                                      
> > >   NVM Express Subsystems                                                                                                 
> > >                                                                                                                                      
> > >   Subsystem        Subsystem-NQN                                                                                    Controllers
> > >   ---------------- ------------------------------------------------------------------------------------------------ ----------------
> > >   nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
> > >   nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
> > >                                                                                                                                    
> > >   NVM Express Controllers                                                                                           
> > >                                                                                                                   
> > >   Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
> > >   -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
> > >   nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0
> > 
> > Shouldn't nvme0n1 be listed under Namespaces for nvme0?
> 
> Oh, I missed that one from the output.  As Keith mentioned, I ran the
> list command again based on the latest nvme-cli.git:
> 
> Please refer the following result.  I think it's okay not to send the
> cover letter again :)
> 
> # nvme --version
> nvme version 1.13.48.g33c6
> 
> # nvme list -v
> NVM Express Subsystems
> 
> Subsystem        Subsystem-NQN                                                                                    Controllers
> ---------------- ------------------------------------------------------------------------------------------------ ----------------
> nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
> nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
> 
> NVM Express Controllers
> 
> Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces      
> -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
> nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0 nvme0n1
> nvme1    foo                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:07.0   nvme-subsys1 
> nvme2    bar                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:08.0   nvme-subsys1 
> nvme3    baz                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:09.0   nvme-subsys1 nvme1c3n1
> 
> NVM Express Namespaces
> 
> Device       NSID     Usage                      Format           Controllers     
> ------------ -------- -------------------------- ---------------- ----------------
> nvme0n1      3        268.44  MB / 268.44  MB    512   B +  0 B   nvme0
> nvme1n1      2        268.44  MB / 268.44  MB    512   B +  0 B   nvme3

That looks better, but hmm. Shouldnt the namespace be named `nvme1c3n1`
here has well? Is that also an issue with nvme-cli?
Minwoo Im Jan. 20, 2021, 11:46 a.m. UTC | #8
On 21-01-20 08:52:14, Klaus Jensen wrote:
> On Jan 20 09:44, Minwoo Im wrote:
> > On 21-01-19 19:18:16, Klaus Jensen wrote:
> > > On Jan 20 02:01, Minwoo Im wrote:
> > > > Hello,
> > > > 
> > > > This patch series is third one to support multi-controller and namespace
> > > > sharing in multi-path.  This series introduced subsystem scheme to
> > > > manage controller(s) and namespace(s) in the subsystem.
> > > > 
> > > > This series has new patches from the V2:  'detached' parameter has been
> > > > added to the nvme-ns device.  This will decide whether to attach the
> > > > namespace to controller(s) in the current subsystem or not.  If it's
> > > > given with true, then it will be just allocated in the subsystem, but
> > > > not attaching to any controllers in the subsystem.  Otherwise, it will
> > > > automatically attach to all the controllers in the subsystem.  The other
> > > > t hing is that the last patch implemented Identify Active Namespace ID
> > > > List command handler apart from the Allocated Namespace ID List.
> > > > 
> > > > Run with:
> > > >   -device nvme,serial=qux,id=nvme3
> > > >   -device nvme-ns,id=ns3,drive=drv12,nsid=3,bus=nvme3
> > > > 
> > > >   -device nvme-subsys,id=subsys0
> > > >   -device nvme,serial=foo,id=nvme0,subsys=subsys0
> > > >   -device nvme,serial=bar,id=nvme1,subsys=subsys0
> > > >   -device nvme,serial=baz,id=nvme2,subsys=subsys0
> > > >   -device nvme-ns,id=ns1,drive=drv10,nsid=1,subsys=subsys0,detached=true
> > > >   -device nvme-ns,id=ns2,drive=drv11,nsid=2,bus=nvme2
> > > > 
> > > > nvme-cli:
> > > >   root@vm:~/work# nvme list -v                                                                                                      
> > > >   NVM Express Subsystems                                                                                                 
> > > >                                                                                                                                      
> > > >   Subsystem        Subsystem-NQN                                                                                    Controllers
> > > >   ---------------- ------------------------------------------------------------------------------------------------ ----------------
> > > >   nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
> > > >   nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
> > > >                                                                                                                                    
> > > >   NVM Express Controllers                                                                                           
> > > >                                                                                                                   
> > > >   Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
> > > >   -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
> > > >   nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0
> > > 
> > > Shouldn't nvme0n1 be listed under Namespaces for nvme0?
> > 
> > Oh, I missed that one from the output.  As Keith mentioned, I ran the
> > list command again based on the latest nvme-cli.git:
> > 
> > Please refer the following result.  I think it's okay not to send the
> > cover letter again :)
> > 
> > # nvme --version
> > nvme version 1.13.48.g33c6
> > 
> > # nvme list -v
> > NVM Express Subsystems
> > 
> > Subsystem        Subsystem-NQN                                                                                    Controllers
> > ---------------- ------------------------------------------------------------------------------------------------ ----------------
> > nvme-subsys0     nqn.2019-08.org.qemu:qux                                                                         nvme0
> > nvme-subsys1     nqn.2019-08.org.qemu:subsys0                                                                     nvme1, nvme2, nvme3
> > 
> > NVM Express Controllers
> > 
> > Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces      
> > -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
> > nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0 nvme0n1
> > nvme1    foo                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:07.0   nvme-subsys1 
> > nvme2    bar                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:08.0   nvme-subsys1 
> > nvme3    baz                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:09.0   nvme-subsys1 nvme1c3n1
> > 
> > NVM Express Namespaces
> > 
> > Device       NSID     Usage                      Format           Controllers     
> > ------------ -------- -------------------------- ---------------- ----------------
> > nvme0n1      3        268.44  MB / 268.44  MB    512   B +  0 B   nvme0
> > nvme1n1      2        268.44  MB / 268.44  MB    512   B +  0 B   nvme3
> 
> That looks better, but hmm. Shouldnt the namespace be named `nvme1c3n1`
> here has well? Is that also an issue with nvme-cli?

No it isn't in this series.  The 'nvme3' controller has not been
provided with 'subsys' parameter which means it does not support CMIC
multi-controller as nvme_init_ctrl() does not set the flag for
NVME_CMIC_MULTI_CTRL:

  -device nvme,serial=qux,id=nvme3

If this has been given with 'subsys' which means it supports for
multi-controller, then it will be given like:

  Device   SN                   MN                                       FR       TxPort Address        Subsystem    Namespaces
  -------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
  nvme0    qux                  QEMU NVMe Ctrl                           1.0      pcie   0000:00:06.0   nvme-subsys0 nvme0c0n1

In short, if 'nvme' device controller is not given with 'subsys', then
it will not support multi-controller and it will never have
GENHD_FL_HIDDEN (nvmeXcYnZ convention).