| Message ID | 20190508003851.32416-1-vishal.l.verma@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | daxctl: add a new reconfigure-device command | expand |
Hi Vishal,
I am trying to use your changes, and getting the following error:
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed
Here is full output:
# ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f
[ 26.891054] dax0.0 initialised, 524288 pages in 9ms
[ 26.894239] random: ndctl: uninitialized urandom read (4 bytes read)
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed
{
"dev":"namespace0.0",
"mode":"devdax",
"map":"mem",
"size":"2046.00 MiB (2145.39 MB)",
"uuid":"6684b3b0-4ab1-45ba-9ce6-48aa046b5fc1",
"daxregion":{
"id":0,
"size":"2046.00 MiB (2145.39 MB)",
"align":2097152
},
"align":2097152
}
# [ 73.670556] random: crng init done
Thank you,
Pasha
On Tue, May 7, 2019 at 8:39 PM Vishal Verma <vishal.l.verma@intel.com> wrote:
>
> Changes in v2:
> - Add examples to the documentation page (Dave Hansen)
> - Clarify documentation regarding the conversion from system-ram to devdax
> - Remove any references to a persistent config from the documentation -
> those can be added when the feature is added.
> - device.c: validate option compatibility
> - daxctl-list: display numa_node for device listings
> - daxctl-list: display mode for device listings
> - make the options more consistent by adding a '-O' short option
> for --attempt-offline
>
> Add a new daxctl-reconfigure-device command that lets us reconfigure DAX
> devices back and forth between 'system-ram' and 'device-dax' modes. It
> also includes facilities to online any newly hot-plugged memory
> (default), and attempt to offline memory before converting away from the
> system-ram mode (not default, requires a --attempt-offline option).
>
> Currently missing from this series is a way to persistently store which
> devices have been 'marked' for use as system-ram. This depends on a
> config system overhaul in ndctl, and patches for those will follow
> separately and are independent of this work.
>
> Example invocations:
>
> 1. Reconfigure dax0.0 to system-ram mode, don’t online the memory
> # daxctl reconfigure-device --mode=system-ram --no-online dax0.0
> [
> {
> "chardev":"dax0.0",
> "size":16777216000,
> "numa_node":2,
> "mode":"system-ram"
> }
> ]
>
> 2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory
> # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0
> {
> "chardev":"dax0.0",
> "size":"15.63 GiB (16.78 GB)",
> "numa_node":2,
> "mode":"devdax"
> }
>
> 3. Reconfigure all dax devices on region0 to system-ram mode
> # daxctl reconfigure-device --mode=system-ram --region=0 all
> [
> {
> "chardev":"dax0.0",
> "size":16777216000,
> "numa_node":2,
> "mode":"system-ram"
> },
> {
> "chardev":"dax0.1",
> "size":16777216000,
> "numa_node":3,
> "mode":"system-ram"
> }
> ]
>
> These patches can also be found in the 'kmem-pending' branch on github:
> https://github.com/pmem/ndctl/tree/kmem-pending
>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
>
> Vishal Verma (10):
> libdaxctl: add interfaces in support of device modes
> libdaxctl: cache 'subsystem' in daxctl_ctx
> libdaxctl: add interfaces to enable/disable devices
> libdaxctl: add interfaces to get/set the online state for a node
> daxctl/list: add numa_node for device listings
> libdaxctl: add an interface to get the mode for a dax device
> daxctl: add a new reconfigure-device command
> Documentation/daxctl: add a man page for daxctl-reconfigure-device
> contrib/ndctl: fix region-id completions for daxctl
> contrib/ndctl: add bash-completion for daxctl-reconfigure-device
>
> Documentation/daxctl/Makefile.am | 3 +-
> .../daxctl/daxctl-reconfigure-device.txt | 118 ++++
> contrib/ndctl | 34 +-
> daxctl/Makefile.am | 2 +
> daxctl/builtin.h | 1 +
> daxctl/daxctl.c | 1 +
> daxctl/device.c | 237 ++++++++
> daxctl/lib/Makefile.am | 3 +-
> daxctl/lib/libdaxctl-private.h | 21 +
> daxctl/lib/libdaxctl.c | 554 +++++++++++++++++-
> daxctl/lib/libdaxctl.sym | 14 +
> daxctl/libdaxctl.h | 16 +
> util/json.c | 22 +
> 13 files changed, 1015 insertions(+), 11 deletions(-)
> create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
> create mode 100644 daxctl/device.c
>
> --
> 2.20.1
>
On Wed, 2019-05-08 at 16:02 -0400, Pavel Tatashin wrote: > Hi Vishal, > > I am trying to use your changes, and getting the following error: > libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed > > Here is full output: > # ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f > [ 26.891054] dax0.0 initialised, 524288 pages in 9ms > [ 26.894239] random: ndctl: uninitialized urandom read (4 bytes read) > libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed > { > "dev":"namespace0.0", > "mode":"devdax", > "map":"mem", > "size":"2046.00 MiB (2145.39 MB)", > "uuid":"6684b3b0-4ab1-45ba-9ce6-48aa046b5fc1", > "daxregion":{ > "id":0, > "size":"2046.00 MiB (2145.39 MB)", > "align":2097152 > }, > "align":2097152 > } Thanks for the report! I suspect you may be using the "legacy" device-model. You can confirm by seeing if that device even shows up on the "dax" bus. Compare /sys/bus/dax/devices/ vs /sys/class/dax/ If you see stuff in /sys/class/dax and nothing in /sys/bus/dax/devices then you need to configure the system to switch to the device-dax bus-model by running either of the following commands: # daxctl migrate-device-model ...or: # cat <<EOF > /etc/modprobe.d/daxctl.conf blacklist dax_pmem_compat alias nd:t7* dax_pmem EOF ...and then reload the subsystem: ndctl disable-region all ndctl enable-region all That said, if this was the cause, we should improve the failure mode in this case - I'll look into that, and test by forcing dax-class too. Thanks, -Vishal
On Wed, 2019-05-08 at 20:25 +0000, Verma, Vishal L wrote: > On Wed, 2019-05-08 at 16:02 -0400, Pavel Tatashin wrote: > > Hi Vishal, > > > > I am trying to use your changes, and getting the following error: > > libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed > > > > Here is full output: > > # ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f > > [ 26.891054] dax0.0 initialised, 524288 pages in 9ms > > [ 26.894239] random: ndctl: uninitialized urandom read (4 bytes read) > > libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed > > { > > "dev":"namespace0.0", > > "mode":"devdax", > > "map":"mem", > > "size":"2046.00 MiB (2145.39 MB)", > > "uuid":"6684b3b0-4ab1-45ba-9ce6-48aa046b5fc1", > > "daxregion":{ > > "id":0, > > "size":"2046.00 MiB (2145.39 MB)", > > "align":2097152 > > }, > > "align":2097152 > > } > > Thanks for the report! > > I suspect you may be using the "legacy" device-model. You can > confirm by seeing if that device even shows up on the "dax" bus. > Compare /sys/bus/dax/devices/ vs /sys/class/dax/ Hm, I can't reproduce it forcing the dax-class model. Is this with the qemu config I sent earlier? Also, could you post the output of ndctl list -BiRND Thanks, -Vishal
Changes in v2: - Add examples to the documentation page (Dave Hansen) - Clarify documentation regarding the conversion from system-ram to devdax - Remove any references to a persistent config from the documentation - those can be added when the feature is added. - device.c: validate option compatibility - daxctl-list: display numa_node for device listings - daxctl-list: display mode for device listings - make the options more consistent by adding a '-O' short option for --attempt-offline Add a new daxctl-reconfigure-device command that lets us reconfigure DAX devices back and forth between 'system-ram' and 'device-dax' modes. It also includes facilities to online any newly hot-plugged memory (default), and attempt to offline memory before converting away from the system-ram mode (not default, requires a --attempt-offline option). Currently missing from this series is a way to persistently store which devices have been 'marked' for use as system-ram. This depends on a config system overhaul in ndctl, and patches for those will follow separately and are independent of this work. Example invocations: 1. Reconfigure dax0.0 to system-ram mode, don’t online the memory # daxctl reconfigure-device --mode=system-ram --no-online dax0.0 [ { "chardev":"dax0.0", "size":16777216000, "numa_node":2, "mode":"system-ram" } ] 2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0 { "chardev":"dax0.0", "size":"15.63 GiB (16.78 GB)", "numa_node":2, "mode":"devdax" } 3. Reconfigure all dax devices on region0 to system-ram mode # daxctl reconfigure-device --mode=system-ram --region=0 all [ { "chardev":"dax0.0", "size":16777216000, "numa_node":2, "mode":"system-ram" }, { "chardev":"dax0.1", "size":16777216000, "numa_node":3, "mode":"system-ram" } ] These patches can also be found in the 'kmem-pending' branch on github: https://github.com/pmem/ndctl/tree/kmem-pending Cc: Dan Williams <dan.j.williams@intel.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Vishal Verma (10): libdaxctl: add interfaces in support of device modes libdaxctl: cache 'subsystem' in daxctl_ctx libdaxctl: add interfaces to enable/disable devices libdaxctl: add interfaces to get/set the online state for a node daxctl/list: add numa_node for device listings libdaxctl: add an interface to get the mode for a dax device daxctl: add a new reconfigure-device command Documentation/daxctl: add a man page for daxctl-reconfigure-device contrib/ndctl: fix region-id completions for daxctl contrib/ndctl: add bash-completion for daxctl-reconfigure-device Documentation/daxctl/Makefile.am | 3 +- .../daxctl/daxctl-reconfigure-device.txt | 118 ++++ contrib/ndctl | 34 +- daxctl/Makefile.am | 2 + daxctl/builtin.h | 1 + daxctl/daxctl.c | 1 + daxctl/device.c | 237 ++++++++ daxctl/lib/Makefile.am | 3 +- daxctl/lib/libdaxctl-private.h | 21 + daxctl/lib/libdaxctl.c | 554 +++++++++++++++++- daxctl/lib/libdaxctl.sym | 14 + daxctl/libdaxctl.h | 16 + util/json.c | 22 + 13 files changed, 1015 insertions(+), 11 deletions(-) create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt create mode 100644 daxctl/device.c