| Message ID | 20190724215741.18556-9-vishal.l.verma@intel.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | daxctl: add a new reconfigure-device command | expand |
On Wed, Jul 24, 2019 at 2:57 PM Vishal Verma <vishal.l.verma@intel.com> wrote: > > Add a man page describing the new daxctl-reconfigure-device command. > > Cc: Pavel Tatashin <pasha.tatashin@soleen.com> > Cc: Dave Hansen <dave.hansen@linux.intel.com> > Cc: Dan Williams <dan.j.williams@intel.com> > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > --- > Documentation/daxctl/Makefile.am | 3 +- > .../daxctl/daxctl-reconfigure-device.txt | 139 ++++++++++++++++++ > 2 files changed, 141 insertions(+), 1 deletion(-) > create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt > > diff --git a/Documentation/daxctl/Makefile.am b/Documentation/daxctl/Makefile.am > index 6aba035..715fbad 100644 > --- a/Documentation/daxctl/Makefile.am > +++ b/Documentation/daxctl/Makefile.am > @@ -28,7 +28,8 @@ endif > man1_MANS = \ > daxctl.1 \ > daxctl-list.1 \ > - daxctl-migrate-device-model.1 > + daxctl-migrate-device-model.1 \ > + daxctl-reconfigure-device.1 > > CLEANFILES = $(man1_MANS) > > diff --git a/Documentation/daxctl/daxctl-reconfigure-device.txt b/Documentation/daxctl/daxctl-reconfigure-device.txt > new file mode 100644 > index 0000000..fb2b36b > --- /dev/null > +++ b/Documentation/daxctl/daxctl-reconfigure-device.txt > @@ -0,0 +1,139 @@ > +// SPDX-License-Identifier: GPL-2.0 > + > +daxctl-reconfigure-device(1) > +============================ > + > +NAME > +---- > +daxctl-reconfigure-device - Reconfigure a dax device into a different mode > + > +SYNOPSIS > +-------- > +[verse] > +'daxctl reconfigure-device' <dax0.0> [<dax1.0>...<daxY.Z>] [<options>] > + > +EXAMPLES > +-------- > + > +* Reconfigure dax0.0 to system-ram mode, don't online the memory > +---- > +# daxctl reconfigure-device --mode=system-ram --no-online dax0.0 > +[ > + { > + "chardev":"dax0.0", > + "size":16777216000, > + "target_node":2, > + "mode":"system-ram" > + } > +] > +---- > + > +* Reconfigure dax0.0 to devdax mode, attempt to offline the memory > +---- > +# daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0 > +{ > + "chardev":"dax0.0", > + "size":"15.63 GiB (16.78 GB)", > + "target_node":2, > + "mode":"devdax" > +} > +---- > + > +* Reconfigure all dax devices on region0 to system-ram mode > +---- > +# daxctl reconfigure-device --mode=system-ram --region=0 all > +[ > + { > + "chardev":"dax0.0", > + "size":16777216000, > + "target_node":2, > + "mode":"system-ram" > + }, > + { > + "chardev":"dax0.1", > + "size":16777216000, > + "target_node":3, > + "mode":"system-ram" > + } > +] > +---- > + > +* Run a process called 'some-service' using numactl to restrict its cpu > +nodes to '0' and '1', and memory allocations to node 2 (determined using > +daxctl_dev_get_target_node() or 'daxctl list') > +---- > +# daxctl reconfigure-device --mode=system-ram --no-online dax0.0 Any reason to use --no-online in this example? Presumably some-service may not start if node2 has no online memory. > +[ > + { > + "chardev":"dax0.0", > + "size":16777216000, > + "target_node":2, > + "mode":"system-ram" > + } > +] > + > +# numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2 > +---- > + > +DESCRIPTION > +----------- > + > +Reconfigure the operational mode of a dax device. This can be used to convert > +a regular 'devdax' mode device to the 'system-ram' mode which allows for the dax s/allows/arranges/ > +range to be hot-plugged into the system as regular memory. > + > +NOTE: This is a destructive operation. Any data on the dax device *will* be > +lost. > + > +NOTE: Device reconfiguration depends on the dax-bus device model. If dax-class is > +in use (via the dax_pmem_compat driver), the reconfiguration will fail. See > +linkdaxctl:daxctl-migrate-device-model[1] for more information. Let's make sure that do_reconfig() bails with a common error message for the compat case and quote that message here. You can check that by comparing the device path 'subsystem' to /sys/class/dax. I.e. # ls -l /dev/dax0.0 crw------- 1 root root 253, 4 Jul 25 12:22 /dev/dax0.0 # readlink -f /sys/dev/char/253\:4/subsystem /sys/class/dax ...it will be /sys/bus/dax otherwise. > + > +OPTIONS > +------- > +-r:: > +--region=:: > + Restrict the operation to devices belonging to the specified region(s). > + A device-dax region is a contiguous range of memory that hosts one or > + more /dev/daxX.Y devices, where X is the region id and Y is the device > + instance id. > + > +-m:: > +--mode=:: > + Specify the mode to which the dax device(s) should be reconfigured. > + - "system-ram": hotplug the device into system memory. > + > + - "devdax": switch to the normal "device dax" mode. This requires the > + kernel to support hot-unplugging 'kmem' based memory. If this is not > + available, a reboot is the only way to switch back to 'devdax' mode. > + > +-N:: > +--no-online:: > + By default, memory sections provided by system-ram devices will be > + brought online automatically and immediately with the 'online_movable' > + policy. Use this option to disable the automatic onlining behavior. Probably need to mention that the system might online the memory even if this is specified, or for extra credit, coordinate with that auto-online facility to arrange for it to be skipped. > + > +-O:: > +--attempt-offline:: > + When converting from "system-ram" mode to "devdax", it is expected > + that all the memory sections are first made offline. By default, > + daxctl won't touch online memory. However with this option, attempt > + to offline the memory on the NUMA node associated with the dax device > + before converting it back to "devdax" mode. As mentioned in patch 7, this sounds like --force to me.
On Thu, 2019-07-25 at 19:46 -0700, Dan Williams wrote: > On Wed, Jul 24, 2019 at 2:57 PM Vishal Verma <vishal.l.verma@intel.com> wrote: > > Add a man page describing the new daxctl-reconfigure-device command. > > > > Cc: Pavel Tatashin <pasha.tatashin@soleen.com> > > Cc: Dave Hansen <dave.hansen@linux.intel.com> > > Cc: Dan Williams <dan.j.williams@intel.com> > > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > > --- > > Documentation/daxctl/Makefile.am | 3 +- > > .../daxctl/daxctl-reconfigure-device.txt | 139 ++++++++++++++++++ > > 2 files changed, 141 insertions(+), 1 deletion(-) > > create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt > > [..] > > +* Run a process called 'some-service' using numactl to restrict its cpu > > +nodes to '0' and '1', and memory allocations to node 2 (determined using > > +daxctl_dev_get_target_node() or 'daxctl list') > > +---- > > +# daxctl reconfigure-device --mode=system-ram --no-online dax0.0 > > Any reason to use --no-online in this example? Presumably some-service > may not start if node2 has no online memory. > Yep just a copy/paste typo, removing it. > > > +[ > > + { > > + "chardev":"dax0.0", > > + "size":16777216000, > > + "target_node":2, > > + "mode":"system-ram" > > + } > > +] > > + > > +# numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2 > > +---- > > + > > +DESCRIPTION > > +----------- > > + > > +Reconfigure the operational mode of a dax device. This can be used to convert > > +a regular 'devdax' mode device to the 'system-ram' mode which allows for the dax > > s/allows/arranges/ ok. > > > +range to be hot-plugged into the system as regular memory. > > + > > +NOTE: This is a destructive operation. Any data on the dax device *will* be > > +lost. > > + > > +NOTE: Device reconfiguration depends on the dax-bus device model. If dax-class is > > +in use (via the dax_pmem_compat driver), the reconfiguration will fail. See > > +linkdaxctl:daxctl-migrate-device-model[1] for more information. > > Let's make sure that do_reconfig() bails with a common error message > for the compat case and quote that message here. You can check that by > comparing the device path 'subsystem' to /sys/class/dax. I.e. > > # ls -l /dev/dax0.0 > crw------- 1 root root 253, 4 Jul 25 12:22 /dev/dax0.0 > > # readlink -f /sys/dev/char/253\:4/subsystem > /sys/class/dax > > ...it will be /sys/bus/dax otherwise. Hm,. currently the failure is just: libdaxctl: daxctl_dev_enable: dax3.0: failed to enable Should we check the subsystem as above programatically, and print a better error in daxctl-reconfigure-device? > > > + > > +OPTIONS > > +------- > > +-r:: > > +--region=:: > > + Restrict the operation to devices belonging to the specified region(s). > > + A device-dax region is a contiguous range of memory that hosts one or > > + more /dev/daxX.Y devices, where X is the region id and Y is the device > > + instance id. > > + > > +-m:: > > +--mode=:: > > + Specify the mode to which the dax device(s) should be reconfigured. > > + - "system-ram": hotplug the device into system memory. > > + > > + - "devdax": switch to the normal "device dax" mode. This requires the > > + kernel to support hot-unplugging 'kmem' based memory. If this is not > > + available, a reboot is the only way to switch back to 'devdax' mode. > > + > > +-N:: > > +--no-online:: > > + By default, memory sections provided by system-ram devices will be > > + brought online automatically and immediately with the 'online_movable' > > + policy. Use this option to disable the automatic onlining behavior. > > Probably need to mention that the system might online the memory even > if this is specified, or for extra credit, coordinate with that > auto-online facility to arrange for it to be skipped. Good point, added a note regarding this. > > > + > > +-O:: > > +--attempt-offline:: > > + When converting from "system-ram" mode to "devdax", it is expected > > + that all the memory sections are first made offline. By default, > > + daxctl won't touch online memory. However with this option, attempt > > + to offline the memory on the NUMA node associated with the dax device > > + before converting it back to "devdax" mode. > > As mentioned in patch 7, this sounds like --force to me. Done.
diff --git a/Documentation/daxctl/Makefile.am b/Documentation/daxctl/Makefile.am index 6aba035..715fbad 100644 --- a/Documentation/daxctl/Makefile.am +++ b/Documentation/daxctl/Makefile.am @@ -28,7 +28,8 @@ endif man1_MANS = \ daxctl.1 \ daxctl-list.1 \ - daxctl-migrate-device-model.1 + daxctl-migrate-device-model.1 \ + daxctl-reconfigure-device.1 CLEANFILES = $(man1_MANS) diff --git a/Documentation/daxctl/daxctl-reconfigure-device.txt b/Documentation/daxctl/daxctl-reconfigure-device.txt new file mode 100644 index 0000000..fb2b36b --- /dev/null +++ b/Documentation/daxctl/daxctl-reconfigure-device.txt @@ -0,0 +1,139 @@ +// SPDX-License-Identifier: GPL-2.0 + +daxctl-reconfigure-device(1) +============================ + +NAME +---- +daxctl-reconfigure-device - Reconfigure a dax device into a different mode + +SYNOPSIS +-------- +[verse] +'daxctl reconfigure-device' <dax0.0> [<dax1.0>...<daxY.Z>] [<options>] + +EXAMPLES +-------- + +* Reconfigure dax0.0 to system-ram mode, don't online the memory +---- +# daxctl reconfigure-device --mode=system-ram --no-online dax0.0 +[ + { + "chardev":"dax0.0", + "size":16777216000, + "target_node":2, + "mode":"system-ram" + } +] +---- + +* Reconfigure dax0.0 to devdax mode, attempt to offline the memory +---- +# daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0 +{ + "chardev":"dax0.0", + "size":"15.63 GiB (16.78 GB)", + "target_node":2, + "mode":"devdax" +} +---- + +* Reconfigure all dax devices on region0 to system-ram mode +---- +# daxctl reconfigure-device --mode=system-ram --region=0 all +[ + { + "chardev":"dax0.0", + "size":16777216000, + "target_node":2, + "mode":"system-ram" + }, + { + "chardev":"dax0.1", + "size":16777216000, + "target_node":3, + "mode":"system-ram" + } +] +---- + +* Run a process called 'some-service' using numactl to restrict its cpu +nodes to '0' and '1', and memory allocations to node 2 (determined using +daxctl_dev_get_target_node() or 'daxctl list') +---- +# daxctl reconfigure-device --mode=system-ram --no-online dax0.0 +[ + { + "chardev":"dax0.0", + "size":16777216000, + "target_node":2, + "mode":"system-ram" + } +] + +# numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2 +---- + +DESCRIPTION +----------- + +Reconfigure the operational mode of a dax device. This can be used to convert +a regular 'devdax' mode device to the 'system-ram' mode which allows for the dax +range to be hot-plugged into the system as regular memory. + +NOTE: This is a destructive operation. Any data on the dax device *will* be +lost. + +NOTE: Device reconfiguration depends on the dax-bus device model. If dax-class is +in use (via the dax_pmem_compat driver), the reconfiguration will fail. See +linkdaxctl:daxctl-migrate-device-model[1] for more information. + +OPTIONS +------- +-r:: +--region=:: + Restrict the operation to devices belonging to the specified region(s). + A device-dax region is a contiguous range of memory that hosts one or + more /dev/daxX.Y devices, where X is the region id and Y is the device + instance id. + +-m:: +--mode=:: + Specify the mode to which the dax device(s) should be reconfigured. + - "system-ram": hotplug the device into system memory. + + - "devdax": switch to the normal "device dax" mode. This requires the + kernel to support hot-unplugging 'kmem' based memory. If this is not + available, a reboot is the only way to switch back to 'devdax' mode. + +-N:: +--no-online:: + By default, memory sections provided by system-ram devices will be + brought online automatically and immediately with the 'online_movable' + policy. Use this option to disable the automatic onlining behavior. + +-O:: +--attempt-offline:: + When converting from "system-ram" mode to "devdax", it is expected + that all the memory sections are first made offline. By default, + daxctl won't touch online memory. However with this option, attempt + to offline the memory on the NUMA node associated with the dax device + before converting it back to "devdax" mode. + +-u:: +--human:: + By default the command will output machine-friendly raw-integer + data. Instead, with this flag, numbers representing storage size + will be formatted as human readable strings with units, other + fields are converted to hexadecimal strings. + +-v:: +--verbose:: + Emit more debug messages + +include::../copyright.txt[] + +SEE ALSO +-------- +linkdaxctl:daxctl-list[1],daxctl-migrate-device-model[1]
Add a man page describing the new daxctl-reconfigure-device command. Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Dan Williams <dan.j.williams@intel.com> Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> --- Documentation/daxctl/Makefile.am | 3 +- .../daxctl/daxctl-reconfigure-device.txt | 139 ++++++++++++++++++ 2 files changed, 141 insertions(+), 1 deletion(-) create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt