| Message ID | 20190516224053.3655-1-vishal.l.verma@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | daxctl: add a new reconfigure-device command | expand |
On Thu, May 16, 2019 at 6:40 PM Vishal Verma <vishal.l.verma@intel.com> wrote: > > Changes in v3: > - In daxctl_dev_get_mode(), remove the subsystem warning, detect dax-class > and simply make it return devdax Hi Vishal, I am still getting the same error as before: # ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f [ 141.525873] dax0.0 initialised, 524288 pages in 7ms libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed ... I am building it via buildroot, and can share the initramfs file or anything that can help you with fixing this issue. Thank you, Pavel
On Fri, 2019-05-17 at 11:30 -0400, Pavel Tatashin wrote: > On Thu, May 16, 2019 at 6:40 PM Vishal Verma <vishal.l.verma@intel.com > > wrote: > > Changes in v3: > > - In daxctl_dev_get_mode(), remove the subsystem warning, detect > > dax-class > > and simply make it return devdax > > Hi Vishal, > > I am still getting the same error as before: > > # ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f > [ 141.525873] dax0.0 initialised, 524288 pages in 7ms > libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed > ... > > I am building it via buildroot, and can share the initramfs file or > anything that can help you with fixing this issue. Hi Pavel, I've still not been able to hit this in my testing, is it something you hit only after applying these patches? i.e. does plain v65 work?
> Hi Pavel, > > I've still not been able to hit this in my testing, is it something you > hit only after applying these patches? i.e. does plain v65 work? Yes, plain v65 works, but with these patches I see this error. I use buildroot to build initramfs with ndctl. Here is how ndctl.mk looks like: NDCTL_VERSION = ed17fd1 NDCTL_SITE = $(call github,pmem,ndctl,$(NDCTL_VERSION)) NDCTL_LICENSE = GNU Lesser General Public License v2.1 NDCTL_LICENSE_FILES = COPYING #NDCTL_AUTORECv65ONF = YES define NDCTL_RUN_AUTOGEN cd $(@D) && PATH=$(BR_PATH) ./autogen.sh endef NDCTL_PRE_CONFIGURE_HOOKS += NDCTL_RUN_AUTOGEN NDCTL_LDFLAGS = $(TARGET_LDFLAGS) NDCTL_CONF_OPTS = \ --without-bash \ --without-keyutils \ --without-systemd $(eval $(autotools-package)) This version works, but when I change: NDCTL_VERSION = 9ee82c8 I get add_dev error. Pasha
On Fri, 2019-05-17 at 13:41 -0400, Pavel Tatashin wrote: > > Hi Pavel, > > > > I've still not been able to hit this in my testing, is it something > > you > > hit only after applying these patches? i.e. does plain v65 work? > > Yes, plain v65 works, but with these patches I see this error. > Hm, So there are only two patches that touch the add_dax_dev function: 2bf9a8e libdaxctl: add interfaces to enable/disable devices 25be8f4 libdaxctl: add interfaces to get/set the online state for a node And of these, the second one to get the target node doesn't return an error in any case. The first one can fail, so it must be that condition you're hitting, but I'm not sure in what way it is failing. The exact check is: sprintf(path, "%s/modalias", daxdev_base); rc = sysfs_read_attr(ctx, path, buf); /* older kernels may be lack the modalias attribute */ if (rc < 0 && rc != -ENOENT) goto err_read; if (rc == 0) { dev->kmod_list = to_module_list(ctx, buf); if (dev->kmod_list == NULL) goto err_read; } else dbg(ctx, "%s: modalias attribute missing\n", devname); Do you have the file: /sys/bus/dax/devices/dax0.0/modalias And what does it contain? On my system: $ cat /sys/bus/dax/devices/dax0.0/modalias dax:t0 If this is missing, it would seem you're not correctly in 'dax-bus' mode, but that is kind of required for the kmem functionality anyway, so I'm not sure where we might be tripping.
On Mon, 2019-05-20 at 23:34 +0000, Verma, Vishal L wrote: > On Fri, 2019-05-17 at 13:41 -0400, Pavel Tatashin wrote: > > > Hi Pavel, > > > > > > I've still not been able to hit this in my testing, is it something > > > you > > > hit only after applying these patches? i.e. does plain v65 work? > > > > Yes, plain v65 works, but with these patches I see this error. > > > Hm, So there are only two patches that touch the add_dax_dev function: > > 2bf9a8e libdaxctl: add interfaces to enable/disable devices > 25be8f4 libdaxctl: add interfaces to get/set the online state for a node > > And of these, the second one to get the target node doesn't return an > error in any case. > > The first one can fail, so it must be that condition you're hitting, but > I'm not sure in what way it is failing. > > The exact check is: > > sprintf(path, "%s/modalias", daxdev_base); > rc = sysfs_read_attr(ctx, path, buf); > /* older kernels may be lack the modalias attribute */ > if (rc < 0 && rc != -ENOENT) > goto err_read; > if (rc == 0) { > dev->kmod_list = to_module_list(ctx, buf); > if (dev->kmod_list == NULL) > goto err_read; Dan points out that it might actually be the kmod portion that might be failing. Is libkmod present in the buildroot setup, and has a depmod run completed successfully before this point? In any case, this incremental patch should /at least/ delay the error until you actually try to enable a kmem device. I'll fold this into the next version of the series. 8<---- From 2df9b6401a69833fa709ab1ad83ac27b545aeb9e Mon Sep 17 00:00:00 2001 From: Vishal Verma <vishal.l.verma@intel.com> Date: Wed, 22 May 2019 12:01:28 -0600 Subject: [ndctl PATCH] fixup! libdaxctl: add interfaces to enable/disable devices --- daxctl/lib/libdaxctl.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/daxctl/lib/libdaxctl.c b/daxctl/lib/libdaxctl.c index 5c85328..9d23d12 100644 --- a/daxctl/lib/libdaxctl.c +++ b/daxctl/lib/libdaxctl.c @@ -393,12 +393,8 @@ static void *add_dax_dev(void *parent, int id, const char *daxdev_base) /* older kernels may be lack the modalias attribute */ if (rc < 0 && rc != -ENOENT) goto err_read; - if (rc == 0) { + if (rc == 0) dev->kmod_list = to_module_list(ctx, buf); - if (dev->kmod_list == NULL) - goto err_read; - } else - dbg(ctx, "%s: modalias attribute missing\n", devname); sprintf(path, "%s/target_node", daxdev_base); if (sysfs_read_attr(ctx, path, buf) == 0)
Changes in v3: - In daxctl_dev_get_mode(), remove the subsystem warning, detect dax-class and simply make it return devdax Changes in v2: - Add examples to the documentation page (Dave Hansen) - Clarify documentation regarding the conversion from system-ram to devdax - Remove any references to a persistent config from the documentation - those can be added when the feature is added. - device.c: validate option compatibility - daxctl-list: display numa_node for device listings - daxctl-list: display mode for device listings - make the options more consistent by adding a '-O' short option for --attempt-offline Add a new daxctl-reconfigure-device command that lets us reconfigure DAX devices back and forth between 'system-ram' and 'device-dax' modes. It also includes facilities to online any newly hot-plugged memory (default), and attempt to offline memory before converting away from the system-ram mode (not default, requires a --attempt-offline option). Currently missing from this series is a way to persistently store which devices have been 'marked' for use as system-ram. This depends on a config system overhaul in ndctl, and patches for those will follow separately and are independent of this work. Example invocations: 1. Reconfigure dax0.0 to system-ram mode, don’t online the memory # daxctl reconfigure-device --mode=system-ram --no-online dax0.0 [ { "chardev":"dax0.0", "size":16777216000, "numa_node":2, "mode":"system-ram" } ] 2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0 { "chardev":"dax0.0", "size":"15.63 GiB (16.78 GB)", "numa_node":2, "mode":"devdax" } 3. Reconfigure all dax devices on region0 to system-ram mode # daxctl reconfigure-device --mode=system-ram --region=0 all [ { "chardev":"dax0.0", "size":16777216000, "numa_node":2, "mode":"system-ram" }, { "chardev":"dax0.1", "size":16777216000, "numa_node":3, "mode":"system-ram" } ] These patches can also be found in the 'kmem-pending' branch on github: https://github.com/pmem/ndctl/tree/kmem-pending Cc: Dan Williams <dan.j.williams@intel.com> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Pavel Tatashin <pasha.tatashin@soleen.com> Vishal Verma (10): libdaxctl: add interfaces in support of device modes libdaxctl: cache 'subsystem' in daxctl_ctx libdaxctl: add interfaces to enable/disable devices libdaxctl: add interfaces to get/set the online state for a node daxctl/list: add numa_node for device listings libdaxctl: add an interface to get the mode for a dax device daxctl: add a new reconfigure-device command Documentation/daxctl: add a man page for daxctl-reconfigure-device contrib/ndctl: fix region-id completions for daxctl contrib/ndctl: add bash-completion for daxctl-reconfigure-device Documentation/daxctl/Makefile.am | 3 +- .../daxctl/daxctl-reconfigure-device.txt | 118 ++++ contrib/ndctl | 34 +- daxctl/Makefile.am | 2 + daxctl/builtin.h | 1 + daxctl/daxctl.c | 1 + daxctl/device.c | 237 ++++++++ daxctl/lib/Makefile.am | 3 +- daxctl/lib/libdaxctl-private.h | 21 + daxctl/lib/libdaxctl.c | 552 +++++++++++++++++- daxctl/lib/libdaxctl.sym | 14 + daxctl/libdaxctl.h | 16 + util/json.c | 22 + 13 files changed, 1013 insertions(+), 11 deletions(-) create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt create mode 100644 daxctl/device.c