| Message ID | 20191002234925.9190-5-vishal.l.verma@intel.com (mailing list archive) |
|---|---|
| State | Superseded |
| Commit | 358ee60e99edf22538952375e1c3cf741efafd34 |
| Headers | show |
| Series | fixes and movability for system-ram mode | expand |
On Wed, Oct 2, 2019 at 4:49 PM Vishal Verma <vishal.l.verma@intel.com> wrote: > > By default, daxctl always attempts to online new memory sections as > 'movable' so that routine kernel allocations aren't serviced from this > memory, and the memory is later removable via hot-unplug. > > System configuration, or other agents (such as udev rules) may race > 'daxctl' to online memory, and this may result in the memory not being > 'movable'. Add an interface to query the movability of a memory object > associated with a dax device. > > This is in preparation to both display a 'movable' attribute in device > listings, as well as optionally allowing memory to be onlined as > non-movable. > > Cc: Dan Williams <dan.j.williams@intel.com> > Reported-by: Ben Olson <ben.olson@intel.com> > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > --- > daxctl/lib/libdaxctl-private.h | 20 +++++++++ > daxctl/lib/libdaxctl.c | 77 ++++++++++++++++++++++++++++++++-- > daxctl/lib/libdaxctl.sym | 5 +++ > daxctl/libdaxctl.h | 1 + > 4 files changed, 100 insertions(+), 3 deletions(-) > > diff --git a/daxctl/lib/libdaxctl-private.h b/daxctl/lib/libdaxctl-private.h > index 7ba3c46..82939bb 100644 > --- a/daxctl/lib/libdaxctl-private.h > +++ b/daxctl/lib/libdaxctl-private.h > @@ -44,6 +44,25 @@ enum memory_op { > MEM_SET_ONLINE, > MEM_IS_ONLINE, > MEM_COUNT, > + MEM_FIND_ZONE, This is private so the naming is not too big a concern, but isn't this a MEM_GET_ZONE? A find operation to me is something that can succeed to find nothing, whereas a get operation fail assumes the association can always be retrieved barring exceptional conditions.
On Fri, 2019-10-18 at 11:54 -0700, Dan Williams wrote: > On Wed, Oct 2, 2019 at 4:49 PM Vishal Verma <vishal.l.verma@intel.com> wrote: > > By default, daxctl always attempts to online new memory sections as > > 'movable' so that routine kernel allocations aren't serviced from this > > memory, and the memory is later removable via hot-unplug. > > > > System configuration, or other agents (such as udev rules) may race > > 'daxctl' to online memory, and this may result in the memory not being > > 'movable'. Add an interface to query the movability of a memory object > > associated with a dax device. > > > > This is in preparation to both display a 'movable' attribute in device > > listings, as well as optionally allowing memory to be onlined as > > non-movable. > > > > Cc: Dan Williams <dan.j.williams@intel.com> > > Reported-by: Ben Olson <ben.olson@intel.com> > > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > > --- > > daxctl/lib/libdaxctl-private.h | 20 +++++++++ > > daxctl/lib/libdaxctl.c | 77 ++++++++++++++++++++++++++++++++-- > > daxctl/lib/libdaxctl.sym | 5 +++ > > daxctl/libdaxctl.h | 1 + > > 4 files changed, 100 insertions(+), 3 deletions(-) > > > > diff --git a/daxctl/lib/libdaxctl-private.h b/daxctl/lib/libdaxctl-private.h > > index 7ba3c46..82939bb 100644 > > --- a/daxctl/lib/libdaxctl-private.h > > +++ b/daxctl/lib/libdaxctl-private.h > > @@ -44,6 +44,25 @@ enum memory_op { > > MEM_SET_ONLINE, > > MEM_IS_ONLINE, > > MEM_COUNT, > > + MEM_FIND_ZONE, > > This is private so the naming is not too big a concern, but isn't this > a MEM_GET_ZONE? A find operation to me is something that can succeed > to find nothing, whereas a get operation fail assumes the association > can always be retrieved barring exceptional conditions. Hm, my personal view of find vs. get was that 'get' grabs a reference (or similar) to something which we know how to get to (have a pointer directly) etc. 'Find' is more of a 'go searching for' something - and it may involve walking and looping over data structures. But I'm not too opposed to this, and can change to 'get' if that follows convention better.
On Fri, Oct 18, 2019 at 12:57 PM Verma, Vishal L <vishal.l.verma@intel.com> wrote: > > > On Fri, 2019-10-18 at 11:54 -0700, Dan Williams wrote: > > On Wed, Oct 2, 2019 at 4:49 PM Vishal Verma <vishal.l.verma@intel.com> wrote: > > > By default, daxctl always attempts to online new memory sections as > > > 'movable' so that routine kernel allocations aren't serviced from this > > > memory, and the memory is later removable via hot-unplug. > > > > > > System configuration, or other agents (such as udev rules) may race > > > 'daxctl' to online memory, and this may result in the memory not being > > > 'movable'. Add an interface to query the movability of a memory object > > > associated with a dax device. > > > > > > This is in preparation to both display a 'movable' attribute in device > > > listings, as well as optionally allowing memory to be onlined as > > > non-movable. > > > > > > Cc: Dan Williams <dan.j.williams@intel.com> > > > Reported-by: Ben Olson <ben.olson@intel.com> > > > Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> > > > --- > > > daxctl/lib/libdaxctl-private.h | 20 +++++++++ > > > daxctl/lib/libdaxctl.c | 77 ++++++++++++++++++++++++++++++++-- > > > daxctl/lib/libdaxctl.sym | 5 +++ > > > daxctl/libdaxctl.h | 1 + > > > 4 files changed, 100 insertions(+), 3 deletions(-) > > > > > > diff --git a/daxctl/lib/libdaxctl-private.h b/daxctl/lib/libdaxctl-private.h > > > index 7ba3c46..82939bb 100644 > > > --- a/daxctl/lib/libdaxctl-private.h > > > +++ b/daxctl/lib/libdaxctl-private.h > > > @@ -44,6 +44,25 @@ enum memory_op { > > > MEM_SET_ONLINE, > > > MEM_IS_ONLINE, > > > MEM_COUNT, > > > + MEM_FIND_ZONE, > > > > This is private so the naming is not too big a concern, but isn't this > > a MEM_GET_ZONE? A find operation to me is something that can succeed > > to find nothing, whereas a get operation fail assumes the association > > can always be retrieved barring exceptional conditions. > > Hm, my personal view of find vs. get was that 'get' grabs a reference > (or similar) to something which we know how to get to (have a pointer > directly) etc. Oh, that's get as in get/put. I'm talking about get/set. Where the zone is a property memblock that can set and retrieved. > 'Find' is more of a 'go searching for' something - and it > may involve walking and looping over data structures. > > But I'm not too opposed to this, and can change to 'get' if that follows > convention better. Like I said it's private, so it's not a big deal (i.e. API users won't see it), but ndctl does have a few apis that walk and loop over data structures using a get verb, or the ones that retrieve the dynamic state of a sysfs attribute. I will grant you that in hindsight some of the "get_by" apis might have been better named "find".
diff --git a/daxctl/lib/libdaxctl-private.h b/daxctl/lib/libdaxctl-private.h index 7ba3c46..82939bb 100644 --- a/daxctl/lib/libdaxctl-private.h +++ b/daxctl/lib/libdaxctl-private.h @@ -44,6 +44,25 @@ enum memory_op { MEM_SET_ONLINE, MEM_IS_ONLINE, MEM_COUNT, + MEM_FIND_ZONE, +}; + +/* OR-able flags, 1, 2, 4, 8 etc */ +enum memory_op_status { + MEM_ST_OK = 0, + MEM_ST_ZONE_INCONSISTENT = 1, +}; + +enum memory_zones { + MEM_ZONE_UNKNOWN = 1, + MEM_ZONE_MOVABLE, + MEM_ZONE_NORMAL, +}; + +static const char *zone_strings[] = { + [MEM_ZONE_UNKNOWN] = "mixed", + [MEM_ZONE_NORMAL] = "Normal", + [MEM_ZONE_MOVABLE] = "Movable", }; /** @@ -86,6 +105,7 @@ struct daxctl_memory { size_t buf_len; char *node_path; unsigned long block_size; + enum memory_zones zone; }; diff --git a/daxctl/lib/libdaxctl.c b/daxctl/lib/libdaxctl.c index 6243857..4460174 100644 --- a/daxctl/lib/libdaxctl.c +++ b/daxctl/lib/libdaxctl.c @@ -1159,6 +1159,58 @@ static int offline_one_memblock(struct daxctl_memory *mem, char *memblock) return rc; } +static int memblock_find_zone(struct daxctl_memory *mem, char *memblock, + int *status) +{ + struct daxctl_dev *dev = daxctl_memory_get_dev(mem); + const char *devname = daxctl_dev_get_devname(dev); + struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev); + enum memory_zones cur_zone; + int len = mem->buf_len, rc; + char buf[SYSFS_ATTR_SIZE]; + char *path = mem->mem_buf; + const char *node_path; + + rc = memblock_is_online(mem, memblock); + if (rc < 0) + return rc; + if (rc == 0) + return -ENXIO; + + node_path = daxctl_memory_get_node_path(mem); + if (!node_path) + return -ENXIO; + + rc = snprintf(path, len, "%s/%s/valid_zones", node_path, memblock); + if (rc < 0) + return -ENOMEM; + + rc = sysfs_read_attr(ctx, path, buf); + if (rc) { + err(ctx, "%s: Failed to read %s: %s\n", + devname, path, strerror(-rc)); + return rc; + } + + if (strcmp(buf, zone_strings[MEM_ZONE_MOVABLE]) == 0) + cur_zone = MEM_ZONE_MOVABLE; + else if (strcmp(buf, zone_strings[MEM_ZONE_NORMAL]) == 0) + cur_zone = MEM_ZONE_NORMAL; + else + cur_zone = MEM_ZONE_UNKNOWN; + + if (mem->zone) { + if (mem->zone == cur_zone) + return 0; + else + *status |= MEM_ST_ZONE_INCONSISTENT; + } else { + mem->zone = cur_zone; + } + + return 0; +} + static bool memblock_in_dev(struct daxctl_memory *mem, const char *memblock) { const char *mem_base = "/sys/devices/system/memory/"; @@ -1211,7 +1263,7 @@ static bool memblock_in_dev(struct daxctl_memory *mem, const char *memblock) } static int op_for_one_memblock(struct daxctl_memory *mem, char *memblock, - enum memory_op op) + enum memory_op op, int *status) { struct daxctl_dev *dev = daxctl_memory_get_dev(mem); const char *devname = daxctl_dev_get_devname(dev); @@ -1234,6 +1286,8 @@ static int op_for_one_memblock(struct daxctl_memory *mem, char *memblock, return !rc; case MEM_COUNT: return 0; + case MEM_FIND_ZONE: + return memblock_find_zone(mem, memblock, status); } err(ctx, "%s: BUG: unknown op: %d\n", devname, op); @@ -1245,8 +1299,8 @@ static int daxctl_memory_op(struct daxctl_memory *mem, enum memory_op op) struct daxctl_dev *dev = daxctl_memory_get_dev(mem); const char *devname = daxctl_dev_get_devname(dev); struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev); + int rc, count = 0, status_flags = 0; const char *node_path; - int rc, count = 0; struct dirent *de; DIR *node_dir; @@ -1265,7 +1319,8 @@ static int daxctl_memory_op(struct daxctl_memory *mem, enum memory_op op) if (strncmp(de->d_name, "memory", 6) == 0) { if (!memblock_in_dev(mem, de->d_name)) continue; - rc = op_for_one_memblock(mem, de->d_name, op); + rc = op_for_one_memblock(mem, de->d_name, op, + &status_flags); if (rc < 0) goto out_dir; if (rc == 0) @@ -1273,6 +1328,10 @@ static int daxctl_memory_op(struct daxctl_memory *mem, enum memory_op op) } errno = 0; } + + if (status_flags & MEM_ST_ZONE_INCONSISTENT) + mem->zone = MEM_ZONE_UNKNOWN; + if (errno) { rc = -errno; goto out_dir; @@ -1303,3 +1362,15 @@ DAXCTL_EXPORT int daxctl_memory_num_sections(struct daxctl_memory *mem) { return daxctl_memory_op(mem, MEM_COUNT); } + +DAXCTL_EXPORT int daxctl_memory_is_movable(struct daxctl_memory *mem) +{ + int rc; + + /* Start a fresh zone scan, clear any previous info */ + mem->zone = 0; + rc = daxctl_memory_op(mem, MEM_FIND_ZONE); + if (rc < 0) + return rc; + return (mem->zone == MEM_ZONE_MOVABLE) ? 1 : 0; +} diff --git a/daxctl/lib/libdaxctl.sym b/daxctl/lib/libdaxctl.sym index bc18604..041d9a5 100644 --- a/daxctl/lib/libdaxctl.sym +++ b/daxctl/lib/libdaxctl.sym @@ -69,3 +69,8 @@ global: daxctl_memory_is_online; daxctl_memory_num_sections; } LIBDAXCTL_5; + +LIBDAXCTL_7 { +global: + daxctl_memory_is_movable; +} LIBDAXCTL_6; diff --git a/daxctl/libdaxctl.h b/daxctl/libdaxctl.h index fb6c3b1..8d5f8b7 100644 --- a/daxctl/libdaxctl.h +++ b/daxctl/libdaxctl.h @@ -84,6 +84,7 @@ int daxctl_memory_online(struct daxctl_memory *mem); int daxctl_memory_offline(struct daxctl_memory *mem); int daxctl_memory_is_online(struct daxctl_memory *mem); int daxctl_memory_num_sections(struct daxctl_memory *mem); +int daxctl_memory_is_movable(struct daxctl_memory *mem); #define daxctl_dev_foreach(region, dev) \ for (dev = daxctl_dev_get_first(region); \
By default, daxctl always attempts to online new memory sections as 'movable' so that routine kernel allocations aren't serviced from this memory, and the memory is later removable via hot-unplug. System configuration, or other agents (such as udev rules) may race 'daxctl' to online memory, and this may result in the memory not being 'movable'. Add an interface to query the movability of a memory object associated with a dax device. This is in preparation to both display a 'movable' attribute in device listings, as well as optionally allowing memory to be onlined as non-movable. Cc: Dan Williams <dan.j.williams@intel.com> Reported-by: Ben Olson <ben.olson@intel.com> Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> --- daxctl/lib/libdaxctl-private.h | 20 +++++++++ daxctl/lib/libdaxctl.c | 77 ++++++++++++++++++++++++++++++++-- daxctl/lib/libdaxctl.sym | 5 +++ daxctl/libdaxctl.h | 1 + 4 files changed, 100 insertions(+), 3 deletions(-)