| Message ID | 20191210080628.5264-2-sjpark@amazon.de (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | xenbus/backend: Add a memory pressure handler callback | expand |
On 10.12.19 09:06, SeongJae Park wrote: > Granting pages consumes backend system memory. In systems configured > with insufficient spare memory for those pages, it can cause a memory > pressure situation. However, finding the optimal amount of the spare > memory is challenging for large systems having dynamic resource > utilization patterns. Also, such a static configuration might lack a > flexibility. > > To mitigate such problems, this commit adds a memory reclaim callback to > 'xenbus_driver'. Using this facility, 'xenbus' would be able to monitor > a memory pressure and request specific devices of specific backend > drivers which causing the given pressure to voluntarily release its > memory. > > That said, this commit simply requests every callback registered driver > to release its memory for every domain, rather than issueing the > requests to the drivers and the domain in charge. Such things will be > done in a futur. Also, this commit focuses on memory only. However, it > would be ablt to be extended for general resources. > > Signed-off-by: SeongJae Park <sjpark@amazon.de> Reviewed-by: Juergen Gross <jgross@suse.com> Juergen
On Tue, Dec 10, 2019 at 08:06:27AM +0000, SeongJae Park wrote: > Granting pages consumes backend system memory. In systems configured > with insufficient spare memory for those pages, it can cause a memory > pressure situation. However, finding the optimal amount of the spare > memory is challenging for large systems having dynamic resource > utilization patterns. Also, such a static configuration might lack a s/lack a/lack/ > flexibility. > > To mitigate such problems, this commit adds a memory reclaim callback to > 'xenbus_driver'. Using this facility, 'xenbus' would be able to monitor > a memory pressure and request specific devices of specific backend s/monitor a/monitor/ > drivers which causing the given pressure to voluntarily release its ...which are causing... > memory. > > That said, this commit simply requests every callback registered driver > to release its memory for every domain, rather than issueing the s/issueing/issuing/ > requests to the drivers and the domain in charge. Such things will be I'm afraid I don't understand the "domain in charge" part of this sentence. > done in a futur. Also, this commit focuses on memory only. However, it ... done in a future change. Also I think the period after only should be removed in order to tie both sentences together. > would be ablt to be extended for general resources. s/ablt/able/ > > Signed-off-by: SeongJae Park <sjpark@amazon.de> > --- > drivers/xen/xenbus/xenbus_probe_backend.c | 31 +++++++++++++++++++++++ > include/xen/xenbus.h | 1 + > 2 files changed, 32 insertions(+) > > diff --git a/drivers/xen/xenbus/xenbus_probe_backend.c b/drivers/xen/xenbus/xenbus_probe_backend.c > index b0bed4faf44c..5a5ba29e39df 100644 > --- a/drivers/xen/xenbus/xenbus_probe_backend.c > +++ b/drivers/xen/xenbus/xenbus_probe_backend.c > @@ -248,6 +248,34 @@ static int backend_probe_and_watch(struct notifier_block *notifier, > return NOTIFY_DONE; > } > > +static int xenbus_backend_reclaim(struct device *dev, void *data) > +{ > + struct xenbus_driver *drv; Newline and const. > + if (!dev->driver) > + return -ENOENT; > + drv = to_xenbus_driver(dev->driver); > + if (drv && drv->reclaim) > + drv->reclaim(to_xenbus_device(dev)); You seem to completely ignore the return of the reclaim hook... > + return 0; > +} > + > +/* > + * Returns 0 always because we are using shrinker to only detect memory > + * pressure. > + */ > +static unsigned long xenbus_backend_shrink_count(struct shrinker *shrinker, > + struct shrink_control *sc) > +{ > + bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, > + xenbus_backend_reclaim); > + return 0; > +} > + > +static struct shrinker xenbus_backend_shrinker = { > + .count_objects = xenbus_backend_shrink_count, > + .seeks = DEFAULT_SEEKS, > +}; > + > static int __init xenbus_probe_backend_init(void) > { > static struct notifier_block xenstore_notifier = { > @@ -264,6 +292,9 @@ static int __init xenbus_probe_backend_init(void) > > register_xenstore_notifier(&xenstore_notifier); > > + if (register_shrinker(&xenbus_backend_shrinker)) > + pr_warn("shrinker registration failed\n"); > + > return 0; > } > subsys_initcall(xenbus_probe_backend_init); > diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h > index 869c816d5f8c..cdb075e4182f 100644 > --- a/include/xen/xenbus.h > +++ b/include/xen/xenbus.h > @@ -104,6 +104,7 @@ struct xenbus_driver { > struct device_driver driver; > int (*read_otherend_details)(struct xenbus_device *dev); > int (*is_ready)(struct xenbus_device *dev); > + unsigned (*reclaim)(struct xenbus_device *dev); ... hence I wonder why it's returning an unsigned when it's just ignored. IMO it should return an int to signal errors, and the return should be ignored. Also, I think it would preferable for this function to take an extra parameter to describe the resource the driver should attempt to free (ie: memory or interrupts for example). I'm however not able to find any existing Linux type to describe such resources. Thanks, Roger.
On Tue, Dec 10, 2019 at 11:16:35AM +0100, Roger Pau Monné wrote: > On Tue, Dec 10, 2019 at 08:06:27AM +0000, SeongJae Park wrote: > > diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h > > index 869c816d5f8c..cdb075e4182f 100644 > > --- a/include/xen/xenbus.h > > +++ b/include/xen/xenbus.h > > @@ -104,6 +104,7 @@ struct xenbus_driver { > > struct device_driver driver; > > int (*read_otherend_details)(struct xenbus_device *dev); > > int (*is_ready)(struct xenbus_device *dev); > > + unsigned (*reclaim)(struct xenbus_device *dev); > > ... hence I wonder why it's returning an unsigned when it's just > ignored. > > IMO it should return an int to signal errors, and the return should be > ignored. Meant to write 'shouldn't be ignored' sorry. Roger.
On Tue, Dec 10, 2019 at 11:21 AM Roger Pau Monné <roger.pau@citrix.com> wrote: > > On Tue, Dec 10, 2019 at 11:16:35AM +0100, Roger Pau Monné wrote: > > On Tue, Dec 10, 2019 at 08:06:27AM +0000, SeongJae Park wrote: > > > diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h > > > index 869c816d5f8c..cdb075e4182f 100644 > > > --- a/include/xen/xenbus.h > > > +++ b/include/xen/xenbus.h > > > @@ -104,6 +104,7 @@ struct xenbus_driver { > > > struct device_driver driver; > > > int (*read_otherend_details)(struct xenbus_device *dev); > > > int (*is_ready)(struct xenbus_device *dev); > > > + unsigned (*reclaim)(struct xenbus_device *dev); > > > > ... hence I wonder why it's returning an unsigned when it's just > > ignored. > > > > IMO it should return an int to signal errors, and the return should be > > ignored. > > Meant to write 'shouldn't be ignored' sorry. Thanks for good opinions and comments! I will apply your comments in the next version. Thanks, SeongJae Park > > Roger.
On Tue, 10 Dec 2019 11:16:35 +0100 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > > Granting pages consumes backend system memory. In systems configured > > with insufficient spare memory for those pages, it can cause a memory > > pressure situation. However, finding the optimal amount of the spare > > memory is challenging for large systems having dynamic resource > > utilization patterns. Also, such a static configuration might lack a > > s/lack a/lack/ > > > flexibility. > > > > To mitigate such problems, this commit adds a memory reclaim callback to > > 'xenbus_driver'. Using this facility, 'xenbus' would be able to monitor > > a memory pressure and request specific devices of specific backend > > s/monitor a/monitor/ > > > drivers which causing the given pressure to voluntarily release its > > ...which are causing... > > > memory. > > > > That said, this commit simply requests every callback registered driver > > to release its memory for every domain, rather than issueing the > > s/issueing/issuing/ > > > requests to the drivers and the domain in charge. Such things will be > > I'm afraid I don't understand the "domain in charge" part of this > sentence. > > > done in a futur. Also, this commit focuses on memory only. However, it > > ... done in a future change. Also I think the period after only should > be removed in order to tie both sentences together. > > > would be ablt to be extended for general resources. > > s/ablt/able/ > > > > > Signed-off-by: SeongJae Park <sjpark@amazon.de> > > --- > > drivers/xen/xenbus/xenbus_probe_backend.c | 31 +++++++++++++++++++++++ > > include/xen/xenbus.h | 1 + > > 2 files changed, 32 insertions(+) > > > > diff --git a/drivers/xen/xenbus/xenbus_probe_backend.c b/drivers/xen/xenbus/xenbus_probe_backend.c > > index b0bed4faf44c..5a5ba29e39df 100644 > > --- a/drivers/xen/xenbus/xenbus_probe_backend.c > > +++ b/drivers/xen/xenbus/xenbus_probe_backend.c > > @@ -248,6 +248,34 @@ static int backend_probe_and_watch(struct notifier_block *notifier, > > return NOTIFY_DONE; > > } > > > > +static int xenbus_backend_reclaim(struct device *dev, void *data) > > +{ > > + struct xenbus_driver *drv; > > Newline and const. > > > + if (!dev->driver) > > + return -ENOENT; > > + drv = to_xenbus_driver(dev->driver); > > + if (drv && drv->reclaim) > > + drv->reclaim(to_xenbus_device(dev)); > > You seem to completely ignore the return of the reclaim hook... > > > + return 0; > > +} > > + > > +/* > > + * Returns 0 always because we are using shrinker to only detect memory > > + * pressure. > > + */ > > +static unsigned long xenbus_backend_shrink_count(struct shrinker *shrinker, > > + struct shrink_control *sc) > > +{ > > + bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, > > + xenbus_backend_reclaim); > > + return 0; > > +} > > + > > +static struct shrinker xenbus_backend_shrinker = { > > + .count_objects = xenbus_backend_shrink_count, > > + .seeks = DEFAULT_SEEKS, > > +}; > > + > > static int __init xenbus_probe_backend_init(void) > > { > > static struct notifier_block xenstore_notifier = { > > @@ -264,6 +292,9 @@ static int __init xenbus_probe_backend_init(void) > > > > register_xenstore_notifier(&xenstore_notifier); > > > > + if (register_shrinker(&xenbus_backend_shrinker)) > > + pr_warn("shrinker registration failed\n"); > > + > > return 0; > > } > > subsys_initcall(xenbus_probe_backend_init); > > diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h > > index 869c816d5f8c..cdb075e4182f 100644 > > --- a/include/xen/xenbus.h > > +++ b/include/xen/xenbus.h > > @@ -104,6 +104,7 @@ struct xenbus_driver { > > struct device_driver driver; > > int (*read_otherend_details)(struct xenbus_device *dev); > > int (*is_ready)(struct xenbus_device *dev); > > + unsigned (*reclaim)(struct xenbus_device *dev); > > ... hence I wonder why it's returning an unsigned when it's just > ignored. > > IMO it should return an int to signal errors, and the return should be > ignored. I first thought similarly and set the callback to return something. However, as this callback is called to simply notify the memory pressure and ask the driver to free its memory as many as possible, I couldn't easily imagine what kind of errors that need to be handled by its caller can occur in the callback, especially because current blkback's callback implementation has no such error. So, if you and others agree, I would like to simply set the return type to 'void' for now and defer the error handling to a future change. > > Also, I think it would preferable for this function to take an extra > parameter to describe the resource the driver should attempt to free > (ie: memory or interrupts for example). I'm however not able to find > any existing Linux type to describe such resources. Yes, such extention would be the right direction. However, because there is no existing Linux type to describe the type of resources to reclaim as you also mentioned, there could be many different opinions about its implementation detail. In my opinion, it could be also possible to simply add another callback for another resource type. That said, because currently we have an use case and an implementation for the memory pressure only, I would like to let it as is for now and defer the extension as a future work, if you and others have no objection. Thanks, SeongJae Park > > Thanks, Roger. > >
On Wed, Dec 11, 2019 at 04:50:58AM +0100, SeongJae Park wrote: > On Tue, 10 Dec 2019 11:16:35 +0100 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > > > diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h > > > index 869c816d5f8c..cdb075e4182f 100644 > > > --- a/include/xen/xenbus.h > > > +++ b/include/xen/xenbus.h > > > @@ -104,6 +104,7 @@ struct xenbus_driver { > > > struct device_driver driver; > > > int (*read_otherend_details)(struct xenbus_device *dev); > > > int (*is_ready)(struct xenbus_device *dev); > > > + unsigned (*reclaim)(struct xenbus_device *dev); > > > > ... hence I wonder why it's returning an unsigned when it's just > > ignored. > > > > IMO it should return an int to signal errors, and the return should be > > ignored. > > I first thought similarly and set the callback to return something. However, > as this callback is called to simply notify the memory pressure and ask the > driver to free its memory as many as possible, I couldn't easily imagine what > kind of errors that need to be handled by its caller can occur in the callback, > especially because current blkback's callback implementation has no such error. > So, if you and others agree, I would like to simply set the return type to > 'void' for now and defer the error handling to a future change. Yes, I also wondered the same, but seeing you returned an integer I assumed there was interest in returning some kind of value. If there's nothing to return let's just make it void. > > > > Also, I think it would preferable for this function to take an extra > > parameter to describe the resource the driver should attempt to free > > (ie: memory or interrupts for example). I'm however not able to find > > any existing Linux type to describe such resources. > > Yes, such extention would be the right direction. However, because there is no > existing Linux type to describe the type of resources to reclaim as you also > mentioned, there could be many different opinions about its implementation > detail. In my opinion, it could be also possible to simply add another > callback for another resource type. That said, because currently we have an > use case and an implementation for the memory pressure only, I would like to > let it as is for now and defer the extension as a future work, if you and > others have no objection. Ack, can I please ask the callback to be named reclaim_memory or some such then? Thanks, Roger.
diff --git a/drivers/xen/xenbus/xenbus_probe_backend.c b/drivers/xen/xenbus/xenbus_probe_backend.c index b0bed4faf44c..5a5ba29e39df 100644 --- a/drivers/xen/xenbus/xenbus_probe_backend.c +++ b/drivers/xen/xenbus/xenbus_probe_backend.c @@ -248,6 +248,34 @@ static int backend_probe_and_watch(struct notifier_block *notifier, return NOTIFY_DONE; } +static int xenbus_backend_reclaim(struct device *dev, void *data) +{ + struct xenbus_driver *drv; + if (!dev->driver) + return -ENOENT; + drv = to_xenbus_driver(dev->driver); + if (drv && drv->reclaim) + drv->reclaim(to_xenbus_device(dev)); + return 0; +} + +/* + * Returns 0 always because we are using shrinker to only detect memory + * pressure. + */ +static unsigned long xenbus_backend_shrink_count(struct shrinker *shrinker, + struct shrink_control *sc) +{ + bus_for_each_dev(&xenbus_backend.bus, NULL, NULL, + xenbus_backend_reclaim); + return 0; +} + +static struct shrinker xenbus_backend_shrinker = { + .count_objects = xenbus_backend_shrink_count, + .seeks = DEFAULT_SEEKS, +}; + static int __init xenbus_probe_backend_init(void) { static struct notifier_block xenstore_notifier = { @@ -264,6 +292,9 @@ static int __init xenbus_probe_backend_init(void) register_xenstore_notifier(&xenstore_notifier); + if (register_shrinker(&xenbus_backend_shrinker)) + pr_warn("shrinker registration failed\n"); + return 0; } subsys_initcall(xenbus_probe_backend_init); diff --git a/include/xen/xenbus.h b/include/xen/xenbus.h index 869c816d5f8c..cdb075e4182f 100644 --- a/include/xen/xenbus.h +++ b/include/xen/xenbus.h @@ -104,6 +104,7 @@ struct xenbus_driver { struct device_driver driver; int (*read_otherend_details)(struct xenbus_device *dev); int (*is_ready)(struct xenbus_device *dev); + unsigned (*reclaim)(struct xenbus_device *dev); }; static inline struct xenbus_driver *to_xenbus_driver(struct device_driver *drv)
Granting pages consumes backend system memory. In systems configured with insufficient spare memory for those pages, it can cause a memory pressure situation. However, finding the optimal amount of the spare memory is challenging for large systems having dynamic resource utilization patterns. Also, such a static configuration might lack a flexibility. To mitigate such problems, this commit adds a memory reclaim callback to 'xenbus_driver'. Using this facility, 'xenbus' would be able to monitor a memory pressure and request specific devices of specific backend drivers which causing the given pressure to voluntarily release its memory. That said, this commit simply requests every callback registered driver to release its memory for every domain, rather than issueing the requests to the drivers and the domain in charge. Such things will be done in a futur. Also, this commit focuses on memory only. However, it would be ablt to be extended for general resources. Signed-off-by: SeongJae Park <sjpark@amazon.de> --- drivers/xen/xenbus/xenbus_probe_backend.c | 31 +++++++++++++++++++++++ include/xen/xenbus.h | 1 + 2 files changed, 32 insertions(+)