| Message ID | 20181205023426.24029-2-richard.weiyang@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [1/2] admin-guide/memory-hotplug.rst: remove locking internal part from admin-guide | expand |
On 05.12.18 03:34, Wei Yang wrote: > Currently locking for memory hotplug is a little complicated. > > Generally speaking, we leverage the two global lock: > > * device_hotplug_lock > * mem_hotplug_lock > > to serialise the process. > > While for the long term, we are willing to have more fine-grained lock > to provide higher scalability. > > This patch divides Locking Internal section based on these two global > locks to help readers to understand it. Also it adds some new finding to > enrich it. > > [David: words arrangement] > > Signed-off-by: Wei Yang <richard.weiyang@gmail.com> > --- > Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- > 1 file changed, 24 insertions(+), 3 deletions(-) > > diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst > index de7467e48067..95662b283328 100644 > --- a/Documentation/core-api/memory-hotplug.rst > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. > Locking Internals > ================= > > +There are three locks involved in memory-hotplug, two global lock and one local > +lock: > + > +- device_hotplug_lock > +- mem_hotplug_lock > +- device_lock > + Do we really only ever use these three and not anything else when adding/removing/onlining/offlining memory? (I am thinking e.g. about pgdat_resize_lock) If so, you should phrase that maybe more generally Or add more details :) "In addition to fine grained locks like pgdat_resize_lock, there are three locks involved ..." > +Currently, they are twisted together for all kinds of reasons. The following > +part is divided into device_hotplug_lock and mem_hotplug_lock parts > +respectively to describe those tricky situations. > + > +device_hotplug_lock > +--------------------- > + > When adding/removing memory that uses memory block devices (i.e. ordinary RAM), > the device_hotplug_lock should be held to: > > @@ -111,13 +125,20 @@ As the device is visible to user space before taking the device_lock(), this > can result in a lock inversion. > > onlining/offlining of memory should be done via device_online()/ > -device_offline() - to make sure it is properly synchronized to actions > -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) > +device_offline() - to make sure it is properly synchronized to actions via > +sysfs. Even mem_hotplug_lock is used to protect the process, because of the > +lock inversion described above, holding device_hotplug_lock is still advised > +(to e.g. protect online_type) > + > +mem_hotplug_lock > +--------------------- > > When adding/removing/onlining/offlining memory or adding/removing > heterogeneous/device memory, we should always hold the mem_hotplug_lock in > write mode to serialise memory hotplug (e.g. access to global/zone > -variables). > +variables). Currently, we take advantage of this to serialise sparsemem's > +mem_section handling in sparse_add_one_section() and > +sparse_remove_one_section(). > > In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read > mode allows for a quite efficient get_online_mems/put_online_mems >
On Wed, Dec 05, 2018 at 10:34:26AM +0800, Wei Yang wrote: > Currently locking for memory hotplug is a little complicated. > > Generally speaking, we leverage the two global lock: > > * device_hotplug_lock > * mem_hotplug_lock > > to serialise the process. > > While for the long term, we are willing to have more fine-grained lock > to provide higher scalability. > > This patch divides Locking Internal section based on these two global > locks to help readers to understand it. Also it adds some new finding to > enrich it. > > [David: words arrangement] > > Signed-off-by: Wei Yang <richard.weiyang@gmail.com> > --- > Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- > 1 file changed, 24 insertions(+), 3 deletions(-) > > diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst > index de7467e48067..95662b283328 100644 > --- a/Documentation/core-api/memory-hotplug.rst > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. > Locking Internals > ================= > > +There are three locks involved in memory-hotplug, two global lock and one local typo: ^locks > +lock: > + > +- device_hotplug_lock > +- mem_hotplug_lock > +- device_lock > + > +Currently, they are twisted together for all kinds of reasons. The following > +part is divided into device_hotplug_lock and mem_hotplug_lock parts > +respectively to describe those tricky situations. > + > +device_hotplug_lock > +--------------------- > + > When adding/removing memory that uses memory block devices (i.e. ordinary RAM), > the device_hotplug_lock should be held to: > > @@ -111,13 +125,20 @@ As the device is visible to user space before taking the device_lock(), this > can result in a lock inversion. > > onlining/offlining of memory should be done via device_online()/ > -device_offline() - to make sure it is properly synchronized to actions > -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) > +device_offline() - to make sure it is properly synchronized to actions via > +sysfs. Even mem_hotplug_lock is used to protect the process, because of the I think it should be "Even if mem_hotplug_lock ..." > +lock inversion described above, holding device_hotplug_lock is still advised > +(to e.g. protect online_type) > + > +mem_hotplug_lock > +--------------------- > > When adding/removing/onlining/offlining memory or adding/removing > heterogeneous/device memory, we should always hold the mem_hotplug_lock in > write mode to serialise memory hotplug (e.g. access to global/zone > -variables). > +variables). Currently, we take advantage of this to serialise sparsemem's > +mem_section handling in sparse_add_one_section() and > +sparse_remove_one_section(). > > In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read > mode allows for a quite efficient get_online_mems/put_online_mems > -- > 2.15.1 >
On Wed, Dec 05, 2018 at 09:08:47AM +0100, David Hildenbrand wrote: >On 05.12.18 03:34, Wei Yang wrote: >> Currently locking for memory hotplug is a little complicated. >> >> Generally speaking, we leverage the two global lock: >> >> * device_hotplug_lock >> * mem_hotplug_lock >> >> to serialise the process. >> >> While for the long term, we are willing to have more fine-grained lock >> to provide higher scalability. >> >> This patch divides Locking Internal section based on these two global >> locks to help readers to understand it. Also it adds some new finding to >> enrich it. >> >> [David: words arrangement] >> >> Signed-off-by: Wei Yang <richard.weiyang@gmail.com> >> --- >> Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- >> 1 file changed, 24 insertions(+), 3 deletions(-) >> >> diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst >> index de7467e48067..95662b283328 100644 >> --- a/Documentation/core-api/memory-hotplug.rst >> +++ b/Documentation/core-api/memory-hotplug.rst >> @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. >> Locking Internals >> ================= >> >> +There are three locks involved in memory-hotplug, two global lock and one local >> +lock: >> + >> +- device_hotplug_lock >> +- mem_hotplug_lock >> +- device_lock >> + > >Do we really only ever use these three and not anything else when >adding/removing/onlining/offlining memory? > >(I am thinking e.g. about pgdat_resize_lock) Yes there are more than those three, pgdat_resize_lock is one of them. > >If so, you should phrase that maybe more generally Or add more details :) Yep, while I don't get a whole picture about the pgdat_resize_lock. The usage of this lock scatter in many places. > >"In addition to fine grained locks like pgdat_resize_lock, there are >three locks involved ..." > Sounds better :-)
On Wed, Dec 05, 2018 at 10:40:45AM +0200, Mike Rapoport wrote: >On Wed, Dec 05, 2018 at 10:34:26AM +0800, Wei Yang wrote: >> Currently locking for memory hotplug is a little complicated. >> >> Generally speaking, we leverage the two global lock: >> >> * device_hotplug_lock >> * mem_hotplug_lock >> >> to serialise the process. >> >> While for the long term, we are willing to have more fine-grained lock >> to provide higher scalability. >> >> This patch divides Locking Internal section based on these two global >> locks to help readers to understand it. Also it adds some new finding to >> enrich it. >> >> [David: words arrangement] >> >> Signed-off-by: Wei Yang <richard.weiyang@gmail.com> >> --- >> Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- >> 1 file changed, 24 insertions(+), 3 deletions(-) >> >> diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst >> index de7467e48067..95662b283328 100644 >> --- a/Documentation/core-api/memory-hotplug.rst >> +++ b/Documentation/core-api/memory-hotplug.rst >> @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. >> Locking Internals >> ================= >> >> +There are three locks involved in memory-hotplug, two global lock and one local > >typo: ^locks > Thanks :-) >> +lock: >> + >> +- device_hotplug_lock >> +- mem_hotplug_lock >> +- device_lock >> + >> +Currently, they are twisted together for all kinds of reasons. The following >> +part is divided into device_hotplug_lock and mem_hotplug_lock parts >> +respectively to describe those tricky situations. >> + >> +device_hotplug_lock >> +--------------------- >> + >> When adding/removing memory that uses memory block devices (i.e. ordinary RAM), >> the device_hotplug_lock should be held to: >> >> @@ -111,13 +125,20 @@ As the device is visible to user space before taking the device_lock(), this >> can result in a lock inversion. >> >> onlining/offlining of memory should be done via device_online()/ >> -device_offline() - to make sure it is properly synchronized to actions >> -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) >> +device_offline() - to make sure it is properly synchronized to actions via >> +sysfs. Even mem_hotplug_lock is used to protect the process, because of the > >I think it should be "Even if mem_hotplug_lock ..." > Ah, my poor English, will fix it in next version. :-) >> +lock inversion described above, holding device_hotplug_lock is still advised >> +(to e.g. protect online_type) >> + >> +mem_hotplug_lock >> +--------------------- >> >> When adding/removing/onlining/offlining memory or adding/removing >> heterogeneous/device memory, we should always hold the mem_hotplug_lock in >> write mode to serialise memory hotplug (e.g. access to global/zone >> -variables). >> +variables). Currently, we take advantage of this to serialise sparsemem's >> +mem_section handling in sparse_add_one_section() and >> +sparse_remove_one_section(). >> >> In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read >> mode allows for a quite efficient get_online_mems/put_online_mems >> -- >> 2.15.1 >> > >-- >Sincerely yours, >Mike.
On Wed 05-12-18 10:34:26, Wei Yang wrote: > Currently locking for memory hotplug is a little complicated. > > Generally speaking, we leverage the two global lock: > > * device_hotplug_lock > * mem_hotplug_lock > > to serialise the process. > > While for the long term, we are willing to have more fine-grained lock > to provide higher scalability. > > This patch divides Locking Internal section based on these two global > locks to help readers to understand it. Also it adds some new finding to > enrich it. > > [David: words arrangement] > > Signed-off-by: Wei Yang <richard.weiyang@gmail.com> For a love of mine I cannot find the locking description by Oscar. Maybe it never existed and I just made it up ;) But if it is not imaginary then my recollection is that it was much more comprehensive. If not then even this is a good start. > --- > Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- > 1 file changed, 24 insertions(+), 3 deletions(-) > > diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst > index de7467e48067..95662b283328 100644 > --- a/Documentation/core-api/memory-hotplug.rst > +++ b/Documentation/core-api/memory-hotplug.rst > @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. > Locking Internals > ================= > > +There are three locks involved in memory-hotplug, two global lock and one local > +lock: > + > +- device_hotplug_lock > +- mem_hotplug_lock > +- device_lock > + > +Currently, they are twisted together for all kinds of reasons. The following > +part is divided into device_hotplug_lock and mem_hotplug_lock parts > +respectively to describe those tricky situations. > + > +device_hotplug_lock > +--------------------- > + > When adding/removing memory that uses memory block devices (i.e. ordinary RAM), > the device_hotplug_lock should be held to: > > @@ -111,13 +125,20 @@ As the device is visible to user space before taking the device_lock(), this > can result in a lock inversion. > > onlining/offlining of memory should be done via device_online()/ > -device_offline() - to make sure it is properly synchronized to actions > -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) > +device_offline() - to make sure it is properly synchronized to actions via > +sysfs. Even mem_hotplug_lock is used to protect the process, because of the > +lock inversion described above, holding device_hotplug_lock is still advised > +(to e.g. protect online_type) > + > +mem_hotplug_lock > +--------------------- > > When adding/removing/onlining/offlining memory or adding/removing > heterogeneous/device memory, we should always hold the mem_hotplug_lock in > write mode to serialise memory hotplug (e.g. access to global/zone > -variables). > +variables). Currently, we take advantage of this to serialise sparsemem's > +mem_section handling in sparse_add_one_section() and > +sparse_remove_one_section(). > > In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read > mode allows for a quite efficient get_online_mems/put_online_mems > -- > 2.15.1 >
On Wed, Dec 05, 2018 at 01:13:10PM +0100, Michal Hocko wrote: >On Wed 05-12-18 10:34:26, Wei Yang wrote: >> Currently locking for memory hotplug is a little complicated. >> >> Generally speaking, we leverage the two global lock: >> >> * device_hotplug_lock >> * mem_hotplug_lock >> >> to serialise the process. >> >> While for the long term, we are willing to have more fine-grained lock >> to provide higher scalability. >> >> This patch divides Locking Internal section based on these two global >> locks to help readers to understand it. Also it adds some new finding to >> enrich it. >> >> [David: words arrangement] >> >> Signed-off-by: Wei Yang <richard.weiyang@gmail.com> > >For a love of mine I cannot find the locking description by Oscar. Maybe >it never existed and I just made it up ;) But if it is not imaginary >then my recollection is that it was much more comprehensive. If not then >even this is a good start. Thanks. If Oscar has already has some work on it, this could be a complement to his work :-) > >> --- >> Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- >> 1 file changed, 24 insertions(+), 3 deletions(-) >> >> diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst >> index de7467e48067..95662b283328 100644 >> --- a/Documentation/core-api/memory-hotplug.rst >> +++ b/Documentation/core-api/memory-hotplug.rst >> @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. >> Locking Internals >> ================= >> >> +There are three locks involved in memory-hotplug, two global lock and one local >> +lock: >> + >> +- device_hotplug_lock >> +- mem_hotplug_lock >> +- device_lock >> + >> +Currently, they are twisted together for all kinds of reasons. The following >> +part is divided into device_hotplug_lock and mem_hotplug_lock parts >> +respectively to describe those tricky situations. >> + >> +device_hotplug_lock >> +--------------------- >> + >> When adding/removing memory that uses memory block devices (i.e. ordinary RAM), >> the device_hotplug_lock should be held to: >> >> @@ -111,13 +125,20 @@ As the device is visible to user space before taking the device_lock(), this >> can result in a lock inversion. >> >> onlining/offlining of memory should be done via device_online()/ >> -device_offline() - to make sure it is properly synchronized to actions >> -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) >> +device_offline() - to make sure it is properly synchronized to actions via >> +sysfs. Even mem_hotplug_lock is used to protect the process, because of the >> +lock inversion described above, holding device_hotplug_lock is still advised >> +(to e.g. protect online_type) >> + >> +mem_hotplug_lock >> +--------------------- >> >> When adding/removing/onlining/offlining memory or adding/removing >> heterogeneous/device memory, we should always hold the mem_hotplug_lock in >> write mode to serialise memory hotplug (e.g. access to global/zone >> -variables). >> +variables). Currently, we take advantage of this to serialise sparsemem's >> +mem_section handling in sparse_add_one_section() and >> +sparse_remove_one_section(). >> >> In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read >> mode allows for a quite efficient get_online_mems/put_online_mems >> -- >> 2.15.1 >> > >-- >Michal Hocko >SUSE Labs
diff --git a/Documentation/core-api/memory-hotplug.rst b/Documentation/core-api/memory-hotplug.rst index de7467e48067..95662b283328 100644 --- a/Documentation/core-api/memory-hotplug.rst +++ b/Documentation/core-api/memory-hotplug.rst @@ -89,6 +89,20 @@ NOTIFY_STOP stops further processing of the notification queue. Locking Internals ================= +There are three locks involved in memory-hotplug, two global lock and one local +lock: + +- device_hotplug_lock +- mem_hotplug_lock +- device_lock + +Currently, they are twisted together for all kinds of reasons. The following +part is divided into device_hotplug_lock and mem_hotplug_lock parts +respectively to describe those tricky situations. + +device_hotplug_lock +--------------------- + When adding/removing memory that uses memory block devices (i.e. ordinary RAM), the device_hotplug_lock should be held to: @@ -111,13 +125,20 @@ As the device is visible to user space before taking the device_lock(), this can result in a lock inversion. onlining/offlining of memory should be done via device_online()/ -device_offline() - to make sure it is properly synchronized to actions -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) +device_offline() - to make sure it is properly synchronized to actions via +sysfs. Even mem_hotplug_lock is used to protect the process, because of the +lock inversion described above, holding device_hotplug_lock is still advised +(to e.g. protect online_type) + +mem_hotplug_lock +--------------------- When adding/removing/onlining/offlining memory or adding/removing heterogeneous/device memory, we should always hold the mem_hotplug_lock in write mode to serialise memory hotplug (e.g. access to global/zone -variables). +variables). Currently, we take advantage of this to serialise sparsemem's +mem_section handling in sparse_add_one_section() and +sparse_remove_one_section(). In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read mode allows for a quite efficient get_online_mems/put_online_mems
Currently locking for memory hotplug is a little complicated. Generally speaking, we leverage the two global lock: * device_hotplug_lock * mem_hotplug_lock to serialise the process. While for the long term, we are willing to have more fine-grained lock to provide higher scalability. This patch divides Locking Internal section based on these two global locks to help readers to understand it. Also it adds some new finding to enrich it. [David: words arrangement] Signed-off-by: Wei Yang <richard.weiyang@gmail.com> --- Documentation/core-api/memory-hotplug.rst | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-)