| Message ID | 1460757887-18128-1-git-send-email-javier@osg.samsung.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi Javier, On 16 April 2016 at 03:34, Javier Martinez Canillas <javier@osg.samsung.com> wrote: > The exynos5 I2C controller driver always prepares and enables a clock > before using it and then disables unprepares it when the clock is not > used anymore. > > But this can cause a possible ABBA deadlock in some scenarios since a > driver that uses regmap to access its I2C registers, will first grab > the regmap lock and then the I2C xfer function will grab the prepare > lock when preparing the I2C clock. But since the clock driver also > uses regmap for I2C accesses, preparing a clock will first grab the > prepare lock and then the regmap lock when using the regmap API. > > An example of this happens on the Exynos5422 Odroid XU board where a > s2mps11 PMIC is used and both the s2mps11 regulators and clk drivers > share the same I2C regmap. > > The possible deadlock is reported by the kernel lockdep: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock(sec_core:428:(regmap)->lock); > lock(prepare_lock); > lock(sec_core:428:(regmap)->lock); > lock(prepare_lock); > > *** DEADLOCK *** > > Fix this by only preparing the clock on probe and {en,dis}able in the > rest of the driver. > > This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA > deadlock by keeping clock prepared") that fixes the same bug in other > driver for an I2C controller found in Samsung SoCs. > > Reported-by: Anand Moon <linux.amoon@gmail.com> > Signed-off-by: Javier Martinez Canillas <javier@osg.samsung.com> > > --- > > drivers/i2c/busses/i2c-exynos5.c | 20 +++++++++++++++----- > 1 file changed, 15 insertions(+), 5 deletions(-) > > diff --git a/drivers/i2c/busses/i2c-exynos5.c b/drivers/i2c/busses/i2c-exynos5.c > index b29c7500461a..602633747149 100644 > --- a/drivers/i2c/busses/i2c-exynos5.c > +++ b/drivers/i2c/busses/i2c-exynos5.c > @@ -671,7 +671,9 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, > return -EIO; > } > > - clk_prepare_enable(i2c->clk); > + ret = clk_enable(i2c->clk); > + if (ret) > + return ret; > > for (i = 0; i < num; i++, msgs++) { > stop = (i == num - 1); > @@ -695,7 +697,7 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, > } > > out: > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > return ret; > } > > @@ -799,6 +801,10 @@ static int exynos5_i2c_probe(struct platform_device *pdev) > > platform_set_drvdata(pdev, i2c); > > + clk_disable(i2c->clk); > + > + return 0; > + > err_clk: > clk_disable_unprepare(i2c->clk); > return ret; > @@ -810,6 +816,8 @@ static int exynos5_i2c_remove(struct platform_device *pdev) > > i2c_del_adapter(&i2c->adap); > > + clk_unprepare(i2c->clk); > + > return 0; > } > > @@ -830,16 +838,18 @@ static int exynos5_i2c_resume_noirq(struct device *dev) > struct exynos5_i2c *i2c = platform_get_drvdata(pdev); > int ret = 0; > > - clk_prepare_enable(i2c->clk); > + ret = clk_enable(i2c->clk); > + if (ret) > + return ret; > > ret = exynos5_hsi2c_clock_setup(i2c); > if (ret) { > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > return ret; > } > > exynos5_i2c_init(i2c); > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > i2c->suspended = 0; > > return 0; > -- > 2.5.5 > Thanks for fixing this bug. I also tried the same thing on clk, but missed the return on failed. Reviewed-by: Anand Moon <linux.amoon@gmail.com> Tested-by: Anand Moon <linux.amoon@gmail.com> Best Regards -Anand Moon
On Fri, Apr 15, 2016 at 06:04:47PM -0400, Javier Martinez Canillas wrote: > The exynos5 I2C controller driver always prepares and enables a clock > before using it and then disables unprepares it when the clock is not > used anymore. > > But this can cause a possible ABBA deadlock in some scenarios since a > driver that uses regmap to access its I2C registers, will first grab > the regmap lock and then the I2C xfer function will grab the prepare > lock when preparing the I2C clock. But since the clock driver also > uses regmap for I2C accesses, preparing a clock will first grab the > prepare lock and then the regmap lock when using the regmap API. > > An example of this happens on the Exynos5422 Odroid XU board where a > s2mps11 PMIC is used and both the s2mps11 regulators and clk drivers > share the same I2C regmap. > > The possible deadlock is reported by the kernel lockdep: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock(sec_core:428:(regmap)->lock); > lock(prepare_lock); > lock(sec_core:428:(regmap)->lock); > lock(prepare_lock); > > *** DEADLOCK *** > > Fix this by only preparing the clock on probe and {en,dis}able in the > rest of the driver. > > This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA > deadlock by keeping clock prepared") that fixes the same bug in other > driver for an I2C controller found in Samsung SoCs. I wish this would be fixed by introducing more granular clock locks (e.g. per controller) instead of implementing another workaround. I think this driver shouldn't care about this deadlock... although I see that this is the simplest solution for now. > > Reported-by: Anand Moon <linux.amoon@gmail.com> > Signed-off-by: Javier Martinez Canillas <javier@osg.samsung.com> > > --- > > drivers/i2c/busses/i2c-exynos5.c | 20 +++++++++++++++----- > 1 file changed, 15 insertions(+), 5 deletions(-) > > diff --git a/drivers/i2c/busses/i2c-exynos5.c b/drivers/i2c/busses/i2c-exynos5.c > index b29c7500461a..602633747149 100644 > --- a/drivers/i2c/busses/i2c-exynos5.c > +++ b/drivers/i2c/busses/i2c-exynos5.c > @@ -671,7 +671,9 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, > return -EIO; > } > > - clk_prepare_enable(i2c->clk); > + ret = clk_enable(i2c->clk); > + if (ret) > + return ret; > > for (i = 0; i < num; i++, msgs++) { > stop = (i == num - 1); > @@ -695,7 +697,7 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, > } > > out: > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > return ret; > } > > @@ -799,6 +801,10 @@ static int exynos5_i2c_probe(struct platform_device *pdev) > > platform_set_drvdata(pdev, i2c); > > + clk_disable(i2c->clk); > + > + return 0; > + > err_clk: > clk_disable_unprepare(i2c->clk); > return ret; > @@ -810,6 +816,8 @@ static int exynos5_i2c_remove(struct platform_device *pdev) > > i2c_del_adapter(&i2c->adap); > > + clk_unprepare(i2c->clk); > + > return 0; > } Please unprepare the clock when suspending. There is no point of having it prepared in that level. Best regards, Krzysztof
Hello Anand, On 04/16/2016 08:15 AM, Anand Moon wrote: [snip] >> > > Thanks for fixing this bug. I also tried the same thing on clk, but > missed the return on failed. > > Reviewed-by: Anand Moon <linux.amoon@gmail.com> > Tested-by: Anand Moon <linux.amoon@gmail.com> > Thanks a lot for your testing and review. > Best Regards > -Anand Moon > Best regards,
Hello Krzysztof, Thanks a lot for your feedback. On 04/16/2016 12:11 PM, Krzysztof Kozlowski wrote: > On Fri, Apr 15, 2016 at 06:04:47PM -0400, Javier Martinez Canillas wrote: [snip] >> >> Fix this by only preparing the clock on probe and {en,dis}able in the >> rest of the driver. >> >> This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA >> deadlock by keeping clock prepared") that fixes the same bug in other >> driver for an I2C controller found in Samsung SoCs. > > I wish this would be fixed by introducing more granular clock locks > (e.g. per controller) instead of implementing another workaround. > I think this driver shouldn't care about this deadlock... although I see > that this is the simplest solution for now. > Agreed, but that would be a much intrusive core change affecting every single platform so I didn't feel brave enough to attempt it :) But regardless of the ABBA deadlock, there are reasons why the clk API is split into an {,un}prepare and {en,dis}able functions (e.g: non-atomic vs atomic) and it is a common pattern for drivers to prepare the clock(s) on setup (i.e: probe), unprepare on driver removal, and just {en,dis}able the clock(s) during runtime. So I believe this patch is good on its own and at least makes the driver more consistent with most I2C controller drivers that do the same w.r.t clocks. The fact that the deadlock is fixed by this change is just a nice side effect IMHO. [snip] >> @@ -810,6 +816,8 @@ static int exynos5_i2c_remove(struct platform_device *pdev) >> >> i2c_del_adapter(&i2c->adap); >> >> + clk_unprepare(i2c->clk); >> + >> return 0; >> } > > Please unprepare the clock when suspending. There is no point of having > it prepared in that level. > Yes, I in fact thought the same when writing the patch but was reluctant to change the prepared state in suspend because I don't have a way to test the S2R path in this board due broken firmware that prevents the cores to enter into deep sleep states (I believe is the same issue we faced with CPUidle). But I'll do the change that you suggested since I agree with you. > Best regards, > Krzysztof > Best regards,
On Sat, Apr 16, 2016 at 08:58:49PM -0400, Javier Martinez Canillas wrote: > But regardless of the ABBA deadlock, there are reasons why the clk API is > split into an {,un}prepare and {en,dis}able functions (e.g: non-atomic vs > atomic) and it is a common pattern for drivers to prepare the clock(s) on > setup (i.e: probe), unprepare on driver removal, and just {en,dis}able the > clock(s) during runtime. The reason to split prepare+enable is to differentiate the sleeping and non-sleeping paths. Not to encourage doing one type in probe() and rest in other places. The clock consumer does not care whether prepare or enable is a noop. It shouldn't care, because the clock provider might change and consumer remains the same. On the other hand the consumer is interested whether it can block or not. The argument that common pattern is to prepare in probe does not convince me. If a clock enable is an noop and everything is done in prepare/unprepare, then following such "common pattern" would be equal to keeping the clock on always, even if it could be unprepared (gated). > So I believe this patch is good on its own As I said above, no. The pattern might be misleading. The prepare path is there for sleeping purposes. Not for probes. Choosing such pattern effectively makes clock handling worthless in certain clock providers (these which enable is a noop). > and at least makes the driver more > consistent with most I2C controller drivers that do the same w.r.t clocks. I cannot respond to that argument because I do not know most I2C controllers. Best regards, Krzysztof
Hello, On 2016-04-16 00:04, Javier Martinez Canillas wrote: > The exynos5 I2C controller driver always prepares and enables a clock > before using it and then disables unprepares it when the clock is not > used anymore. > > But this can cause a possible ABBA deadlock in some scenarios since a > driver that uses regmap to access its I2C registers, will first grab > the regmap lock and then the I2C xfer function will grab the prepare > lock when preparing the I2C clock. But since the clock driver also > uses regmap for I2C accesses, preparing a clock will first grab the > prepare lock and then the regmap lock when using the regmap API. > > An example of this happens on the Exynos5422 Odroid XU board where a > s2mps11 PMIC is used and both the s2mps11 regulators and clk drivers > share the same I2C regmap. > > The possible deadlock is reported by the kernel lockdep: > > Possible unsafe locking scenario: > > CPU0 CPU1 > ---- ---- > lock(sec_core:428:(regmap)->lock); > lock(prepare_lock); > lock(sec_core:428:(regmap)->lock); > lock(prepare_lock); > > *** DEADLOCK *** > > Fix this by only preparing the clock on probe and {en,dis}able in the > rest of the driver. > > This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA > deadlock by keeping clock prepared") that fixes the same bug in other > driver for an I2C controller found in Samsung SoCs. I'm sorry, but this is not the right approach imho. It is just a workaround applied to specific driver, it also duplicates incorrect clock usage pattern (there is really no point keeping clock prepared all the time). IMHO this ABBA deadlock should be really fixed in clocks core (probably by removing global prepare mutex and replacing it with per clock controller mutexes). Without a proper patch this issue will hit us again with other i2c controllers or other drivers as well. > Reported-by: Anand Moon <linux.amoon@gmail.com> > Signed-off-by: Javier Martinez Canillas <javier@osg.samsung.com> > > --- > > drivers/i2c/busses/i2c-exynos5.c | 20 +++++++++++++++----- > 1 file changed, 15 insertions(+), 5 deletions(-) > > diff --git a/drivers/i2c/busses/i2c-exynos5.c b/drivers/i2c/busses/i2c-exynos5.c > index b29c7500461a..602633747149 100644 > --- a/drivers/i2c/busses/i2c-exynos5.c > +++ b/drivers/i2c/busses/i2c-exynos5.c > @@ -671,7 +671,9 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, > return -EIO; > } > > - clk_prepare_enable(i2c->clk); > + ret = clk_enable(i2c->clk); > + if (ret) > + return ret; > > for (i = 0; i < num; i++, msgs++) { > stop = (i == num - 1); > @@ -695,7 +697,7 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, > } > > out: > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > return ret; > } > > @@ -799,6 +801,10 @@ static int exynos5_i2c_probe(struct platform_device *pdev) > > platform_set_drvdata(pdev, i2c); > > + clk_disable(i2c->clk); > + > + return 0; > + > err_clk: > clk_disable_unprepare(i2c->clk); > return ret; > @@ -810,6 +816,8 @@ static int exynos5_i2c_remove(struct platform_device *pdev) > > i2c_del_adapter(&i2c->adap); > > + clk_unprepare(i2c->clk); > + > return 0; > } > > @@ -830,16 +838,18 @@ static int exynos5_i2c_resume_noirq(struct device *dev) > struct exynos5_i2c *i2c = platform_get_drvdata(pdev); > int ret = 0; > > - clk_prepare_enable(i2c->clk); > + ret = clk_enable(i2c->clk); > + if (ret) > + return ret; > > ret = exynos5_hsi2c_clock_setup(i2c); > if (ret) { > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > return ret; > } > > exynos5_i2c_init(i2c); > - clk_disable_unprepare(i2c->clk); > + clk_disable(i2c->clk); > i2c->suspended = 0; > > return 0; Best regards
Hello Marek, On 04/18/2016 03:50 AM, Marek Szyprowski wrote: > Hello, > > On 2016-04-16 00:04, Javier Martinez Canillas wrote: >> The exynos5 I2C controller driver always prepares and enables a clock >> before using it and then disables unprepares it when the clock is not >> used anymore. >> >> But this can cause a possible ABBA deadlock in some scenarios since a >> driver that uses regmap to access its I2C registers, will first grab >> the regmap lock and then the I2C xfer function will grab the prepare >> lock when preparing the I2C clock. But since the clock driver also >> uses regmap for I2C accesses, preparing a clock will first grab the >> prepare lock and then the regmap lock when using the regmap API. >> >> An example of this happens on the Exynos5422 Odroid XU board where a >> s2mps11 PMIC is used and both the s2mps11 regulators and clk drivers >> share the same I2C regmap. >> >> The possible deadlock is reported by the kernel lockdep: >> >> Possible unsafe locking scenario: >> >> CPU0 CPU1 >> ---- ---- >> lock(sec_core:428:(regmap)->lock); >> lock(prepare_lock); >> lock(sec_core:428:(regmap)->lock); >> lock(prepare_lock); >> >> *** DEADLOCK *** >> >> Fix this by only preparing the clock on probe and {en,dis}able in the >> rest of the driver. >> >> This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA >> deadlock by keeping clock prepared") that fixes the same bug in other >> driver for an I2C controller found in Samsung SoCs. > > I'm sorry, but this is not the right approach imho. It is just a workaround > applied to specific driver, it also duplicates incorrect clock usage > pattern (there is really no point keeping clock prepared all the time). > > IMHO this ABBA deadlock should be really fixed in clocks core (probably > by removing global prepare mutex and replacing it with per clock > controller mutexes). Without a proper patch this issue will hit us again > with other i2c controllers or other drivers as well. > Agreed, Krzysztof mentioned the same before. I'll take a look to the global mutex to see how I can make it more fine grained to prevent these deadlocks. >> Reported-by: Anand Moon <linux.amoon@gmail.com> >> Signed-off-by: Javier Martinez Canillas <javier@osg.samsung.com> >> Best regards,
[adding clk maintainers/list to cc] On 04/18/2016 09:29 AM, Javier Martinez Canillas wrote: > Hello Marek, > > On 04/18/2016 03:50 AM, Marek Szyprowski wrote: >> Hello, >> >> On 2016-04-16 00:04, Javier Martinez Canillas wrote: >>> The exynos5 I2C controller driver always prepares and enables a clock >>> before using it and then disables unprepares it when the clock is not >>> used anymore. >>> >>> But this can cause a possible ABBA deadlock in some scenarios since a >>> driver that uses regmap to access its I2C registers, will first grab >>> the regmap lock and then the I2C xfer function will grab the prepare >>> lock when preparing the I2C clock. But since the clock driver also >>> uses regmap for I2C accesses, preparing a clock will first grab the >>> prepare lock and then the regmap lock when using the regmap API. >>> >>> An example of this happens on the Exynos5422 Odroid XU board where a >>> s2mps11 PMIC is used and both the s2mps11 regulators and clk drivers >>> share the same I2C regmap. >>> >>> The possible deadlock is reported by the kernel lockdep: >>> >>> Possible unsafe locking scenario: >>> >>> CPU0 CPU1 >>> ---- ---- >>> lock(sec_core:428:(regmap)->lock); >>> lock(prepare_lock); >>> lock(sec_core:428:(regmap)->lock); >>> lock(prepare_lock); >>> >>> *** DEADLOCK *** >>> >>> Fix this by only preparing the clock on probe and {en,dis}able in the >>> rest of the driver. >>> >>> This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA >>> deadlock by keeping clock prepared") that fixes the same bug in other >>> driver for an I2C controller found in Samsung SoCs. >> >> I'm sorry, but this is not the right approach imho. It is just a workaround >> applied to specific driver, it also duplicates incorrect clock usage >> pattern (there is really no point keeping clock prepared all the time). >> >> IMHO this ABBA deadlock should be really fixed in clocks core (probably >> by removing global prepare mutex and replacing it with per clock >> controller mutexes). Without a proper patch this issue will hit us again >> with other i2c controllers or other drivers as well. >> > > Agreed, Krzysztof mentioned the same before. I'll take a look to the global > mutex to see how I can make it more fine grained to prevent these deadlocks. > I've been looking at the Common Clk Framework and it seems that removing the global prepare_lock mutex and making more fine grained (i.e: per controller) is not trivial since clocks prepare operations are propagated to the parents so the lock is not just used to have mutual exclusion of the clocks in the same controller but is to prevent concurrent access to the whole hierarchy. Also, I wonder why the clock usage pattern of leaving the clock prepared and just {en,dis}able the clock at runtime is that bad since many drivers do it? In fact, most I2C controllers drivers manage their clocks that way and that is why I thought that this patch had merits on its own and the fact that the deadlock was fixed was just a nice side effect of $SUBJECT. Finally, Krzysztof was able to hit the deadlock so this is not just about a theoretical deadlock but an important bug that should be fixed and $SUBJECT is a possible fix until the CCF gets rids of the global mutex IMHO. Stephen, Mike: Some context has already been removed in the thread (including the patch diff) so you can find the original patch here if you need more context: https://lkml.org/lkml/2016/4/15/900 Best regards,
diff --git a/drivers/i2c/busses/i2c-exynos5.c b/drivers/i2c/busses/i2c-exynos5.c index b29c7500461a..602633747149 100644 --- a/drivers/i2c/busses/i2c-exynos5.c +++ b/drivers/i2c/busses/i2c-exynos5.c @@ -671,7 +671,9 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, return -EIO; } - clk_prepare_enable(i2c->clk); + ret = clk_enable(i2c->clk); + if (ret) + return ret; for (i = 0; i < num; i++, msgs++) { stop = (i == num - 1); @@ -695,7 +697,7 @@ static int exynos5_i2c_xfer(struct i2c_adapter *adap, } out: - clk_disable_unprepare(i2c->clk); + clk_disable(i2c->clk); return ret; } @@ -799,6 +801,10 @@ static int exynos5_i2c_probe(struct platform_device *pdev) platform_set_drvdata(pdev, i2c); + clk_disable(i2c->clk); + + return 0; + err_clk: clk_disable_unprepare(i2c->clk); return ret; @@ -810,6 +816,8 @@ static int exynos5_i2c_remove(struct platform_device *pdev) i2c_del_adapter(&i2c->adap); + clk_unprepare(i2c->clk); + return 0; } @@ -830,16 +838,18 @@ static int exynos5_i2c_resume_noirq(struct device *dev) struct exynos5_i2c *i2c = platform_get_drvdata(pdev); int ret = 0; - clk_prepare_enable(i2c->clk); + ret = clk_enable(i2c->clk); + if (ret) + return ret; ret = exynos5_hsi2c_clock_setup(i2c); if (ret) { - clk_disable_unprepare(i2c->clk); + clk_disable(i2c->clk); return ret; } exynos5_i2c_init(i2c); - clk_disable_unprepare(i2c->clk); + clk_disable(i2c->clk); i2c->suspended = 0; return 0;
The exynos5 I2C controller driver always prepares and enables a clock before using it and then disables unprepares it when the clock is not used anymore. But this can cause a possible ABBA deadlock in some scenarios since a driver that uses regmap to access its I2C registers, will first grab the regmap lock and then the I2C xfer function will grab the prepare lock when preparing the I2C clock. But since the clock driver also uses regmap for I2C accesses, preparing a clock will first grab the prepare lock and then the regmap lock when using the regmap API. An example of this happens on the Exynos5422 Odroid XU board where a s2mps11 PMIC is used and both the s2mps11 regulators and clk drivers share the same I2C regmap. The possible deadlock is reported by the kernel lockdep: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(sec_core:428:(regmap)->lock); lock(prepare_lock); lock(sec_core:428:(regmap)->lock); lock(prepare_lock); *** DEADLOCK *** Fix this by only preparing the clock on probe and {en,dis}able in the rest of the driver. This patch is similar to commit 34e81ad5f0b6 ("i2c: s3c2410: fix ABBA deadlock by keeping clock prepared") that fixes the same bug in other driver for an I2C controller found in Samsung SoCs. Reported-by: Anand Moon <linux.amoon@gmail.com> Signed-off-by: Javier Martinez Canillas <javier@osg.samsung.com> --- drivers/i2c/busses/i2c-exynos5.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)