| Message ID | 20200303015859.18813-1-dongli.zhang@oracle.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [1/2] xenbus: req->body should be updated before req->state | expand |
Hi, On 03/03/2020 01:58, Dongli Zhang wrote: > The req->body should be updated before req->state is updated and the > order should be guaranteed by a barrier. > > Otherwise, read_reply() might return req->body = NULL. > > Below is sample callstack when the issue is reproduced on purpose by > reordering the updates of req->body and req->state and adding delay in > code between updates of req->state and req->body. > > [ 22.356105] general protection fault: 0000 [#1] SMP PTI > [ 22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6 > [ 22.366727] Hardware name: Xen HVM domU, BIOS ... > [ 22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 > ... ... > [ 22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246 > [ 22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000 > [ 22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d > [ 22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd > [ 22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000 > [ 22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000 > [ 22.423726] FS: 0000000000000000(0000) GS:ffff9d953bc80000(0000) knlGS:0000000000000000 > [ 22.429898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0 > [ 22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 22.450342] Call Trace: > [ 22.452509] simple_strtoull+0x27/0x70 > [ 22.455572] xenbus_transaction_start+0x31/0x50 > [ 22.459104] netback_changed+0x76c/0xcc1 [xen_netfront] > [ 22.463279] ? find_watch+0x40/0x40 > [ 22.466156] xenwatch_thread+0xb4/0x150 > [ 22.469309] ? wait_woken+0x80/0x80 > [ 22.472198] kthread+0x10e/0x130 > [ 22.474925] ? kthread_park+0x80/0x80 > [ 22.477946] ret_from_fork+0x35/0x40 > [ 22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront xen_blkfront > [ 22.486783] ---[ end trace a9222030a747c3f7 ]--- > [ 22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 > > The "while" is changed to "do while" so that wait_event() is used as a > barrier. The correct barrier for read_reply() should be virt_rmb(). While on x86, this is equivalent to barrier(), on Arm this will be a dmb(ish) to prevent the processor re-ordering memory access. Therefore the barrier in test_reply() (called by wait_event()) is not going to be sufficient for Arm. > > Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com> > --- > drivers/xen/xenbus/xenbus_comms.c | 2 ++ > drivers/xen/xenbus/xenbus_xs.c | 4 ++-- > 2 files changed, 4 insertions(+), 2 deletions(-) > > diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c > index d239fc3c5e3d..852ed161fc2a 100644 > --- a/drivers/xen/xenbus/xenbus_comms.c > +++ b/drivers/xen/xenbus/xenbus_comms.c > @@ -313,6 +313,8 @@ static int process_msg(void) > req->msg.type = state.msg.type; > req->msg.len = state.msg.len; > req->body = state.body; > + /* write body, then update state */ > + virt_wmb(); > req->state = xb_req_state_got_reply; > req->cb(req); > } else > diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c > index ddc18da61834..f5b0a6a72ad3 100644 > --- a/drivers/xen/xenbus/xenbus_xs.c > +++ b/drivers/xen/xenbus/xenbus_xs.c > @@ -202,7 +202,7 @@ static bool test_reply(struct xb_req_data *req) > > static void *read_reply(struct xb_req_data *req) > { > - while (req->state != xb_req_state_got_reply) { > + do { > wait_event(req->wq, test_reply(req)); > > if (!xenbus_ok()) > @@ -216,7 +216,7 @@ static void *read_reply(struct xb_req_data *req) > if (req->err) > return ERR_PTR(req->err); > > - } > + } while (req->state != xb_req_state_got_reply); > > return req->body; > } > Cheers,
On 3/3/20 1:40 AM, Julien Grall wrote: > Hi, > > On 03/03/2020 01:58, Dongli Zhang wrote: >> The req->body should be updated before req->state is updated and the >> order should be guaranteed by a barrier. >> >> Otherwise, read_reply() might return req->body = NULL. >> >> Below is sample callstack when the issue is reproduced on purpose by >> reordering the updates of req->body and req->state and adding delay in >> code between updates of req->state and req->body. >> >> [ 22.356105] general protection fault: 0000 [#1] SMP PTI >> [ 22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6 >> [ 22.366727] Hardware name: Xen HVM domU, BIOS ... >> [ 22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 >> ... ... >> [ 22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246 >> [ 22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000 >> [ 22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d >> [ 22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd >> [ 22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000 >> [ 22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000 >> [ 22.423726] FS: 0000000000000000(0000) GS:ffff9d953bc80000(0000) >> knlGS:0000000000000000 >> [ 22.429898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0 >> [ 22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [ 22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> [ 22.450342] Call Trace: >> [ 22.452509] simple_strtoull+0x27/0x70 >> [ 22.455572] xenbus_transaction_start+0x31/0x50 >> [ 22.459104] netback_changed+0x76c/0xcc1 [xen_netfront] >> [ 22.463279] ? find_watch+0x40/0x40 >> [ 22.466156] xenwatch_thread+0xb4/0x150 >> [ 22.469309] ? wait_woken+0x80/0x80 >> [ 22.472198] kthread+0x10e/0x130 >> [ 22.474925] ? kthread_park+0x80/0x80 >> [ 22.477946] ret_from_fork+0x35/0x40 >> [ 22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront >> xen_blkfront >> [ 22.486783] ---[ end trace a9222030a747c3f7 ]--- >> [ 22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 >> >> The "while" is changed to "do while" so that wait_event() is used as a >> barrier. > > The correct barrier for read_reply() should be virt_rmb(). While on x86, this is > equivalent to barrier(), on Arm this will be a dmb(ish) to prevent the processor > re-ordering memory access. > > Therefore the barrier in test_reply() (called by wait_event()) is not going to > be sufficient for Arm. Sorry that I just erroneously thought wait_event() would be used as read barrier. I would change barrier() to virt_rmb() for read_reply() in v2. Thank you very much! Dongli Zhang
Hi, On 03/03/2020 17:36, dongli.zhang@oracle.com wrote: > > > On 3/3/20 1:40 AM, Julien Grall wrote: >> Hi, >> >> On 03/03/2020 01:58, Dongli Zhang wrote: >>> The req->body should be updated before req->state is updated and the >>> order should be guaranteed by a barrier. >>> >>> Otherwise, read_reply() might return req->body = NULL. >>> >>> Below is sample callstack when the issue is reproduced on purpose by >>> reordering the updates of req->body and req->state and adding delay in >>> code between updates of req->state and req->body. >>> >>> [ 22.356105] general protection fault: 0000 [#1] SMP PTI >>> [ 22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6 >>> [ 22.366727] Hardware name: Xen HVM domU, BIOS ... >>> [ 22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 >>> ... ... >>> [ 22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246 >>> [ 22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000 >>> [ 22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d >>> [ 22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd >>> [ 22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000 >>> [ 22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000 >>> [ 22.423726] FS: 0000000000000000(0000) GS:ffff9d953bc80000(0000) >>> knlGS:0000000000000000 >>> [ 22.429898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0 >>> [ 22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>> [ 22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >>> [ 22.450342] Call Trace: >>> [ 22.452509] simple_strtoull+0x27/0x70 >>> [ 22.455572] xenbus_transaction_start+0x31/0x50 >>> [ 22.459104] netback_changed+0x76c/0xcc1 [xen_netfront] >>> [ 22.463279] ? find_watch+0x40/0x40 >>> [ 22.466156] xenwatch_thread+0xb4/0x150 >>> [ 22.469309] ? wait_woken+0x80/0x80 >>> [ 22.472198] kthread+0x10e/0x130 >>> [ 22.474925] ? kthread_park+0x80/0x80 >>> [ 22.477946] ret_from_fork+0x35/0x40 >>> [ 22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront >>> xen_blkfront >>> [ 22.486783] ---[ end trace a9222030a747c3f7 ]--- >>> [ 22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 >>> >>> The "while" is changed to "do while" so that wait_event() is used as a >>> barrier. >> >> The correct barrier for read_reply() should be virt_rmb(). While on x86, this is >> equivalent to barrier(), on Arm this will be a dmb(ish) to prevent the processor >> re-ordering memory access. >> >> Therefore the barrier in test_reply() (called by wait_event()) is not going to >> be sufficient for Arm. > > Sorry that I just erroneously thought wait_event() would be used as read barrier. I was also kind of expecting wait_event() to contain a memory barrier. But it does not at least if condition is valid before waiting. Cheers,
diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c index d239fc3c5e3d..852ed161fc2a 100644 --- a/drivers/xen/xenbus/xenbus_comms.c +++ b/drivers/xen/xenbus/xenbus_comms.c @@ -313,6 +313,8 @@ static int process_msg(void) req->msg.type = state.msg.type; req->msg.len = state.msg.len; req->body = state.body; + /* write body, then update state */ + virt_wmb(); req->state = xb_req_state_got_reply; req->cb(req); } else diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c index ddc18da61834..f5b0a6a72ad3 100644 --- a/drivers/xen/xenbus/xenbus_xs.c +++ b/drivers/xen/xenbus/xenbus_xs.c @@ -202,7 +202,7 @@ static bool test_reply(struct xb_req_data *req) static void *read_reply(struct xb_req_data *req) { - while (req->state != xb_req_state_got_reply) { + do { wait_event(req->wq, test_reply(req)); if (!xenbus_ok()) @@ -216,7 +216,7 @@ static void *read_reply(struct xb_req_data *req) if (req->err) return ERR_PTR(req->err); - } + } while (req->state != xb_req_state_got_reply); return req->body; }
The req->body should be updated before req->state is updated and the order should be guaranteed by a barrier. Otherwise, read_reply() might return req->body = NULL. Below is sample callstack when the issue is reproduced on purpose by reordering the updates of req->body and req->state and adding delay in code between updates of req->state and req->body. [ 22.356105] general protection fault: 0000 [#1] SMP PTI [ 22.361185] CPU: 2 PID: 52 Comm: xenwatch Not tainted 5.5.0xen+ #6 [ 22.366727] Hardware name: Xen HVM domU, BIOS ... [ 22.372245] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 ... ... [ 22.392163] RSP: 0018:ffffb2d64023fdf0 EFLAGS: 00010246 [ 22.395933] RAX: 0000000000000000 RBX: 75746e7562755f6d RCX: 0000000000000000 [ 22.400871] RDX: 0000000000000000 RSI: ffffb2d64023fdfc RDI: 75746e7562755f6d [ 22.405874] RBP: 0000000000000000 R08: 00000000000001e8 R09: 0000000000cdcdcd [ 22.410945] R10: ffffb2d6402ffe00 R11: ffff9d95395eaeb0 R12: ffff9d9535935000 [ 22.417613] R13: ffff9d9526d4a000 R14: ffff9d9526f4f340 R15: ffff9d9537654000 [ 22.423726] FS: 0000000000000000(0000) GS:ffff9d953bc80000(0000) knlGS:0000000000000000 [ 22.429898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 22.434342] CR2: 000000c4206a9000 CR3: 00000001ea3fc002 CR4: 00000000001606e0 [ 22.439645] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 22.444941] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 22.450342] Call Trace: [ 22.452509] simple_strtoull+0x27/0x70 [ 22.455572] xenbus_transaction_start+0x31/0x50 [ 22.459104] netback_changed+0x76c/0xcc1 [xen_netfront] [ 22.463279] ? find_watch+0x40/0x40 [ 22.466156] xenwatch_thread+0xb4/0x150 [ 22.469309] ? wait_woken+0x80/0x80 [ 22.472198] kthread+0x10e/0x130 [ 22.474925] ? kthread_park+0x80/0x80 [ 22.477946] ret_from_fork+0x35/0x40 [ 22.480968] Modules linked in: xen_kbdfront xen_fbfront(+) xen_netfront xen_blkfront [ 22.486783] ---[ end trace a9222030a747c3f7 ]--- [ 22.490424] RIP: 0010:_parse_integer_fixup_radix+0x6/0x60 The "while" is changed to "do while" so that wait_event() is used as a barrier. Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com> --- drivers/xen/xenbus/xenbus_comms.c | 2 ++ drivers/xen/xenbus/xenbus_xs.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)