| Message ID | 20221128002043.1555543-3-mizhang@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Deprecate BUG() in pte_list_remove() in shadow mmu | expand |
On Mon, Nov 28, 2022, Mingwei Zhang wrote: > Replace BUG() in pte_list_remove() with KVM_BUG() to avoid crashing the > host. MMU bug is difficult to discover due to various racing conditions and > corner cases and thus it extremely hard to debug. The situation gets much > worse when it triggers the shutdown of a host. Host machine crash > eliminates everything including the potential clues for debugging. > > From cloud computing service perspective, BUG() or BUG_ON() is probably no > longer appropriate as the host reliability is top priority. I don't think we need to bring "cloud computing" into this. Linus has made it clear over and over and over that BUG() / BUG_ON() need to be avoided unless the alternative is worse. E.g. the BUG() in __handle_changed_spte() is warranted because the alternative is silent corruption of guest data. > Crashing the physical machine is almost never a good option as it eliminates > innocent VMs and cause service outage in a larger scope. Even worse, if > attacker can reliably triggers this code by diverting the control flow or > corrupting the memory, Or if there's a KVM bug, which is waaaaay more likely. > then this becomes vm-of-death attack. This is true of any BUG(), and really of any unexpected fault while holding a spinlock, e.g. NULL pointer derefs in the MMU are almost always fatal as well. > This is a huge attack vector to cloud providers, as the death of one single > host machine is not the end of the story. Without manual interferences, a > failed cloud job may be dispatched to other hosts and continue host crashes > until all of them are dead. > > Because of the above reasons, shrink the scope of crash to the target VM > only. KVM_BUG() and KVM_BUG_ON() requires a valid struct kvm which requires > extra plumbing. Avoid it in this version by just using > kvm_get_running_vcpu()->kvm instead. Stale comment. > Cc: Nagareddy Reddy <nspreddy@google.com> > Cc: Jim Mattson <jmattson@google.com> > Cc: David Matlack <dmatlack@google.com> > Signed-off-by: Mingwei Zhang <mizhang@google.com> > --- > arch/x86/kvm/mmu/mmu.c | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c > index b5a44b8f5f7b..e132d82ab4c0 100644 > --- a/arch/x86/kvm/mmu/mmu.c > +++ b/arch/x86/kvm/mmu/mmu.c > @@ -956,12 +956,12 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, > > if (!rmap_head->val) { > pr_err("%s: %p 0->BUG\n", __func__, spte); These probably need to be ratelimited (or "once"). Bugging the VM will prevent doing anything useful with the VM, but KVM still needs to destroy the VM, which means zapping SPTEs and purging the rmaps. Theoretically, there could be thousands of broken rmaps. > - BUG(); > + KVM_BUG(true, kvm, ""); If you don't want to provide a message, use KVM_BUG_ON(), not an empty message. Though my vote would be to fold the existing pr_err() messages into KVM_BUG(), which would make the WARN much more helpful and would address the pr_err() issue above. The __func__ printing can also go away in that case because the stack track will provide all the necessary info. The only reason not to drop the pr_err() entirely is if a ratelimited message is helpful for debugging failures that occur in production, which I doubt it true. And rather than pass "true", wrap the actual check with the KVM_BUG(). > } else if (!(rmap_head->val & 1)) { > rmap_printk("%p 1->0\n", spte); > if ((u64 *)rmap_head->val != spte) { > pr_err("%s: %p 1->BUG\n", __func__, spte); > - BUG(); > + KVM_BUG(true, kvm, ""); KVM needs to return here, otherwise KVM is knowingly writing a garbage pointer, e.g. will corrupt memory or trigger a fault. > } > rmap_head->val = 0; > } else { Something like this? --- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b5a44b8f5f7b..12790ccb8731 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -954,15 +954,16 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, struct pte_list_desc *prev_desc; int i; - if (!rmap_head->val) { - pr_err("%s: %p 0->BUG\n", __func__, spte); - BUG(); - } else if (!(rmap_head->val & 1)) { + if (KVM_BUG(!rmap_head->val, kvm, "rmap for %p is empty", spte)) + return; + + if (!(rmap_head->val & 1)) { rmap_printk("%p 1->0\n", spte); - if ((u64 *)rmap_head->val != spte) { - pr_err("%s: %p 1->BUG\n", __func__, spte); - BUG(); - } + + if (KVM_BUG((u64 *)rmap_head->val != spte, kvm, + "single rmap for %p doesn't match", spte)) + return; + rmap_head->val = 0; } else { rmap_printk("%p many->many\n", spte); @@ -979,8 +980,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, prev_desc = desc; desc = desc->more; } - pr_err("%s: %p many->many\n", __func__, spte); - BUG(); + KVM_BUG(true, kvm, "no rmap for %p (many->many)", spte); } } base-commit: d74237e747db7f9f27e821e6683d58185e846378 --
> Something like this? > > --- > arch/x86/kvm/mmu/mmu.c | 20 ++++++++++---------- > 1 file changed, 10 insertions(+), 10 deletions(-) > > diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c > index b5a44b8f5f7b..12790ccb8731 100644 > --- a/arch/x86/kvm/mmu/mmu.c > +++ b/arch/x86/kvm/mmu/mmu.c > @@ -954,15 +954,16 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, > struct pte_list_desc *prev_desc; > int i; > > - if (!rmap_head->val) { > - pr_err("%s: %p 0->BUG\n", __func__, spte); > - BUG(); > - } else if (!(rmap_head->val & 1)) { > + if (KVM_BUG(!rmap_head->val, kvm, "rmap for %p is empty", spte)) > + return; > + > + if (!(rmap_head->val & 1)) { > rmap_printk("%p 1->0\n", spte); > - if ((u64 *)rmap_head->val != spte) { > - pr_err("%s: %p 1->BUG\n", __func__, spte); > - BUG(); > - } > + > + if (KVM_BUG((u64 *)rmap_head->val != spte, kvm, > + "single rmap for %p doesn't match", spte)) > + return; > + > rmap_head->val = 0; > } else { > rmap_printk("%p many->many\n", spte); > @@ -979,8 +980,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, > prev_desc = desc; > desc = desc->more; > } > - pr_err("%s: %p many->many\n", __func__, spte); > - BUG(); > + KVM_BUG(true, kvm, "no rmap for %p (many->many)", spte); > } > } > > > base-commit: d74237e747db7f9f27e821e6683d58185e846378 > -- > make sense, will update that in the next version.
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b5a44b8f5f7b..e132d82ab4c0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -956,12 +956,12 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, if (!rmap_head->val) { pr_err("%s: %p 0->BUG\n", __func__, spte); - BUG(); + KVM_BUG(true, kvm, ""); } else if (!(rmap_head->val & 1)) { rmap_printk("%p 1->0\n", spte); if ((u64 *)rmap_head->val != spte) { pr_err("%s: %p 1->BUG\n", __func__, spte); - BUG(); + KVM_BUG(true, kvm, ""); } rmap_head->val = 0; } else { @@ -980,7 +980,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, desc = desc->more; } pr_err("%s: %p many->many\n", __func__, spte); - BUG(); + KVM_BUG(true, kvm, ""); } }
Replace BUG() in pte_list_remove() with KVM_BUG() to avoid crashing the host. MMU bug is difficult to discover due to various racing conditions and corner cases and thus it extremely hard to debug. The situation gets much worse when it triggers the shutdown of a host. Host machine crash eliminates everything including the potential clues for debugging. From cloud computing service perspective, BUG() or BUG_ON() is probably no longer appropriate as the host reliability is top priority. Crashing the physical machine is almost never a good option as it eliminates innocent VMs and cause service outage in a larger scope. Even worse, if attacker can reliably triggers this code by diverting the control flow or corrupting the memory, then this becomes vm-of-death attack. This is a huge attack vector to cloud providers, as the death of one single host machine is not the end of the story. Without manual interferences, a failed cloud job may be dispatched to other hosts and continue host crashes until all of them are dead. Because of the above reasons, shrink the scope of crash to the target VM only. KVM_BUG() and KVM_BUG_ON() requires a valid struct kvm which requires extra plumbing. Avoid it in this version by just using kvm_get_running_vcpu()->kvm instead. Cc: Nagareddy Reddy <nspreddy@google.com> Cc: Jim Mattson <jmattson@google.com> Cc: David Matlack <dmatlack@google.com> Signed-off-by: Mingwei Zhang <mizhang@google.com> --- arch/x86/kvm/mmu/mmu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)