From patchwork Wed Mar 13 04:59:12 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiao Guangrong X-Patchwork-Id: 2261431 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 91D643FCF6 for ; Wed, 13 Mar 2013 04:59:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932766Ab3CME7V (ORCPT ); Wed, 13 Mar 2013 00:59:21 -0400 Received: from e23smtp08.au.ibm.com ([202.81.31.141]:40389 "EHLO e23smtp08.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751848Ab3CME7U (ORCPT ); Wed, 13 Mar 2013 00:59:20 -0400 Received: from /spool/local by e23smtp08.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 13 Mar 2013 14:57:21 +1000 Received: from d23dlp02.au.ibm.com (202.81.31.213) by e23smtp08.au.ibm.com (202.81.31.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 13 Mar 2013 14:57:19 +1000 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [9.190.235.21]) by d23dlp02.au.ibm.com (Postfix) with ESMTP id 726182BB0023; Wed, 13 Mar 2013 15:59:15 +1100 (EST) Received: from d23av01.au.ibm.com (d23av01.au.ibm.com [9.190.234.96]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r2D4xBQR66257036; Wed, 13 Mar 2013 15:59:12 +1100 Received: from d23av01.au.ibm.com (loopback [127.0.0.1]) by d23av01.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r2D4xE7C020748; Wed, 13 Mar 2013 15:59:14 +1100 Received: from localhost.localdomain ([9.123.236.246]) by d23av01.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r2D4xCK4020696; Wed, 13 Mar 2013 15:59:13 +1100 Message-ID: <514007A0.1040400@linux.vnet.ibm.com> Date: Wed, 13 Mar 2013 12:59:12 +0800 From: Xiao Guangrong User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Xiao Guangrong CC: Marcelo Tosatti , Gleb Natapov , LKML , KVM Subject: [PATCH 6/6] KVM: MMU: fast zap all shadow pages References: <514006AC.2020904@linux.vnet.ibm.com> In-Reply-To: <514006AC.2020904@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13031304-5140-0000-0000-000002E2633A Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The current kvm_mmu_zap_all is really slow - it is holding mmu-lock to walk and zap all shadow pages one by one, also it need to zap all guest page's rmap and all shadow page's parent spte list. Particularly, things become worse if guest uses more memory or vcpus. It is not good for scalability. Since all shadow page will be zapped, we can directly zap the mmu-cache and rmap so that vcpu will fault on the new mmu-cache, after that, we can directly free the memory used by old mmu-cache. The root shadow page is little especial since they are currently used by vcpus, we can not directly free them. So, we zap the root shadow pages and re-add them into the new mmu-cache. After this patch, kvm_mmu_zap_all can be faster 113% than before Signed-off-by: Xiao Guangrong --- arch/x86/kvm/mmu.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++----- 1 files changed, 56 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index e326099..536d9ce 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -4186,18 +4186,68 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, int slot) void kvm_mmu_zap_all(struct kvm *kvm) { - struct kvm_mmu_page *sp, *node; + LIST_HEAD(root_mmu_pages); LIST_HEAD(invalid_list); + struct list_head pte_list_descs; + struct kvm_mmu_cache *cache = &kvm->arch.mmu_cache; + struct kvm_mmu_page *sp, *node; + struct pte_list_desc *desc, *ndesc; + int root_sp = 0; spin_lock(&kvm->mmu_lock); + restart: - list_for_each_entry_safe(sp, node, - &kvm->arch.mmu_cache.active_mmu_pages, link) - if (kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list)) - goto restart; + /* + * The root shadow pages are being used on vcpus that can not + * directly removed, we filter them out and re-add them to the + * new mmu cache. + */ + list_for_each_entry_safe(sp, node, &cache->active_mmu_pages, link) + if (sp->root_count) { + int ret; + + root_sp++; + ret = kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list); + list_move(&sp->link, &root_mmu_pages); + if (ret) + goto restart; + } + + list_splice(&cache->active_mmu_pages, &invalid_list); + list_replace(&cache->pte_list_descs, &pte_list_descs); + + /* + * Reset the mmu cache so that later vcpu will fault on the new + * mmu cache. + */ + memset(cache, 0, sizeof(*cache)); + kvm_mmu_init(kvm); + + /* + * Now, the mmu cache has been reset, we can re-add the root shadow + * pages into the cache. + */ + list_replace(&root_mmu_pages, &cache->active_mmu_pages); + kvm_mod_used_mmu_pages(kvm, root_sp); + + /* Reset gfn's rmap and lpage info. */ + kvm_clear_all_gfn_page_info(kvm); + + /* + * Flush all TLBs so that vcpu can not use the invalid mappings. + * Do not disturb vcpus if root shadow pages have been zapped + * since KVM_REQ_MMU_RELOAD will force TLB to be flushed. + */ + if (!root_sp && !list_empty(&invalid_list)) + kvm_flush_remote_tlbs(kvm); - kvm_mmu_commit_zap_page(kvm, &invalid_list); spin_unlock(&kvm->mmu_lock); + + list_for_each_entry_safe(sp, node, &invalid_list, link) + kvm_mmu_free_page(sp); + + list_for_each_entry_safe(desc, ndesc, &pte_list_descs, list) + mmu_free_pte_list_desc(desc); } static int mmu_shrink(struct shrinker *shrink, struct shrink_control *sc)