From patchwork Wed May 10 12:18:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Boehr X-Patchwork-Id: 13236859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B72DC7EE23 for ; Wed, 10 May 2023 12:18:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236949AbjEJMSd (ORCPT ); Wed, 10 May 2023 08:18:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236086AbjEJMSb (ORCPT ); Wed, 10 May 2023 08:18:31 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA5A510C; Wed, 10 May 2023 05:18:30 -0700 (PDT) Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 34ABg64u014648; Wed, 10 May 2023 12:18:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=3L7qnAz8G30cCWR+DN6YlluAJCDFcaB+9TpKo4Z9VTg=; b=NS8S0HCLCF/QJthzh4GwdvzA5gDKp7kRfuOg/xhY+RL9ahrwISjlpZKRWbnJ8+yKYBlR RNPYuDJUF3D/Cpf/EVbGvBryfkooU+pQruE5SqpoZxUUFtucpDpXb92tp3HKYsggoeml OWzoBF6OVM14yc1SMd4YQ9JtP6Thrafhg3yhfjpdwby/JkZRfRKmYEshcrqzgWoJ4zhp jaIP2n7svs1V/Pe6GJOu9Ar/Iccbw7pyHYOnuBjUJRiFbTwqynkEcuMuURZSysztHjFJ il3PKEcj0oRcWNzO7f/gNeQiy3JNObvmHkxhEXlpzcgSlz0FEUue87nIG7ZHeY1vJBiz 3A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3qgameh53p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 May 2023 12:18:30 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 34ACGxoE000846; Wed, 10 May 2023 12:18:29 GMT Received: from ppma01fra.de.ibm.com (46.49.7a9f.ip4.static.sl-reverse.com [159.122.73.70]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3qgameh52b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 May 2023 12:18:29 +0000 Received: from pps.filterd (ppma01fra.de.ibm.com [127.0.0.1]) by ppma01fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 34A4idac003745; Wed, 10 May 2023 12:18:27 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma01fra.de.ibm.com (PPS) with ESMTPS id 3qf7e0ruhp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 May 2023 12:18:26 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 34ACINJK25559702 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 10 May 2023 12:18:23 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 73B5020040; Wed, 10 May 2023 12:18:23 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3A2152004D; Wed, 10 May 2023 12:18:23 +0000 (GMT) Received: from t35lp63.lnxne.boe (unknown [9.152.108.100]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 10 May 2023 12:18:23 +0000 (GMT) From: Nico Boehr To: borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, david@redhat.com Cc: kvm@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH v2 1/2] KVM: s390: add stat counter for shadow gmap events Date: Wed, 10 May 2023 14:18:21 +0200 Message-Id: <20230510121822.546629-2-nrb@linux.ibm.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230510121822.546629-1-nrb@linux.ibm.com> References: <20230510121822.546629-1-nrb@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: K-GXTKQR0lwiWhELBTa9YLhxpF44Zenr X-Proofpoint-GUID: Mxrwtf9d1uWWxh_gnA_Z_lV2SMF16BZ2 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-05-10_04,2023-05-05_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 bulkscore=0 priorityscore=1501 mlxlogscore=999 impostorscore=0 phishscore=0 adultscore=0 suspectscore=0 spamscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2304280000 definitions=main-2305100095 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The shadow gmap tracks memory of nested guests (guest-3). In certain scenarios, the shadow gmap needs to be rebuilt, which is a costly operation since it involves a SIE exit into guest-1 for every entry in the respective shadow level. Add kvm stat counters when new shadow structures are created at various levels. Also add a counter gmap_shadow_acquire when a completely fresh shadow gmap is created. Note that when several levels are shadowed at once, counters on all affected levels will be increased. Also note that not all page table levels need to be present and a ASCE can directly point to e.g. a segment table. In this case, a new segment table will always be equivalent to a new shadow gmap and hence will be counted as gmap_shadow_acquire and not as gmap_shadow_segment. Signed-off-by: Nico Boehr --- arch/s390/include/asm/kvm_host.h | 6 ++++++ arch/s390/kvm/gaccess.c | 7 +++++++ arch/s390/kvm/kvm-s390.c | 8 +++++++- arch/s390/kvm/vsie.c | 1 + 4 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h index 2bbc3d54959d..d35e03e82d3d 100644 --- a/arch/s390/include/asm/kvm_host.h +++ b/arch/s390/include/asm/kvm_host.h @@ -777,6 +777,12 @@ struct kvm_vm_stat { u64 inject_service_signal; u64 inject_virtio; u64 aen_forward; + u64 gmap_shadow_acquire; + u64 gmap_shadow_r1_te; + u64 gmap_shadow_r2_te; + u64 gmap_shadow_r3_te; + u64 gmap_shadow_sg_te; + u64 gmap_shadow_pg_te; }; struct kvm_arch_memory_slot { diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/gaccess.c index 3eb85f254881..6f4292ad0e4a 100644 --- a/arch/s390/kvm/gaccess.c +++ b/arch/s390/kvm/gaccess.c @@ -1382,6 +1382,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, unsigned long *pgt, int *dat_protection, int *fake) { + struct kvm *kvm; struct gmap *parent; union asce asce; union vaddress vaddr; @@ -1390,6 +1391,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, *fake = 0; *dat_protection = 0; + kvm = sg->private; parent = sg->parent; vaddr.addr = saddr; asce.val = sg->orig_asce; @@ -1450,6 +1452,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, rc = gmap_shadow_r2t(sg, saddr, rfte.val, *fake); if (rc) return rc; + kvm->stat.gmap_shadow_r1_te++; } fallthrough; case ASCE_TYPE_REGION2: { @@ -1478,6 +1481,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, rc = gmap_shadow_r3t(sg, saddr, rste.val, *fake); if (rc) return rc; + kvm->stat.gmap_shadow_r2_te++; } fallthrough; case ASCE_TYPE_REGION3: { @@ -1515,6 +1519,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, rc = gmap_shadow_sgt(sg, saddr, rtte.val, *fake); if (rc) return rc; + kvm->stat.gmap_shadow_r3_te++; } fallthrough; case ASCE_TYPE_SEGMENT: { @@ -1548,6 +1553,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, rc = gmap_shadow_pgt(sg, saddr, ste.val, *fake); if (rc) return rc; + kvm->stat.gmap_shadow_sg_te++; } } /* Return the parent address of the page table */ @@ -1618,6 +1624,7 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, pte.p |= dat_protection; if (!rc) rc = gmap_shadow_page(sg, saddr, __pte(pte.val)); + vcpu->kvm->stat.gmap_shadow_pg_te++; ipte_unlock(vcpu->kvm); mmap_read_unlock(sg->mm); return rc; diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 17b81659cdb2..ded4149e145b 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -66,7 +66,13 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = { STATS_DESC_COUNTER(VM, inject_pfault_done), STATS_DESC_COUNTER(VM, inject_service_signal), STATS_DESC_COUNTER(VM, inject_virtio), - STATS_DESC_COUNTER(VM, aen_forward) + STATS_DESC_COUNTER(VM, aen_forward), + STATS_DESC_COUNTER(VM, gmap_shadow_acquire), + STATS_DESC_COUNTER(VM, gmap_shadow_r1_te), + STATS_DESC_COUNTER(VM, gmap_shadow_r2_te), + STATS_DESC_COUNTER(VM, gmap_shadow_r3_te), + STATS_DESC_COUNTER(VM, gmap_shadow_sg_te), + STATS_DESC_COUNTER(VM, gmap_shadow_pg_te), }; const struct kvm_stats_header kvm_vm_stats_header = { diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c index 8d6b765abf29..beb3be037722 100644 --- a/arch/s390/kvm/vsie.c +++ b/arch/s390/kvm/vsie.c @@ -1221,6 +1221,7 @@ static int acquire_gmap_shadow(struct kvm_vcpu *vcpu, if (IS_ERR(gmap)) return PTR_ERR(gmap); gmap->private = vcpu->kvm; + vcpu->kvm->stat.gmap_shadow_acquire++; WRITE_ONCE(vsie_page->gmap, gmap); return 0; } From patchwork Wed May 10 12:18:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nico Boehr X-Patchwork-Id: 13236860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CB80C77B7C for ; Wed, 10 May 2023 12:18:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236476AbjEJMSf (ORCPT ); Wed, 10 May 2023 08:18:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234032AbjEJMSc (ORCPT ); Wed, 10 May 2023 08:18:32 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D5441BFE; Wed, 10 May 2023 05:18:31 -0700 (PDT) Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 34ABsGGc009362; Wed, 10 May 2023 12:18:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=rFtdxM5wh8F+niGGBbt2eXCPP2Ny+n/Z9JmbUoTdo8E=; b=SVvd2yMdogBzrCRTKwe8ajD7DETo+ZuGrDXq+RBbGJHNos7JaEf0MaGucwTDcKFAhdcZ 0Oou+Kh5oQSyuDY3hYnigznDxcNUqIjM3Jt/YM0fR8zt6fi8cs0IQGQnGMqsTIbg8Fds wyaDVg5oaP5GGkWLqwT9rFFJwKmU4tyMZrl+XIUk24OVYw51mOklhkFmPBIYDDjlaVds c78MpMo0lZ5LuoTV8UGaTjP4jqcXAKO/9EhUD+UuJ+QGxNb1vOZRZw5fpSUlmFahOHWk 4BDM9f1TgqrCgdiqIcsqF4QmMANlsEQMn8ACAP/FeI5EzcOguERnTFNhPgolnT7EfyAf VA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3qg8h0vv59-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 May 2023 12:18:30 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 34ABkdYY014629; Wed, 10 May 2023 12:18:30 GMT Received: from ppma05fra.de.ibm.com (6c.4a.5195.ip4.static.sl-reverse.com [149.81.74.108]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3qg8h0vv45-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 May 2023 12:18:29 +0000 Received: from pps.filterd (ppma05fra.de.ibm.com [127.0.0.1]) by ppma05fra.de.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 34A9Fqrk024133; Wed, 10 May 2023 12:18:27 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma05fra.de.ibm.com (PPS) with ESMTPS id 3qf7mhgusf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 May 2023 12:18:27 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 34ACINLI25559704 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 10 May 2023 12:18:24 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DC6D42004B; Wed, 10 May 2023 12:18:23 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7D13C20043; Wed, 10 May 2023 12:18:23 +0000 (GMT) Received: from t35lp63.lnxne.boe (unknown [9.152.108.100]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 10 May 2023 12:18:23 +0000 (GMT) From: Nico Boehr To: borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, david@redhat.com Cc: kvm@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH v2 2/2] KVM: s390: add tracepoint in gmap notifier Date: Wed, 10 May 2023 14:18:22 +0200 Message-Id: <20230510121822.546629-3-nrb@linux.ibm.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230510121822.546629-1-nrb@linux.ibm.com> References: <20230510121822.546629-1-nrb@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: Z_tImuWbdeLoTYxXj72wKBrW2xzXQuLo X-Proofpoint-GUID: IyS2mfRU6b8QmL2heQEDkSt8kYbXUjzr X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-05-10_04,2023-05-05_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 malwarescore=0 phishscore=0 adultscore=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2304280000 definitions=main-2305100095 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The gmap notifier is called for changes in table entries with the notifier bit set. To diagnose performance issues, it can be useful to see what causes certain changes in the gmap. Hence, add a tracepoint in the gmap notifier. Signed-off-by: Nico Boehr Acked-by: David Hildenbrand --- arch/s390/kvm/kvm-s390.c | 2 ++ arch/s390/kvm/trace-s390.h | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index ded4149e145b..e8476c023b07 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -3982,6 +3982,8 @@ static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start, unsigned long prefix; unsigned long i; + trace_kvm_s390_gmap_notifier(start, end, gmap_is_shadow(gmap)); + if (gmap_is_shadow(gmap)) return; if (start >= 1UL << 31) diff --git a/arch/s390/kvm/trace-s390.h b/arch/s390/kvm/trace-s390.h index 6f0209d45164..5dabd0b64d6e 100644 --- a/arch/s390/kvm/trace-s390.h +++ b/arch/s390/kvm/trace-s390.h @@ -333,6 +333,29 @@ TRACE_EVENT(kvm_s390_airq_suppressed, __entry->id, __entry->isc) ); +/* + * Trace point for gmap notifier calls. + */ +TRACE_EVENT(kvm_s390_gmap_notifier, + TP_PROTO(unsigned long start, unsigned long end, unsigned int shadow), + TP_ARGS(start, end, shadow), + + TP_STRUCT__entry( + __field(unsigned long, start) + __field(unsigned long, end) + __field(unsigned int, shadow) + ), + + TP_fast_assign( + __entry->start = start; + __entry->end = end; + __entry->shadow = shadow; + ), + + TP_printk("gmap notified (start:0x%lx end:0x%lx shadow:%d)", + __entry->start, __entry->end, __entry->shadow) + ); + #endif /* _TRACE_KVMS390_H */