From patchwork Wed Mar 27 15:22:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A9BB0C47DD9 for ; Wed, 27 Mar 2024 15:22:49 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698601.1090487 (Exim 4.92) (envelope-from ) id 1rpV6v-0007gH-EE; Wed, 27 Mar 2024 15:22:41 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698601.1090487; Wed, 27 Mar 2024 15:22:41 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV6v-0007gA-BP; Wed, 27 Mar 2024 15:22:41 +0000 Received: by outflank-mailman (input) for mailman id 698601; Wed, 27 Mar 2024 15:22:40 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV6u-0007fn-EF for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:22:40 +0000 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id d84f055e-ec4d-11ee-a1ef-f123f15fe8a2; Wed, 27 Mar 2024 16:22:38 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id CFEAA5FF5A; Wed, 27 Mar 2024 15:22:37 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 53EBB13215; Wed, 27 Mar 2024 15:22:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id enLTEr05BGZqfQAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:22:37 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d84f055e-ec4d-11ee-a1ef-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552957; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HU+H+gC+IivP7v8BYwH4cbldKan0Qm9QDP70wtFCLOI=; b=MN9eZgY0qfMPhdRF/3BSZq9YkhkZG0013tMUplr8gu1JpUrBOvBmaGHCLJw/h3WsumyCJo DZGhOhlbKuzzRq98jkshXqFQramFzRda6bT67UfGuqUM9IdzkHEZoA1DFph2BwYSl3xKJK zQXPYE8kYC090+9hzeTantVavEd2kKs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552957; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HU+H+gC+IivP7v8BYwH4cbldKan0Qm9QDP70wtFCLOI=; b=MN9eZgY0qfMPhdRF/3BSZq9YkhkZG0013tMUplr8gu1JpUrBOvBmaGHCLJw/h3WsumyCJo DZGhOhlbKuzzRq98jkshXqFQramFzRda6bT67UfGuqUM9IdzkHEZoA1DFph2BwYSl3xKJK zQXPYE8kYC090+9hzeTantVavEd2kKs= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Stefano Stabellini , Julien Grall , Bertrand Marquis , Michal Orzel , Volodymyr Babchuk , Andrew Cooper , George Dunlap , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Tamas K Lengyel , Lukasz Hawrylko , "Daniel P. Smith" , =?utf-8?q?Mateusz_M=C3=B3?= =?utf-8?q?wka?= Subject: [PATCH v6 1/8] xen/spinlock: add explicit non-recursive locking functions Date: Wed, 27 Mar 2024 16:22:22 +0100 Message-Id: <20240327152229.25847-2-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 X-Spamd-Result: default: False [8.80 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BAYES_SPAM(5.10)[100.00%]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; NEURAL_HAM_SHORT(-0.20)[-0.999]; RCPT_COUNT_TWELVE(0.00)[15]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[] Authentication-Results: smtp-out2.suse.de; none In order to prepare a type-safe recursive spinlock structure, add explicitly non-recursive locking functions to be used for non-recursive locking of spinlocks, which are used recursively, too. Signed-off-by: Juergen Gross Acked-by: Jan Beulich Acked-by: Michal Orzel --- V2: - rename functions (Jan Beulich) - get rid of !! in pcidevs_locked() (Jan Beulich) V5: - remove spurious change (Julien Grall) - add nrspin_lock() description (Julien Grall) --- xen/arch/arm/mm.c | 4 ++-- xen/arch/x86/domain.c | 12 ++++++------ xen/arch/x86/mm.c | 12 ++++++------ xen/arch/x86/mm/mem_sharing.c | 8 ++++---- xen/arch/x86/mm/p2m-pod.c | 4 ++-- xen/arch/x86/mm/p2m.c | 4 ++-- xen/arch/x86/tboot.c | 4 ++-- xen/common/domctl.c | 4 ++-- xen/common/grant_table.c | 10 +++++----- xen/common/memory.c | 4 ++-- xen/common/numa.c | 4 ++-- xen/common/page_alloc.c | 16 ++++++++-------- xen/drivers/char/console.c | 16 ++++++++-------- xen/include/xen/spinlock.h | 29 +++++++++++++++++++++++------ 14 files changed, 74 insertions(+), 57 deletions(-) diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index b15a18a494..def939172c 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -105,7 +105,7 @@ void share_xen_page_with_guest(struct page_info *page, struct domain *d, if ( page_get_owner(page) == d ) return; - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); /* * The incremented type count pins as writable or read-only. @@ -136,7 +136,7 @@ void share_xen_page_with_guest(struct page_info *page, struct domain *d, page_list_add_tail(page, &d->xenpage_list); } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); } int xenmem_add_to_physmap_one( diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c index a11c55f921..33a2830d9d 100644 --- a/xen/arch/x86/domain.c +++ b/xen/arch/x86/domain.c @@ -212,7 +212,7 @@ void dump_pageframe_info(struct domain *d) { unsigned long total[MASK_EXTR(PGT_type_mask, PGT_type_mask) + 1] = {}; - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_for_each ( page, &d->page_list ) { unsigned int index = MASK_EXTR(page->u.inuse.type_info, @@ -231,13 +231,13 @@ void dump_pageframe_info(struct domain *d) _p(mfn_x(page_to_mfn(page))), page->count_info, page->u.inuse.type_info); } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); } if ( is_hvm_domain(d) ) p2m_pod_dump_data(d); - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_for_each ( page, &d->xenpage_list ) { @@ -253,7 +253,7 @@ void dump_pageframe_info(struct domain *d) page->count_info, page->u.inuse.type_info); } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); } void update_guest_memory_policy(struct vcpu *v, @@ -2448,10 +2448,10 @@ int domain_relinquish_resources(struct domain *d) d->arch.auto_unmask = 0; } - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_splice(&d->arch.relmem_list, &d->page_list); INIT_PAGE_LIST_HEAD(&d->arch.relmem_list); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); PROGRESS(xen): diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index 62f5b811bb..b4d125db39 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -482,7 +482,7 @@ void share_xen_page_with_guest(struct page_info *page, struct domain *d, set_gpfn_from_mfn(mfn_x(page_to_mfn(page)), INVALID_M2P_ENTRY); - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); /* The incremented type count pins as writable or read-only. */ page->u.inuse.type_info = @@ -502,7 +502,7 @@ void share_xen_page_with_guest(struct page_info *page, struct domain *d, page_list_add_tail(page, &d->xenpage_list); } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); } void make_cr3(struct vcpu *v, mfn_t mfn) @@ -3597,11 +3597,11 @@ long do_mmuext_op( { bool drop_ref; - spin_lock(&pg_owner->page_alloc_lock); + nrspin_lock(&pg_owner->page_alloc_lock); drop_ref = (pg_owner->is_dying && test_and_clear_bit(_PGT_pinned, &page->u.inuse.type_info)); - spin_unlock(&pg_owner->page_alloc_lock); + nrspin_unlock(&pg_owner->page_alloc_lock); if ( drop_ref ) { pin_drop: @@ -4424,7 +4424,7 @@ int steal_page( * that it might be upon return from alloc_domheap_pages with * MEMF_no_owner set. */ - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); BUG_ON(page->u.inuse.type_info & (PGT_count_mask | PGT_locked | PGT_pinned)); @@ -4436,7 +4436,7 @@ int steal_page( if ( !(memflags & MEMF_no_refcount) && !domain_adjust_tot_pages(d, -1) ) drop_dom_ref = true; - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); if ( unlikely(drop_dom_ref) ) put_domain(d); diff --git a/xen/arch/x86/mm/mem_sharing.c b/xen/arch/x86/mm/mem_sharing.c index f58576c702..da28266ef0 100644 --- a/xen/arch/x86/mm/mem_sharing.c +++ b/xen/arch/x86/mm/mem_sharing.c @@ -740,11 +740,11 @@ static int page_make_private(struct domain *d, struct page_info *page) if ( !get_page(page, dom_cow) ) return -EINVAL; - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); if ( d->is_dying ) { - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); put_page(page); return -EBUSY; } @@ -752,7 +752,7 @@ static int page_make_private(struct domain *d, struct page_info *page) expected_type = (PGT_shared_page | PGT_validated | PGT_locked | 2); if ( page->u.inuse.type_info != expected_type ) { - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); put_page(page); return -EEXIST; } @@ -769,7 +769,7 @@ static int page_make_private(struct domain *d, struct page_info *page) if ( domain_adjust_tot_pages(d, 1) == 1 ) get_knownalive_domain(d); page_list_add_tail(page, &d->page_list); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); put_page(page); diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c index 674f321cf6..c48ea169b7 100644 --- a/xen/arch/x86/mm/p2m-pod.c +++ b/xen/arch/x86/mm/p2m-pod.c @@ -27,7 +27,7 @@ static always_inline void lock_page_alloc(struct p2m_domain *p2m) { page_alloc_mm_pre_lock(p2m->domain); - spin_lock(&(p2m->domain->page_alloc_lock)); + nrspin_lock(&(p2m->domain->page_alloc_lock)); page_alloc_mm_post_lock(p2m->domain, p2m->domain->arch.page_alloc_unlock_level); } @@ -35,7 +35,7 @@ static always_inline void lock_page_alloc(struct p2m_domain *p2m) static inline void unlock_page_alloc(struct p2m_domain *p2m) { page_alloc_mm_unlock(p2m->domain->arch.page_alloc_unlock_level); - spin_unlock(&(p2m->domain->page_alloc_lock)); + nrspin_unlock(&(p2m->domain->page_alloc_lock)); } /* diff --git a/xen/arch/x86/mm/p2m.c b/xen/arch/x86/mm/p2m.c index 4c5a79eb53..ce742c12e0 100644 --- a/xen/arch/x86/mm/p2m.c +++ b/xen/arch/x86/mm/p2m.c @@ -2234,7 +2234,7 @@ void audit_p2m(struct domain *d, /* Audit part two: walk the domain's page allocation list, checking * the m2p entries. */ - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_for_each ( page, &d->page_list ) { mfn = mfn_x(page_to_mfn(page)); @@ -2286,7 +2286,7 @@ void audit_p2m(struct domain *d, P2M_PRINTK("OK: mfn=%#lx, gfn=%#lx, p2mfn=%#lx\n", mfn, gfn, mfn_x(p2mfn)); } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); pod_unlock(p2m); p2m_unlock(p2m); diff --git a/xen/arch/x86/tboot.c b/xen/arch/x86/tboot.c index 4c254b4e34..ba0700d2d5 100644 --- a/xen/arch/x86/tboot.c +++ b/xen/arch/x86/tboot.c @@ -205,14 +205,14 @@ static void tboot_gen_domain_integrity(const uint8_t key[TB_KEY_SIZE], continue; printk("MACing Domain %u\n", d->domain_id); - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_for_each(page, &d->page_list) { void *pg = __map_domain_page(page); vmac_update(pg, PAGE_SIZE, &ctx); unmap_domain_page(pg); } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); if ( is_iommu_enabled(d) && is_vtd ) { diff --git a/xen/common/domctl.c b/xen/common/domctl.c index d94a9dae91..17d67651a7 100644 --- a/xen/common/domctl.c +++ b/xen/common/domctl.c @@ -621,14 +621,14 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl) { uint64_t new_max = op->u.max_mem.max_memkb >> (PAGE_SHIFT - 10); - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); /* * NB. We removed a check that new_max >= current tot_pages; this means * that the domain will now be allowed to "ratchet" down to new_max. In * the meantime, while tot > max, all new allocations are disallowed. */ d->max_pages = min(new_max, (uint64_t)(typeof(d->max_pages))-1); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); break; } diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c index 7708930882..e98e9da520 100644 --- a/xen/common/grant_table.c +++ b/xen/common/grant_table.c @@ -2376,7 +2376,7 @@ gnttab_transfer( mfn = page_to_mfn(page); } - spin_lock(&e->page_alloc_lock); + nrspin_lock(&e->page_alloc_lock); /* * Check that 'e' will accept the page and has reservation @@ -2387,7 +2387,7 @@ gnttab_transfer( unlikely(domain_tot_pages(e) >= e->max_pages) || unlikely(!(e->tot_pages + 1)) ) { - spin_unlock(&e->page_alloc_lock); + nrspin_unlock(&e->page_alloc_lock); if ( e->is_dying ) gdprintk(XENLOG_INFO, "Transferee d%d is dying\n", @@ -2411,7 +2411,7 @@ gnttab_transfer( * safely drop the lock and re-aquire it later to add page to the * pagelist. */ - spin_unlock(&e->page_alloc_lock); + nrspin_unlock(&e->page_alloc_lock); okay = gnttab_prepare_for_transfer(e, d, gop.ref); /* @@ -2427,9 +2427,9 @@ gnttab_transfer( * Need to grab this again to safely free our "reserved" * page in the page total */ - spin_lock(&e->page_alloc_lock); + nrspin_lock(&e->page_alloc_lock); drop_dom_ref = !domain_adjust_tot_pages(e, -1); - spin_unlock(&e->page_alloc_lock); + nrspin_unlock(&e->page_alloc_lock); if ( okay /* i.e. e->is_dying due to the surrounding if() */ ) gdprintk(XENLOG_INFO, "Transferee d%d is now dying\n", diff --git a/xen/common/memory.c b/xen/common/memory.c index b3b05c2ec0..b4593f5f45 100644 --- a/xen/common/memory.c +++ b/xen/common/memory.c @@ -770,10 +770,10 @@ static long memory_exchange(XEN_GUEST_HANDLE_PARAM(xen_memory_exchange_t) arg) (1UL << in_chunk_order)) - (j * (1UL << exch.out.extent_order))); - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); drop_dom_ref = (dec_count && !domain_adjust_tot_pages(d, -dec_count)); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); if ( drop_dom_ref ) put_domain(d); diff --git a/xen/common/numa.c b/xen/common/numa.c index 6374ba1721..28a09766fa 100644 --- a/xen/common/numa.c +++ b/xen/common/numa.c @@ -719,13 +719,13 @@ static void cf_check dump_numa(unsigned char key) memset(page_num_node, 0, sizeof(page_num_node)); - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_for_each ( page, &d->page_list ) { i = page_to_nid(page); page_num_node[i]++; } - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); for_each_online_node ( i ) printk(" Node %u: %u\n", i, page_num_node[i]); diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index c94834d71b..4d6ce726e3 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -516,7 +516,7 @@ int domain_set_outstanding_pages(struct domain *d, unsigned long pages) * must always take the global heap_lock rather than only in the much * rarer case that d->outstanding_pages is non-zero */ - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); spin_lock(&heap_lock); /* pages==0 means "unset" the claim. */ @@ -562,7 +562,7 @@ int domain_set_outstanding_pages(struct domain *d, unsigned long pages) out: spin_unlock(&heap_lock); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); return ret; } @@ -2348,7 +2348,7 @@ int assign_pages( int rc = 0; unsigned int i; - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); if ( unlikely(d->is_dying) ) { @@ -2430,7 +2430,7 @@ int assign_pages( } out: - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); return rc; } @@ -2909,9 +2909,9 @@ mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags) ASSERT_ALLOC_CONTEXT(); /* Acquire a page from reserved page list(resv_page_list). */ - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page = page_list_remove_head(&d->resv_page_list); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); if ( unlikely(!page) ) return INVALID_MFN; @@ -2930,9 +2930,9 @@ mfn_t acquire_reserved_page(struct domain *d, unsigned int memflags) */ unprepare_staticmem_pages(page, 1, false); fail: - spin_lock(&d->page_alloc_lock); + nrspin_lock(&d->page_alloc_lock); page_list_add_tail(page, &d->resv_page_list); - spin_unlock(&d->page_alloc_lock); + nrspin_unlock(&d->page_alloc_lock); return INVALID_MFN; } #endif diff --git a/xen/drivers/char/console.c b/xen/drivers/char/console.c index ccd5f8cc14..22f50fc617 100644 --- a/xen/drivers/char/console.c +++ b/xen/drivers/char/console.c @@ -368,9 +368,9 @@ long read_console_ring(struct xen_sysctl_readconsole *op) if ( op->clear ) { - spin_lock_irq(&console_lock); + nrspin_lock_irq(&console_lock); conringc = p - c > conring_size ? p - conring_size : c; - spin_unlock_irq(&console_lock); + nrspin_unlock_irq(&console_lock); } op->count = sofar; @@ -638,7 +638,7 @@ static long guest_console_write(XEN_GUEST_HANDLE_PARAM(char) buffer, if ( is_hardware_domain(cd) ) { /* Use direct console output as it could be interactive */ - spin_lock_irq(&console_lock); + nrspin_lock_irq(&console_lock); console_serial_puts(kbuf, kcount); video_puts(kbuf, kcount); @@ -659,7 +659,7 @@ static long guest_console_write(XEN_GUEST_HANDLE_PARAM(char) buffer, tasklet_schedule(¬ify_dom0_con_ring_tasklet); } - spin_unlock_irq(&console_lock); + nrspin_unlock_irq(&console_lock); } else { @@ -1026,9 +1026,9 @@ void __init console_init_preirq(void) pv_console_set_rx_handler(serial_rx); /* HELLO WORLD --- start-of-day banner text. */ - spin_lock(&console_lock); + nrspin_lock(&console_lock); __putstr(xen_banner()); - spin_unlock(&console_lock); + nrspin_unlock(&console_lock); printk("Xen version %d.%d%s (%s@%s) (%s) %s %s\n", xen_major_version(), xen_minor_version(), xen_extra_version(), xen_compile_by(), xen_compile_domain(), xen_compiler(), @@ -1065,13 +1065,13 @@ void __init console_init_ring(void) } opt_conring_size = PAGE_SIZE << order; - spin_lock_irqsave(&console_lock, flags); + nrspin_lock_irqsave(&console_lock, flags); for ( i = conringc ; i != conringp; i++ ) ring[i & (opt_conring_size - 1)] = conring[i & (conring_size - 1)]; conring = ring; smp_wmb(); /* Allow users of console_force_unlock() to see larger buffer. */ conring_size = opt_conring_size; - spin_unlock_irqrestore(&console_lock, flags); + nrspin_unlock_irqrestore(&console_lock, flags); printk("Allocated console ring of %u KiB.\n", opt_conring_size >> 10); } diff --git a/xen/include/xen/spinlock.h b/xen/include/xen/spinlock.h index 593cba640e..53a33653f9 100644 --- a/xen/include/xen/spinlock.h +++ b/xen/include/xen/spinlock.h @@ -102,6 +102,9 @@ struct lock_profile_qhead { }; #define LOCK_PROFILE_(lockname) { .name = #lockname, .ptr.lock = &(lockname), } +#define RLOCK_PROFILE_(lockname) { .name = #lockname, \ + .ptr.rlock = &(lockname), \ + .is_rlock = true, } #define LOCK_PROFILE_PTR_(name) \ static struct lock_profile * const lock_profile__##name \ __used_section(".lockprofile.data") = \ @@ -118,10 +121,10 @@ struct lock_profile_qhead { LOCK_PROFILE_PTR_(l) #define DEFINE_RSPINLOCK(l) \ rspinlock_t l = SPIN_LOCK_UNLOCKED_(NULL); \ - static struct lock_profile lock_profile_data__##l = LOCK_PROFILE_(l); \ + static struct lock_profile lock_profile_data__##l = RLOCK_PROFILE_(l); \ LOCK_PROFILE_PTR_(l) -#define spin_lock_init_prof__(s, l, locktype) \ +#define spin_lock_init_prof__(s, l, lockptr, locktype, isr) \ do { \ struct lock_profile *prof; \ prof = xzalloc(struct lock_profile); \ @@ -134,13 +137,16 @@ struct lock_profile_qhead { break; \ } \ prof->name = #l; \ - prof->ptr.lock = &(s)->l; \ + prof->ptr.lockptr = &(s)->l; \ + prof->is_rlock = isr; \ prof->next = (s)->profile_head.elem_q; \ (s)->profile_head.elem_q = prof; \ } while( 0 ) -#define spin_lock_init_prof(s, l) spin_lock_init_prof__(s, l, spinlock_t) -#define rspin_lock_init_prof(s, l) spin_lock_init_prof__(s, l, rspinlock_t) +#define spin_lock_init_prof(s, l) \ + spin_lock_init_prof__(s, l, lock, spinlock_t, false) +#define rspin_lock_init_prof(s, l) \ + spin_lock_init_prof__(s, l, rlock, rspinlock_t, true) void _lock_profile_register_struct( int32_t type, struct lock_profile_qhead *qhead, int32_t idx); @@ -274,7 +280,10 @@ static always_inline void spin_lock_if(bool condition, spinlock_t *l) * reentered recursively on the same CPU. All critical regions that may form * part of a recursively-nested set must be protected by these forms. If there * are any critical regions that cannot form part of such a set, they can use - * standard spin_[un]lock(). + * nrspin_[un]lock(). + * The nrspin_[un]lock() forms act the same way as normal spin_[un]lock() + * calls, but operate on rspinlock_t locks. nrspin_lock() and rspin_lock() + * calls are blocking to each other for a specific lock even on the same cpu. */ bool _rspin_trylock(rspinlock_t *lock); void _rspin_lock(rspinlock_t *lock); @@ -298,4 +307,12 @@ static always_inline void rspin_lock(rspinlock_t *lock) #define rspin_unlock(l) _rspin_unlock(l) #define rspin_unlock_irqrestore(l, f) _rspin_unlock_irqrestore(l, f) +#define nrspin_trylock(l) spin_trylock(l) +#define nrspin_lock(l) spin_lock(l) +#define nrspin_unlock(l) spin_unlock(l) +#define nrspin_lock_irq(l) spin_lock_irq(l) +#define nrspin_unlock_irq(l) spin_unlock_irq(l) +#define nrspin_lock_irqsave(l, f) spin_lock_irqsave(l, f) +#define nrspin_unlock_irqrestore(l, f) spin_unlock_irqrestore(l, f) + #endif /* __SPINLOCK_H__ */ From patchwork Wed Mar 27 15:22:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92251C47DD9 for ; Wed, 27 Mar 2024 15:22:53 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698602.1090498 (Exim 4.92) (envelope-from ) id 1rpV6z-0007yG-P8; Wed, 27 Mar 2024 15:22:45 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698602.1090498; Wed, 27 Mar 2024 15:22:45 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV6z-0007y9-Ll; Wed, 27 Mar 2024 15:22:45 +0000 Received: by outflank-mailman (input) for mailman id 698602; Wed, 27 Mar 2024 15:22:44 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV6y-0007Qx-KO for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:22:44 +0000 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id dbacd412-ec4d-11ee-afe3-a90da7624cb6; Wed, 27 Mar 2024 16:22:44 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 791595FF27; Wed, 27 Mar 2024 15:22:43 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 3B6F213215; Wed, 27 Mar 2024 15:22:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id 0sYKDcM5BGazfQAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:22:43 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: dbacd412-ec4d-11ee-afe3-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552963; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CNcfOxsm5v7RMnma3LjjKR0Cscie45Q1Vyko4FogwSU=; b=tmt8BiVpGRS9VE7HVpTowDPulIKHtD1qAhrexfIaueQDYMGV+UI8+ukvXI/sTkTj9Vuelv QKwYeqOTm0+aNmS+VNoi4LxOiI8FyZHC8xLVeMlVmGQNObWpZjl69z4fJ6l8xYplHy7ZZ7 CEHnV8TBo75E1dJ2vxzf5lLDL303wqc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552963; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CNcfOxsm5v7RMnma3LjjKR0Cscie45Q1Vyko4FogwSU=; b=tmt8BiVpGRS9VE7HVpTowDPulIKHtD1qAhrexfIaueQDYMGV+UI8+ukvXI/sTkTj9Vuelv QKwYeqOTm0+aNmS+VNoi4LxOiI8FyZHC8xLVeMlVmGQNObWpZjl69z4fJ6l8xYplHy7ZZ7 CEHnV8TBo75E1dJ2vxzf5lLDL303wqc= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH v6 2/8] xen/spinlock: add another function level Date: Wed, 27 Mar 2024 16:22:23 +0100 Message-Id: <20240327152229.25847-3-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spamd-Result: default: False [-1.51 / 50.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:106:10:150:64:167:received]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[suse.com:+]; MX_GOOD(-0.01)[]; RCPT_COUNT_SEVEN(0.00)[7]; NEURAL_HAM_SHORT(-0.20)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; BAYES_HAM(-3.00)[100.00%]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_MED(-2.00)[suse.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:dkim,suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_TLS_ALL(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:98:from] Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b=tmt8BiVp X-Rspamd-Queue-Id: 791595FF27 Add another function level in spinlock.c hiding the spinlock_t layout from the low level locking code. This is done in preparation of introducing rspinlock_t for recursive locks without having to duplicate all of the locking code. Signed-off-by: Juergen Gross Reviewed-by: Jan Beulich --- V2: - new patch V5: - don't regress spin_is_locked() for rspin-lock (Jan Beulich) - use bool as return type of spin_is_locked_common() and spin_trylock_common() (Jan Beulich) --- xen/common/spinlock.c | 103 ++++++++++++++++++++++++------------- xen/include/xen/spinlock.h | 1 + 2 files changed, 68 insertions(+), 36 deletions(-) diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c index 874ed762b4..648393d95f 100644 --- a/xen/common/spinlock.c +++ b/xen/common/spinlock.c @@ -261,29 +261,31 @@ void spin_debug_disable(void) #ifdef CONFIG_DEBUG_LOCK_PROFILE +#define LOCK_PROFILE_PAR lock->profile #define LOCK_PROFILE_REL \ - if ( lock->profile ) \ + if ( profile ) \ { \ - lock->profile->time_hold += NOW() - lock->profile->time_locked; \ - lock->profile->lock_cnt++; \ + profile->time_hold += NOW() - profile->time_locked; \ + profile->lock_cnt++; \ } #define LOCK_PROFILE_VAR(var, val) s_time_t var = (val) #define LOCK_PROFILE_BLOCK(var) var = var ? : NOW() #define LOCK_PROFILE_BLKACC(tst, val) \ if ( tst ) \ { \ - lock->profile->time_block += lock->profile->time_locked - (val); \ - lock->profile->block_cnt++; \ + profile->time_block += profile->time_locked - (val); \ + profile->block_cnt++; \ } #define LOCK_PROFILE_GOT(val) \ - if ( lock->profile ) \ + if ( profile ) \ { \ - lock->profile->time_locked = NOW(); \ + profile->time_locked = NOW(); \ LOCK_PROFILE_BLKACC(val, val); \ } #else +#define LOCK_PROFILE_PAR NULL #define LOCK_PROFILE_REL #define LOCK_PROFILE_VAR(var, val) #define LOCK_PROFILE_BLOCK(var) @@ -307,17 +309,18 @@ static always_inline uint16_t observe_head(const spinlock_tickets_t *t) return read_atomic(&t->head); } -static void always_inline spin_lock_common(spinlock_t *lock, +static void always_inline spin_lock_common(spinlock_tickets_t *t, + union lock_debug *debug, + struct lock_profile *profile, void (*cb)(void *data), void *data) { spinlock_tickets_t tickets = SPINLOCK_TICKET_INC; LOCK_PROFILE_VAR(block, 0); - check_lock(&lock->debug, false); + check_lock(debug, false); preempt_disable(); - tickets.head_tail = arch_fetch_and_add(&lock->tickets.head_tail, - tickets.head_tail); - while ( tickets.tail != observe_head(&lock->tickets) ) + tickets.head_tail = arch_fetch_and_add(&t->head_tail, tickets.head_tail); + while ( tickets.tail != observe_head(t) ) { LOCK_PROFILE_BLOCK(block); if ( cb ) @@ -325,18 +328,19 @@ static void always_inline spin_lock_common(spinlock_t *lock, arch_lock_relax(); } arch_lock_acquire_barrier(); - got_lock(&lock->debug); + got_lock(debug); LOCK_PROFILE_GOT(block); } void _spin_lock(spinlock_t *lock) { - spin_lock_common(lock, NULL, NULL); + spin_lock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR, NULL, + NULL); } void _spin_lock_cb(spinlock_t *lock, void (*cb)(void *data), void *data) { - spin_lock_common(lock, cb, data); + spin_lock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR, cb, data); } void _spin_lock_irq(spinlock_t *lock) @@ -355,16 +359,23 @@ unsigned long _spin_lock_irqsave(spinlock_t *lock) return flags; } -void _spin_unlock(spinlock_t *lock) +static void always_inline spin_unlock_common(spinlock_tickets_t *t, + union lock_debug *debug, + struct lock_profile *profile) { LOCK_PROFILE_REL; - rel_lock(&lock->debug); + rel_lock(debug); arch_lock_release_barrier(); - add_sized(&lock->tickets.head, 1); + add_sized(&t->head, 1); arch_lock_signal(); preempt_enable(); } +void _spin_unlock(spinlock_t *lock) +{ + spin_unlock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); +} + void _spin_unlock_irq(spinlock_t *lock) { _spin_unlock(lock); @@ -377,6 +388,11 @@ void _spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) local_irq_restore(flags); } +static bool always_inline spin_is_locked_common(const spinlock_tickets_t *t) +{ + return t->head != t->tail; +} + int _spin_is_locked(const spinlock_t *lock) { /* @@ -385,57 +401,70 @@ int _spin_is_locked(const spinlock_t *lock) * ASSERT()s and alike. */ return lock->recurse_cpu == SPINLOCK_NO_CPU - ? lock->tickets.head != lock->tickets.tail + ? spin_is_locked_common(&lock->tickets) : lock->recurse_cpu == smp_processor_id(); } -int _spin_trylock(spinlock_t *lock) +static bool always_inline spin_trylock_common(spinlock_tickets_t *t, + union lock_debug *debug, + struct lock_profile *profile) { spinlock_tickets_t old, new; preempt_disable(); - check_lock(&lock->debug, true); - old = observe_lock(&lock->tickets); + check_lock(debug, true); + old = observe_lock(t); if ( old.head != old.tail ) { preempt_enable(); - return 0; + return false; } new = old; new.tail++; - if ( cmpxchg(&lock->tickets.head_tail, - old.head_tail, new.head_tail) != old.head_tail ) + if ( cmpxchg(&t->head_tail, old.head_tail, new.head_tail) != old.head_tail ) { preempt_enable(); - return 0; + return false; } /* * cmpxchg() is a full barrier so no need for an * arch_lock_acquire_barrier(). */ - got_lock(&lock->debug); + got_lock(debug); LOCK_PROFILE_GOT(0); - return 1; + return true; } -void _spin_barrier(spinlock_t *lock) +int _spin_trylock(spinlock_t *lock) +{ + return spin_trylock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); +} + +static void always_inline spin_barrier_common(spinlock_tickets_t *t, + union lock_debug *debug, + struct lock_profile *profile) { spinlock_tickets_t sample; LOCK_PROFILE_VAR(block, NOW()); - check_barrier(&lock->debug); + check_barrier(debug); smp_mb(); - sample = observe_lock(&lock->tickets); + sample = observe_lock(t); if ( sample.head != sample.tail ) { - while ( observe_head(&lock->tickets) == sample.head ) + while ( observe_head(t) == sample.head ) arch_lock_relax(); - LOCK_PROFILE_BLKACC(lock->profile, block); + LOCK_PROFILE_BLKACC(profile, block); } smp_mb(); } +void _spin_barrier(spinlock_t *lock) +{ + spin_barrier_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); +} + bool _rspin_trylock(rspinlock_t *lock) { unsigned int cpu = smp_processor_id(); @@ -448,7 +477,8 @@ bool _rspin_trylock(rspinlock_t *lock) if ( likely(lock->recurse_cpu != cpu) ) { - if ( !_spin_trylock(lock) ) + if ( !spin_trylock_common(&lock->tickets, &lock->debug, + LOCK_PROFILE_PAR) ) return false; lock->recurse_cpu = cpu; } @@ -466,7 +496,8 @@ void _rspin_lock(rspinlock_t *lock) if ( likely(lock->recurse_cpu != cpu) ) { - _spin_lock(lock); + spin_lock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR, NULL, + NULL); lock->recurse_cpu = cpu; } @@ -490,7 +521,7 @@ void _rspin_unlock(rspinlock_t *lock) if ( likely(--lock->recurse_cnt == 0) ) { lock->recurse_cpu = SPINLOCK_NO_CPU; - spin_unlock(lock); + spin_unlock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); } } diff --git a/xen/include/xen/spinlock.h b/xen/include/xen/spinlock.h index 53a33653f9..8bc4652526 100644 --- a/xen/include/xen/spinlock.h +++ b/xen/include/xen/spinlock.h @@ -165,6 +165,7 @@ extern void cf_check spinlock_profile_reset(unsigned char key); #else struct lock_profile_qhead { }; +struct lock_profile { }; #define SPIN_LOCK_UNLOCKED { \ .recurse_cpu = SPINLOCK_NO_CPU, \ From patchwork Wed Mar 27 15:22:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AA03C47DD9 for ; Wed, 27 Mar 2024 15:23:00 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698603.1090508 (Exim 4.92) (envelope-from ) id 1rpV77-0008Kg-0T; Wed, 27 Mar 2024 15:22:53 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698603.1090508; Wed, 27 Mar 2024 15:22:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV76-0008KX-TG; Wed, 27 Mar 2024 15:22:52 +0000 Received: by outflank-mailman (input) for mailman id 698603; Wed, 27 Mar 2024 15:22:51 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV75-0007fn-A5 for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:22:51 +0000 Received: from smtp-out2.suse.de (smtp-out2.suse.de [2a07:de40:b251:101:10:150:64:2]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id df0bfaac-ec4d-11ee-a1ef-f123f15fe8a2; Wed, 27 Mar 2024 16:22:49 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 27D865FF27; Wed, 27 Mar 2024 15:22:49 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id DAD1213215; Wed, 27 Mar 2024 15:22:48 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id Rwz7M8g5BGbofQAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:22:48 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: df0bfaac-ec4d-11ee-a1ef-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552969; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vaZf67eZzfX4c/jDX5s3XwvN+K6C+pIsABHCTwjljXU=; b=TnOktoH4wn52MC1Ii+5fZ6E8ajjbpKEoiITkTzpwzcphi/z/bzdqYI/cGQJvNFpzCZI058 TGCv8Bx5GMZukNx8pGJT/7SBGR0S3qlALerCXZPSW9vbRC+chUYPzFT5FlG9Wls2iiVu++ QoAscnta7hBtmy65Hh+b7vU99XnQv9o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552969; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vaZf67eZzfX4c/jDX5s3XwvN+K6C+pIsABHCTwjljXU=; b=TnOktoH4wn52MC1Ii+5fZ6E8ajjbpKEoiITkTzpwzcphi/z/bzdqYI/cGQJvNFpzCZI058 TGCv8Bx5GMZukNx8pGJT/7SBGR0S3qlALerCXZPSW9vbRC+chUYPzFT5FlG9Wls2iiVu++ QoAscnta7hBtmy65Hh+b7vU99XnQv9o= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Jan Beulich , Andrew Cooper , George Dunlap , =?utf-8?q?Roger_Pau_Monn=C3=A9?= , Julien Grall , Stefano Stabellini Subject: [PATCH v6 3/8] xen/spinlock: add missing rspin_is_locked() and rspin_barrier() Date: Wed, 27 Mar 2024 16:22:24 +0100 Message-Id: <20240327152229.25847-4-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b=TnOktoH4 X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-1.51 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:98:from]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; DWL_DNSWL_MED(-2.00)[suse.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; DKIM_TRACE(0.00)[suse.com:+]; MX_GOOD(-0.01)[]; RCPT_COUNT_SEVEN(0.00)[8]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:dkim,suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Rspamd-Queue-Id: 27D865FF27 Add rspin_is_locked() and rspin_barrier() in order to prepare differing spinlock_t and rspinlock_t types. Signed-off-by: Juergen Gross Reviewed-by: Jan Beulich --- V2: - partially carved out from V1 patch, partially new V5: - let rspin_is_locked() return bool (Jan Beulich) V6: - Re-add comment to _spin_is_locked() (Jan Beulich) --- xen/arch/x86/mm/p2m-pod.c | 2 +- xen/common/domain.c | 2 +- xen/common/page_alloc.c | 2 +- xen/common/spinlock.c | 26 ++++++++++++++++++++------ xen/drivers/char/console.c | 4 ++-- xen/drivers/passthrough/pci.c | 2 +- xen/include/xen/spinlock.h | 4 ++++ 7 files changed, 30 insertions(+), 12 deletions(-) diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c index c48ea169b7..9750a3a21b 100644 --- a/xen/arch/x86/mm/p2m-pod.c +++ b/xen/arch/x86/mm/p2m-pod.c @@ -374,7 +374,7 @@ int p2m_pod_empty_cache(struct domain *d) /* After this barrier no new PoD activities can happen. */ BUG_ON(!d->is_dying); - spin_barrier(&p2m->pod.lock.lock); + rspin_barrier(&p2m->pod.lock.lock); lock_page_alloc(p2m); diff --git a/xen/common/domain.c b/xen/common/domain.c index ceb44c8266..282c3ab623 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -991,7 +991,7 @@ int domain_kill(struct domain *d) case DOMDYING_alive: domain_pause(d); d->is_dying = DOMDYING_dying; - spin_barrier(&d->domain_lock); + rspin_barrier(&d->domain_lock); argo_destroy(d); vnuma_destroy(d->vnuma); domain_set_outstanding_pages(d, 0); diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 4d6ce726e3..7c1bdfc046 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -477,7 +477,7 @@ unsigned long domain_adjust_tot_pages(struct domain *d, long pages) { long dom_before, dom_after, dom_claimed, sys_before, sys_after; - ASSERT(spin_is_locked(&d->page_alloc_lock)); + ASSERT(rspin_is_locked(&d->page_alloc_lock)); d->tot_pages += pages; /* diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c index 648393d95f..6572c76114 100644 --- a/xen/common/spinlock.c +++ b/xen/common/spinlock.c @@ -396,13 +396,10 @@ static bool always_inline spin_is_locked_common(const spinlock_tickets_t *t) int _spin_is_locked(const spinlock_t *lock) { /* - * Recursive locks may be locked by another CPU, yet we return - * "false" here, making this function suitable only for use in - * ASSERT()s and alike. + * This function is suitable only for use in ASSERT()s and alike, as it + * doesn't tell _who_ is holding the lock. */ - return lock->recurse_cpu == SPINLOCK_NO_CPU - ? spin_is_locked_common(&lock->tickets) - : lock->recurse_cpu == smp_processor_id(); + return spin_is_locked_common(&lock->tickets); } static bool always_inline spin_trylock_common(spinlock_tickets_t *t, @@ -465,6 +462,23 @@ void _spin_barrier(spinlock_t *lock) spin_barrier_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); } +bool _rspin_is_locked(const rspinlock_t *lock) +{ + /* + * Recursive locks may be locked by another CPU, yet we return + * "false" here, making this function suitable only for use in + * ASSERT()s and alike. + */ + return lock->recurse_cpu == SPINLOCK_NO_CPU + ? spin_is_locked_common(&lock->tickets) + : lock->recurse_cpu == smp_processor_id(); +} + +void _rspin_barrier(rspinlock_t *lock) +{ + spin_barrier_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); +} + bool _rspin_trylock(rspinlock_t *lock) { unsigned int cpu = smp_processor_id(); diff --git a/xen/drivers/char/console.c b/xen/drivers/char/console.c index 22f50fc617..d5e6aacc27 100644 --- a/xen/drivers/char/console.c +++ b/xen/drivers/char/console.c @@ -327,7 +327,7 @@ static void cf_check do_dec_thresh(unsigned char key, bool unused) static void conring_puts(const char *str, size_t len) { - ASSERT(spin_is_locked(&console_lock)); + ASSERT(rspin_is_locked(&console_lock)); while ( len-- ) conring[CONRING_IDX_MASK(conringp++)] = *str++; @@ -765,7 +765,7 @@ static void __putstr(const char *str) { size_t len = strlen(str); - ASSERT(spin_is_locked(&console_lock)); + ASSERT(rspin_is_locked(&console_lock)); console_serial_puts(str, len); video_puts(str, len); diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 4fcc7e2cde..5a446d3dce 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -65,7 +65,7 @@ void pcidevs_unlock(void) bool pcidevs_locked(void) { - return !!spin_is_locked(&_pcidevs_lock); + return rspin_is_locked(&_pcidevs_lock); } static struct radix_tree_root pci_segments; diff --git a/xen/include/xen/spinlock.h b/xen/include/xen/spinlock.h index 8bc4652526..148be1e116 100644 --- a/xen/include/xen/spinlock.h +++ b/xen/include/xen/spinlock.h @@ -297,6 +297,8 @@ void _rspin_lock(rspinlock_t *lock); unsigned long _rspin_lock_irqsave(rspinlock_t *lock); void _rspin_unlock(rspinlock_t *lock); void _rspin_unlock_irqrestore(rspinlock_t *lock, unsigned long flags); +bool _rspin_is_locked(const rspinlock_t *lock); +void _rspin_barrier(rspinlock_t *lock); static always_inline void rspin_lock(rspinlock_t *lock) { @@ -307,6 +309,8 @@ static always_inline void rspin_lock(rspinlock_t *lock) #define rspin_trylock(l) lock_evaluate_nospec(_rspin_trylock(l)) #define rspin_unlock(l) _rspin_unlock(l) #define rspin_unlock_irqrestore(l, f) _rspin_unlock_irqrestore(l, f) +#define rspin_barrier(l) _rspin_barrier(l) +#define rspin_is_locked(l) _rspin_is_locked(l) #define nrspin_trylock(l) spin_trylock(l) #define nrspin_lock(l) spin_lock(l) From patchwork Wed Mar 27 15:22:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 66244C47DD9 for ; Wed, 27 Mar 2024 15:23:05 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698604.1090518 (Exim 4.92) (envelope-from ) id 1rpV7B-0000Id-Al; Wed, 27 Mar 2024 15:22:57 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698604.1090518; Wed, 27 Mar 2024 15:22:57 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7B-0000IU-74; Wed, 27 Mar 2024 15:22:57 +0000 Received: by outflank-mailman (input) for mailman id 698604; Wed, 27 Mar 2024 15:22:56 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7A-0007Qx-0p for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:22:56 +0000 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id e26bc377-ec4d-11ee-afe3-a90da7624cb6; Wed, 27 Mar 2024 16:22:55 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C72AE38B5D; Wed, 27 Mar 2024 15:22:54 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 892FB13215; Wed, 27 Mar 2024 15:22:54 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id hSwEIM45BGYxfgAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:22:54 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e26bc377-ec4d-11ee-afe3-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552974; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SgUlnwwBECkoxNJznR++n3A26InW+paL4mR0WOpyZEc=; b=gwMCVjqS7PK6e7UcFJ+jujzCK+MptUD0Hp5A8HLzkvbVGsea0lam8JcEij1IPOlarwVpZn Z330MppBACterCCnPAPqJoxJA8F4BfzTXanFBYycFPh74nkmK+D5Eb59G0q6FdAV5LNKQv jFpCmLRFS6vk8vSMaxPuotNw2s7VGxk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552974; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SgUlnwwBECkoxNJznR++n3A26InW+paL4mR0WOpyZEc=; b=gwMCVjqS7PK6e7UcFJ+jujzCK+MptUD0Hp5A8HLzkvbVGsea0lam8JcEij1IPOlarwVpZn Z330MppBACterCCnPAPqJoxJA8F4BfzTXanFBYycFPh74nkmK+D5Eb59G0q6FdAV5LNKQv jFpCmLRFS6vk8vSMaxPuotNw2s7VGxk= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH v6 4/8] xen/spinlock: split recursive spinlocks from normal ones Date: Wed, 27 Mar 2024 16:22:25 +0100 Message-Id: <20240327152229.25847-5-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 X-Spamd-Result: default: False [0.70 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] Authentication-Results: smtp-out1.suse.de; none Recursive and normal spinlocks are sharing the same data structure for representation of the lock. This has two major disadvantages: - it is not clear from the definition of a lock, whether it is intended to be used recursive or not, while a mixture of both usage variants needs to be - in production builds (builds without CONFIG_DEBUG_LOCKS) the needed data size of an ordinary spinlock is 8 bytes instead of 4, due to the additional recursion data needed (associated with that the rwlock data is using 12 instead of only 8 bytes) Fix that by introducing a struct spinlock_recursive for recursive spinlocks only, and switch recursive spinlock functions to require pointers to this new struct. This allows to check the correct usage at build time. Signed-off-by: Juergen Gross Reviewed-by: Jan Beulich --- V2: - use shorter names (Jan Beulich) - don't embed spinlock_t in rspinlock_t (Jan Beulich) V5: - some style fixes (Jan Beulich) - bool instead of int (Jan Beulich) --- xen/common/spinlock.c | 50 ++++++++++++++++++++++++++ xen/include/xen/spinlock.h | 72 +++++++++++++++++++++++++++++--------- 2 files changed, 105 insertions(+), 17 deletions(-) diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c index 6572c76114..5aaca49a61 100644 --- a/xen/common/spinlock.c +++ b/xen/common/spinlock.c @@ -545,6 +545,56 @@ void _rspin_unlock_irqrestore(rspinlock_t *lock, unsigned long flags) local_irq_restore(flags); } +bool _nrspin_trylock(rspinlock_t *lock) +{ + check_lock(&lock->debug, true); + + if ( unlikely(lock->recurse_cpu != SPINLOCK_NO_CPU) ) + return false; + + return spin_trylock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); +} + +void _nrspin_lock(rspinlock_t *lock) +{ + spin_lock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR, NULL, + NULL); +} + +void _nrspin_unlock(rspinlock_t *lock) +{ + spin_unlock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); +} + +void _nrspin_lock_irq(rspinlock_t *lock) +{ + ASSERT(local_irq_is_enabled()); + local_irq_disable(); + _nrspin_lock(lock); +} + +void _nrspin_unlock_irq(rspinlock_t *lock) +{ + _nrspin_unlock(lock); + local_irq_enable(); +} + +unsigned long _nrspin_lock_irqsave(rspinlock_t *lock) +{ + unsigned long flags; + + local_irq_save(flags); + _nrspin_lock(lock); + + return flags; +} + +void _nrspin_unlock_irqrestore(rspinlock_t *lock, unsigned long flags) +{ + _nrspin_unlock(lock); + local_irq_restore(flags); +} + #ifdef CONFIG_DEBUG_LOCK_PROFILE struct lock_profile_anc { diff --git a/xen/include/xen/spinlock.h b/xen/include/xen/spinlock.h index 148be1e116..f49ba928f0 100644 --- a/xen/include/xen/spinlock.h +++ b/xen/include/xen/spinlock.h @@ -77,8 +77,6 @@ union lock_debug { }; */ struct spinlock; -/* Temporary hack until a dedicated struct rspinlock is existing. */ -#define rspinlock spinlock struct lock_profile { struct lock_profile *next; /* forward link */ @@ -110,6 +108,10 @@ struct lock_profile_qhead { __used_section(".lockprofile.data") = \ &lock_profile_data__##name #define SPIN_LOCK_UNLOCKED_(x) { \ + .debug = LOCK_DEBUG_, \ + .profile = x, \ +} +#define RSPIN_LOCK_UNLOCKED_(x) { \ .recurse_cpu = SPINLOCK_NO_CPU, \ .debug = LOCK_DEBUG_, \ .profile = x, \ @@ -119,8 +121,9 @@ struct lock_profile_qhead { spinlock_t l = SPIN_LOCK_UNLOCKED_(NULL); \ static struct lock_profile lock_profile_data__##l = LOCK_PROFILE_(l); \ LOCK_PROFILE_PTR_(l) +#define RSPIN_LOCK_UNLOCKED RSPIN_LOCK_UNLOCKED_(NULL) #define DEFINE_RSPINLOCK(l) \ - rspinlock_t l = SPIN_LOCK_UNLOCKED_(NULL); \ + rspinlock_t l = RSPIN_LOCK_UNLOCKED_(NULL); \ static struct lock_profile lock_profile_data__##l = RLOCK_PROFILE_(l); \ LOCK_PROFILE_PTR_(l) @@ -145,8 +148,11 @@ struct lock_profile_qhead { #define spin_lock_init_prof(s, l) \ spin_lock_init_prof__(s, l, lock, spinlock_t, false) -#define rspin_lock_init_prof(s, l) \ - spin_lock_init_prof__(s, l, rlock, rspinlock_t, true) +#define rspin_lock_init_prof(s, l) do { \ + spin_lock_init_prof__(s, l, rlock, rspinlock_t, true); \ + (s)->l.recurse_cpu = SPINLOCK_NO_CPU; \ + (s)->l.recurse_cnt = 0; \ + } while (0) void _lock_profile_register_struct( int32_t type, struct lock_profile_qhead *qhead, int32_t idx); @@ -168,11 +174,14 @@ struct lock_profile_qhead { }; struct lock_profile { }; #define SPIN_LOCK_UNLOCKED { \ + .debug = LOCK_DEBUG_, \ +} +#define RSPIN_LOCK_UNLOCKED { \ .recurse_cpu = SPINLOCK_NO_CPU, \ .debug = LOCK_DEBUG_, \ } #define DEFINE_SPINLOCK(l) spinlock_t l = SPIN_LOCK_UNLOCKED -#define DEFINE_RSPINLOCK(l) rspinlock_t l = SPIN_LOCK_UNLOCKED +#define DEFINE_RSPINLOCK(l) rspinlock_t l = RSPIN_LOCK_UNLOCKED #define spin_lock_init_prof(s, l) spin_lock_init(&((s)->l)) #define rspin_lock_init_prof(s, l) rspin_lock_init(&((s)->l)) @@ -193,6 +202,14 @@ typedef union { #define SPINLOCK_TICKET_INC { .head_tail = 0x10000, } typedef struct spinlock { + spinlock_tickets_t tickets; + union lock_debug debug; +#ifdef CONFIG_DEBUG_LOCK_PROFILE + struct lock_profile *profile; +#endif +} spinlock_t; + +typedef struct rspinlock { spinlock_tickets_t tickets; uint16_t recurse_cpu:SPINLOCK_CPU_BITS; #define SPINLOCK_NO_CPU ((1u << SPINLOCK_CPU_BITS) - 1) @@ -203,12 +220,10 @@ typedef struct spinlock { #ifdef CONFIG_DEBUG_LOCK_PROFILE struct lock_profile *profile; #endif -} spinlock_t; - -typedef spinlock_t rspinlock_t; +} rspinlock_t; #define spin_lock_init(l) (*(l) = (spinlock_t)SPIN_LOCK_UNLOCKED) -#define rspin_lock_init(l) (*(l) = (rspinlock_t)SPIN_LOCK_UNLOCKED) +#define rspin_lock_init(l) (*(l) = (rspinlock_t)RSPIN_LOCK_UNLOCKED) void _spin_lock(spinlock_t *lock); void _spin_lock_cb(spinlock_t *lock, void (*cb)(void *data), void *data); @@ -312,12 +327,35 @@ static always_inline void rspin_lock(rspinlock_t *lock) #define rspin_barrier(l) _rspin_barrier(l) #define rspin_is_locked(l) _rspin_is_locked(l) -#define nrspin_trylock(l) spin_trylock(l) -#define nrspin_lock(l) spin_lock(l) -#define nrspin_unlock(l) spin_unlock(l) -#define nrspin_lock_irq(l) spin_lock_irq(l) -#define nrspin_unlock_irq(l) spin_unlock_irq(l) -#define nrspin_lock_irqsave(l, f) spin_lock_irqsave(l, f) -#define nrspin_unlock_irqrestore(l, f) spin_unlock_irqrestore(l, f) +bool _nrspin_trylock(rspinlock_t *lock); +void _nrspin_lock(rspinlock_t *lock); +#define nrspin_lock_irqsave(l, f) \ + ({ \ + BUILD_BUG_ON(sizeof(f) != sizeof(unsigned long)); \ + (f) = _nrspin_lock_irqsave(l); \ + block_lock_speculation(); \ + }) +unsigned long _nrspin_lock_irqsave(rspinlock_t *lock); +void _nrspin_unlock(rspinlock_t *lock); +void _nrspin_lock_irq(rspinlock_t *lock); +void _nrspin_unlock_irq(rspinlock_t *lock); +void _nrspin_unlock_irqrestore(rspinlock_t *lock, unsigned long flags); + +static always_inline void nrspin_lock(rspinlock_t *lock) +{ + _nrspin_lock(lock); + block_lock_speculation(); +} + +static always_inline void nrspin_lock_irq(rspinlock_t *l) +{ + _nrspin_lock_irq(l); + block_lock_speculation(); +} + +#define nrspin_trylock(l) lock_evaluate_nospec(_nrspin_trylock(l)) +#define nrspin_unlock(l) _nrspin_unlock(l) +#define nrspin_unlock_irqrestore(l, f) _nrspin_unlock_irqrestore(l, f) +#define nrspin_unlock_irq(l) _nrspin_unlock_irq(l) #endif /* __SPINLOCK_H__ */ From patchwork Wed Mar 27 15:22:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD12DC47DD9 for ; Wed, 27 Mar 2024 15:23:11 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698606.1090528 (Exim 4.92) (envelope-from ) id 1rpV7H-0000tg-Q2; Wed, 27 Mar 2024 15:23:03 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698606.1090528; Wed, 27 Mar 2024 15:23:03 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7H-0000s6-Jh; Wed, 27 Mar 2024 15:23:03 +0000 Received: by outflank-mailman (input) for mailman id 698606; Wed, 27 Mar 2024 15:23:02 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7G-0007fn-Br for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:23:02 +0000 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id e5c9d67b-ec4d-11ee-a1ef-f123f15fe8a2; Wed, 27 Mar 2024 16:23:00 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6AC1538B5C; Wed, 27 Mar 2024 15:23:00 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 3075A13215; Wed, 27 Mar 2024 15:23:00 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id 9IV1CtQ5BGZNfgAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:23:00 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e5c9d67b-ec4d-11ee-a1ef-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552980; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c8cgLSy0wask2wusg5yB2CZKTJ/Nmqixk9O34STg5Eo=; b=SvsGD8lexVPB5vjcWzjfUkSBry2GtNCZ1NUCyiG7IJNtYS/JAMnu1TJqM/J/RaoEeEzwG6 3LMuC+1MawWMZbEtFDRIIPSuRdi7olmjORl6ouQq8FAGT+S37E0mKJn4/nrI79oNOCiyaT 8Q0BkBIRKMYtLrLV4NehG6dk9Rx5YaU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552980; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c8cgLSy0wask2wusg5yB2CZKTJ/Nmqixk9O34STg5Eo=; b=SvsGD8lexVPB5vjcWzjfUkSBry2GtNCZ1NUCyiG7IJNtYS/JAMnu1TJqM/J/RaoEeEzwG6 3LMuC+1MawWMZbEtFDRIIPSuRdi7olmjORl6ouQq8FAGT+S37E0mKJn4/nrI79oNOCiyaT 8Q0BkBIRKMYtLrLV4NehG6dk9Rx5YaU= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH v6 5/8] xen/spinlock: let all is_locked and trylock variants return bool Date: Wed, 27 Mar 2024 16:22:26 +0100 Message-Id: <20240327152229.25847-6-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 Authentication-Results: smtp-out1.suse.de; none X-Spamd-Result: default: False [3.70 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[] Switch the remaining trylock and is_locked variants to return bool. Signed-off-by: Juergen Gross Reviewed-by: Jan Beulich --- V5: - new patch (Jan Beulich) --- xen/common/spinlock.c | 4 ++-- xen/include/xen/spinlock.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c index 5aaca49a61..7ccb725171 100644 --- a/xen/common/spinlock.c +++ b/xen/common/spinlock.c @@ -393,7 +393,7 @@ static bool always_inline spin_is_locked_common(const spinlock_tickets_t *t) return t->head != t->tail; } -int _spin_is_locked(const spinlock_t *lock) +bool _spin_is_locked(const spinlock_t *lock) { /* * This function is suitable only for use in ASSERT()s and alike, as it @@ -433,7 +433,7 @@ static bool always_inline spin_trylock_common(spinlock_tickets_t *t, return true; } -int _spin_trylock(spinlock_t *lock) +bool _spin_trylock(spinlock_t *lock) { return spin_trylock_common(&lock->tickets, &lock->debug, LOCK_PROFILE_PAR); } diff --git a/xen/include/xen/spinlock.h b/xen/include/xen/spinlock.h index f49ba928f0..3a4092626c 100644 --- a/xen/include/xen/spinlock.h +++ b/xen/include/xen/spinlock.h @@ -234,8 +234,8 @@ void _spin_unlock(spinlock_t *lock); void _spin_unlock_irq(spinlock_t *lock); void _spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags); -int _spin_is_locked(const spinlock_t *lock); -int _spin_trylock(spinlock_t *lock); +bool _spin_is_locked(const spinlock_t *lock); +bool _spin_trylock(spinlock_t *lock); void _spin_barrier(spinlock_t *lock); static always_inline void spin_lock(spinlock_t *l) From patchwork Wed Mar 27 15:22:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D6CFCC54E67 for ; Wed, 27 Mar 2024 15:23:16 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698608.1090538 (Exim 4.92) (envelope-from ) id 1rpV7N-0001ON-1p; Wed, 27 Mar 2024 15:23:09 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698608.1090538; Wed, 27 Mar 2024 15:23:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7M-0001OE-TK; Wed, 27 Mar 2024 15:23:08 +0000 Received: by outflank-mailman (input) for mailman id 698608; Wed, 27 Mar 2024 15:23:07 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7K-0007Qx-V9 for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:23:06 +0000 Received: from smtp-out2.suse.de (smtp-out2.suse.de [2a07:de40:b251:101:10:150:64:2]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id e9177cc7-ec4d-11ee-afe3-a90da7624cb6; Wed, 27 Mar 2024 16:23:06 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 0E8FD20CDD; Wed, 27 Mar 2024 15:23:06 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id C747A13215; Wed, 27 Mar 2024 15:23:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id qHk9L9k5BGZafgAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:23:05 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e9177cc7-ec4d-11ee-afe3-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552986; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BqLZjhhBHT3mnXz0DltiWo9mb+SXQAswcBtFsIHNjqU=; b=eCXUM4ctFmRMh/J+LAGK+4e1y6KxkqEmjC1ysplW7YQaysPSYjXGtyhqKUD1gvSekLojS1 0OBAg7wu6A+GkGTa26PDu6zJer/uDcsAX2adBHRNJGOXA1RrDK6pZ16WetQp//PapgewKy 0O83/8aW6LWvhU/Wi0CV0c7dPXKkSJc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552986; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BqLZjhhBHT3mnXz0DltiWo9mb+SXQAswcBtFsIHNjqU=; b=eCXUM4ctFmRMh/J+LAGK+4e1y6KxkqEmjC1ysplW7YQaysPSYjXGtyhqKUD1gvSekLojS1 0OBAg7wu6A+GkGTa26PDu6zJer/uDcsAX2adBHRNJGOXA1RrDK6pZ16WetQp//PapgewKy 0O83/8aW6LWvhU/Wi0CV0c7dPXKkSJc= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH v6 6/8] xen/spinlock: support higher number of cpus Date: Wed, 27 Mar 2024 16:22:27 +0100 Message-Id: <20240327152229.25847-7-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 X-Spamd-Result: default: False [3.70 / 50.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_SEVEN(0.00)[7]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_TLS_ALL(0.00)[] Authentication-Results: smtp-out2.suse.de; none Allow 16 bits per cpu number, which is the limit imposed by spinlock_tickets_t. This will allow up to 65535 cpus, while increasing only the size of recursive spinlocks in debug builds from 8 to 12 bytes. The current Xen limit of 4095 cpus is imposed by SPINLOCK_CPU_BITS being 12. There are machines available with more cpus than the current Xen limit, so it makes sense to have the possibility to use more cpus. Signed-off-by: Juergen Gross Reviewed-by: Jan Beulich --- V5: - keep previous recursion limit (Julien Grall) V6: - use unsigned int instead of uint32_t (Jan Beulich) --- xen/common/spinlock.c | 2 ++ xen/include/xen/spinlock.h | 20 ++++++++++---------- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c index 7ccb725171..5aa9ba6188 100644 --- a/xen/common/spinlock.c +++ b/xen/common/spinlock.c @@ -485,7 +485,9 @@ bool _rspin_trylock(rspinlock_t *lock) /* Don't allow overflow of recurse_cpu field. */ BUILD_BUG_ON(NR_CPUS > SPINLOCK_NO_CPU); + BUILD_BUG_ON(SPINLOCK_CPU_BITS > sizeof(lock->recurse_cpu) * 8); BUILD_BUG_ON(SPINLOCK_RECURSE_BITS < 3); + BUILD_BUG_ON(SPINLOCK_MAX_RECURSE > ((1u << SPINLOCK_RECURSE_BITS) - 1)); check_lock(&lock->debug, true); diff --git a/xen/include/xen/spinlock.h b/xen/include/xen/spinlock.h index 3a4092626c..db00a24646 100644 --- a/xen/include/xen/spinlock.h +++ b/xen/include/xen/spinlock.h @@ -8,16 +8,16 @@ #include #include -#define SPINLOCK_CPU_BITS 12 +#define SPINLOCK_CPU_BITS 16 #ifdef CONFIG_DEBUG_LOCKS union lock_debug { - uint16_t val; -#define LOCK_DEBUG_INITVAL 0xffff + uint32_t val; +#define LOCK_DEBUG_INITVAL 0xffffffff struct { - uint16_t cpu:SPINLOCK_CPU_BITS; -#define LOCK_DEBUG_PAD_BITS (14 - SPINLOCK_CPU_BITS) - uint16_t :LOCK_DEBUG_PAD_BITS; + unsigned int cpu:SPINLOCK_CPU_BITS; +#define LOCK_DEBUG_PAD_BITS (30 - SPINLOCK_CPU_BITS) + unsigned int :LOCK_DEBUG_PAD_BITS; bool irq_safe:1; bool unseen:1; }; @@ -211,11 +211,11 @@ typedef struct spinlock { typedef struct rspinlock { spinlock_tickets_t tickets; - uint16_t recurse_cpu:SPINLOCK_CPU_BITS; + uint16_t recurse_cpu; #define SPINLOCK_NO_CPU ((1u << SPINLOCK_CPU_BITS) - 1) -#define SPINLOCK_RECURSE_BITS (16 - SPINLOCK_CPU_BITS) - uint16_t recurse_cnt:SPINLOCK_RECURSE_BITS; -#define SPINLOCK_MAX_RECURSE ((1u << SPINLOCK_RECURSE_BITS) - 1) +#define SPINLOCK_RECURSE_BITS 8 + uint8_t recurse_cnt; +#define SPINLOCK_MAX_RECURSE 15 union lock_debug debug; #ifdef CONFIG_DEBUG_LOCK_PROFILE struct lock_profile *profile; From patchwork Wed Mar 27 15:22:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 36F7FC54E67 for ; Wed, 27 Mar 2024 15:23:21 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698613.1090548 (Exim 4.92) (envelope-from ) id 1rpV7R-0001nr-8u; Wed, 27 Mar 2024 15:23:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698613.1090548; Wed, 27 Mar 2024 15:23:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7R-0001ng-5D; Wed, 27 Mar 2024 15:23:13 +0000 Received: by outflank-mailman (input) for mailman id 698613; Wed, 27 Mar 2024 15:23:12 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7Q-0007Qx-PB for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:23:12 +0000 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id ec779207-ec4d-11ee-afe3-a90da7624cb6; Wed, 27 Mar 2024 16:23:12 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id A84AC38B63; Wed, 27 Mar 2024 15:23:11 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 6FBC813215; Wed, 27 Mar 2024 15:23:11 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id +KvmGd85BGaWfgAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:23:11 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ec779207-ec4d-11ee-afe3-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552991; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oXjtiiPFLzYFznlWn/C8ZCBcRUde7S7BFmz8pb92S3c=; b=YFINieC/XziGqNKvYCJxQsWesX9OVovupGp2FPfPA9gmhSpr5N84y4kzTsF4iZbAptFuUZ xtciKUEtp8KiqhKSYoUn+5m9GIXPXHJHy5ymQYP7rr5g4qPH1mOjC2UQu4MCS24jiPq5Ev HOHWXU2XLs9a1zTsQX62+JfG6jEJQ5c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552991; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oXjtiiPFLzYFznlWn/C8ZCBcRUde7S7BFmz8pb92S3c=; b=YFINieC/XziGqNKvYCJxQsWesX9OVovupGp2FPfPA9gmhSpr5N84y4kzTsF4iZbAptFuUZ xtciKUEtp8KiqhKSYoUn+5m9GIXPXHJHy5ymQYP7rr5g4qPH1mOjC2UQu4MCS24jiPq5Ev HOHWXU2XLs9a1zTsQX62+JfG6jEJQ5c= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH v6 7/8] xen/rwlock: raise the number of possible cpus Date: Wed, 27 Mar 2024 16:22:28 +0100 Message-Id: <20240327152229.25847-8-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 X-Spamd-Bar: + Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b="YFINieC/" X-Spamd-Result: default: False [1.49 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; DWL_DNSWL_MED(-2.00)[suse.com:dkim]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; DKIM_TRACE(0.00)[suse.com:+]; MX_GOOD(-0.01)[]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:dkim,suse.com:email]; NEURAL_HAM_LONG(-1.00)[-1.000]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RECEIVED_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:106:10:150:64:167:received]; RCVD_TLS_ALL(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:98:from] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Queue-Id: A84AC38B63 The rwlock handling is limiting the number of cpus to 4095 today. The main reason is the use of the atomic_t data type for the main lock handling, which needs 2 bits for the locking state (writer waiting or write locked), 12 bits for the id of a possible writer, and a 12 bit counter for readers. The limit isn't 4096 due to an off by one sanity check. The atomic_t data type is 32 bits wide, so in theory 15 bits for the writer's cpu id and 15 bits for the reader count seem to be fine, but via read_trylock() more readers than cpus are possible. This means that it is possible to raise the number of cpus to 16384 without changing the rwlock_t data structure. In order to avoid the reader count wrapping to zero, don't let read_trylock() succeed in case the highest bit of the reader's count is set already. This leaves enough headroom for non-recursive readers to enter without risking a wrap. While at it calculate _QW_CPUMASK and _QR_SHIFT from _QW_SHIFT and add a sanity check for not overflowing the atomic_t data type. Signed-off-by: Juergen Gross Reviewed-by: Jan Beulich --- V5: - new patch V6: - add comment to _can_read_lock() (Jan Beulich) --- xen/include/xen/rwlock.h | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/xen/include/xen/rwlock.h b/xen/include/xen/rwlock.h index 65d88b0ef4..232782801d 100644 --- a/xen/include/xen/rwlock.h +++ b/xen/include/xen/rwlock.h @@ -23,12 +23,12 @@ typedef struct { #define rwlock_init(l) (*(l) = (rwlock_t)RW_LOCK_UNLOCKED) /* Writer states & reader shift and bias. */ -#define _QW_CPUMASK 0xfffU /* Writer CPU mask */ -#define _QW_SHIFT 12 /* Writer flags shift */ -#define _QW_WAITING (1U << _QW_SHIFT) /* A writer is waiting */ -#define _QW_LOCKED (3U << _QW_SHIFT) /* A writer holds the lock */ -#define _QW_WMASK (3U << _QW_SHIFT) /* Writer mask */ -#define _QR_SHIFT 14 /* Reader count shift */ +#define _QW_SHIFT 14 /* Writer flags shift */ +#define _QW_CPUMASK ((1U << _QW_SHIFT) - 1) /* Writer CPU mask */ +#define _QW_WAITING (1U << _QW_SHIFT) /* A writer is waiting */ +#define _QW_LOCKED (3U << _QW_SHIFT) /* A writer holds the lock */ +#define _QW_WMASK (3U << _QW_SHIFT) /* Writer mask */ +#define _QR_SHIFT (_QW_SHIFT + 2) /* Reader count shift */ #define _QR_BIAS (1U << _QR_SHIFT) void queue_read_lock_slowpath(rwlock_t *lock); @@ -36,14 +36,21 @@ void queue_write_lock_slowpath(rwlock_t *lock); static inline bool _is_write_locked_by_me(unsigned int cnts) { - BUILD_BUG_ON(_QW_CPUMASK < NR_CPUS); + BUILD_BUG_ON((_QW_CPUMASK + 1) < NR_CPUS); + BUILD_BUG_ON(NR_CPUS * _QR_BIAS > INT_MAX); return (cnts & _QW_WMASK) == _QW_LOCKED && (cnts & _QW_CPUMASK) == smp_processor_id(); } static inline bool _can_read_lock(unsigned int cnts) { - return !(cnts & _QW_WMASK) || _is_write_locked_by_me(cnts); + /* + * If write locked by the caller, no other readers are possible. + * Not allowing the lock holder to read_lock() another 32768 times ought + * to be fine. + */ + return cnts <= INT_MAX && + (!(cnts & _QW_WMASK) || _is_write_locked_by_me(cnts)); } /* From patchwork Wed Mar 27 15:22:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?SsO8cmdlbiBHcm/Dnw==?= X-Patchwork-Id: 13606802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B682BC47DD9 for ; Wed, 27 Mar 2024 15:23:26 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.698614.1090558 (Exim 4.92) (envelope-from ) id 1rpV7Y-0002RK-Gf; Wed, 27 Mar 2024 15:23:20 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 698614.1090558; Wed, 27 Mar 2024 15:23:20 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7Y-0002R8-D0; Wed, 27 Mar 2024 15:23:20 +0000 Received: by outflank-mailman (input) for mailman id 698614; Wed, 27 Mar 2024 15:23:19 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rpV7X-0007fn-3P for xen-devel@lists.xenproject.org; Wed, 27 Mar 2024 15:23:19 +0000 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id efc82369-ec4d-11ee-a1ef-f123f15fe8a2; Wed, 27 Mar 2024 16:23:17 +0100 (CET) Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 48BE320CE4; Wed, 27 Mar 2024 15:23:17 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 0EAD313215; Wed, 27 Mar 2024 15:23:17 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap2.dmz-prg2.suse.org with ESMTPSA id wSkuAuU5BGbIfgAAn2gu4w (envelope-from ); Wed, 27 Mar 2024 15:23:17 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: efc82369-ec4d-11ee-a1ef-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552997; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=svDhVGjKFzIPpcWjFYARn/2rWUfeidU+xfRiYlSjnX4=; b=hchFnFTpmvg2xILUuW8mlqB2nh83QyCO/EmARie/ZORSYtuJnBUo/zyJLY2HjVHqx09Elz aMaCJd0+9xLNObzCsolh41txPSQCTP0SXoMvzmysaK9xrz4NosL+/bITjscww81oqY4oMf 9BN6qdzi0893vEzcaiyNbLNQnCouXuQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1711552997; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=svDhVGjKFzIPpcWjFYARn/2rWUfeidU+xfRiYlSjnX4=; b=hchFnFTpmvg2xILUuW8mlqB2nh83QyCO/EmARie/ZORSYtuJnBUo/zyJLY2HjVHqx09Elz aMaCJd0+9xLNObzCsolh41txPSQCTP0SXoMvzmysaK9xrz4NosL+/bITjscww81oqY4oMf 9BN6qdzi0893vEzcaiyNbLNQnCouXuQ= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini Subject: [PATCH v6 8/8] xen: allow up to 16383 cpus Date: Wed, 27 Mar 2024 16:22:29 +0100 Message-Id: <20240327152229.25847-9-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20240327152229.25847-1-jgross@suse.com> References: <20240327152229.25847-1-jgross@suse.com> MIME-Version: 1.0 X-Spamd-Result: default: False [3.70 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[] Authentication-Results: smtp-out2.suse.de; none With lock handling now allowing up to 16384 cpus (spinlocks can handle 65535 cpus, rwlocks can handle 16384 cpus), raise the allowed limit for the number of cpus to be configured to 16383. The new limit is imposed by IOMMU_CMD_BUFFER_MAX_ENTRIES and QINVAL_MAX_ENTRY_NR required to be larger than 2 * CONFIG_NR_CPUS. Signed-off-by: Juergen Gross Acked-by: Jan Beulich --- V5: - new patch (Jan Beulich) --- xen/arch/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig index 67ba38f32f..308ce129a8 100644 --- a/xen/arch/Kconfig +++ b/xen/arch/Kconfig @@ -6,7 +6,7 @@ config PHYS_ADDR_T_32 config NR_CPUS int "Maximum number of CPUs" - range 1 4095 + range 1 16383 default "256" if X86 default "8" if ARM && RCAR3 default "4" if ARM && QEMU