From patchwork Tue Jun 8 11:45:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12306661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0255C47082 for ; Tue, 8 Jun 2021 12:17:07 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A92D36124C for ; Tue, 8 Jun 2021 12:17:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A92D36124C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Fmkowi0LfGmSTJcHE/n8CKKNVVYguvyLyRwjtJM7utw=; b=ezcQjlkuFNiazDyCMM+tVvquoC 1jNhqYYV+NTWYlBUDsLguNS/XJK+YkaH7ixFsOoW2/3eOyMUboOueYcyyl6U73zpBX/uf2OwksnVQ XemkQl8hVUxnFx6eEUHETECm4zMlyZg+f2tTpCBO8/Haidz2jwiRvLDWnuRFJKQzlqPU1cyUei8t6 q1saN0hXOkmJ73BkQfFkrJkqrt/9QNcMFXH0oH7eg2k5L8wqPc8QkVxMMXAYcZQN545cLvgUf3e3L y+TiMNUUXHHubsC6yUaa5CbDvZP56EYeTEqD+Df5ZuO47uIxYIO7u71jXnjyKTtsVLRovl2rWm+LN Y0OmyI+A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lqad2-008Ghy-16; Tue, 08 Jun 2021 12:14:45 +0000 Received: from mail-ej1-f74.google.com ([209.85.218.74]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lqaBh-0086cX-Ko for linux-arm-kernel@lists.infradead.org; Tue, 08 Jun 2021 11:46:31 +0000 Received: by mail-ej1-f74.google.com with SMTP id hz18-20020a1709072cf2b02903fbaae9f4faso6653591ejc.4 for ; Tue, 08 Jun 2021 04:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vxwHJ+G94OJ1pE095VNOnJheGz25iQDynuEQjCESIh0=; b=qaed+O4Pp9KTEzvs+APK7fEuJPXtpKOQgrKsvMynvTCDksxLejAvbkSsyykvk7D8sS S8yR429zh5Q4ejjDIGva2qL7h2gIADTNFNMYwCoCH7pWsJg+c/D/RQF4oqP7Fgp4StAW +40ljDrP3JbD1vmJYd9ufdc85M0oGXSwb7Z6GmmDI1I56uYN3310sFRXr2EprFL06o0R H/ApsliuhwPxG8ykxG+ifXKqhtcBXzStm7uLNdf1Z+UozesEAohJUpYCFAbHR6eqQg77 4YmpUv4RainpfmuH97cTwY0tAH62FWahweNsmaxYoZcCa3x1nBeGaKijk6TxnmFCR4di XHAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vxwHJ+G94OJ1pE095VNOnJheGz25iQDynuEQjCESIh0=; b=eu4kyU6sq0RCyaPXvZH0jw3pMzZ42nfiwTx2UD4Vb4Vq8fdZMd+CDanYvRDpmYYTkL cTpM4QmT8SFwvIOdn2NJJm0XN5nA7R4J5imtKqm8tLZ7+BZyrlJ+riMU4iABf8EX057r lhffMtDsnSs+eTo8vRAtc3XTTtuaQuBq4L1mpO35fEk9W0SGU6YvuaELRkOEWzNyNJ0d CTJJCf9OwLfX61biTuD+KUEq6raQHBQ9Wx8da3GIIMi+lRbb9m+BV1cLo9vD+YcCUeNy DU+HlKBfGsz81HzGIKPHCx6Fv8HEOT5U7ioF+xwNLCKsHcAzYP+JRpKwn5PW8Qgqzc2I 18ug== X-Gm-Message-State: AOAM530zl8DDx/4OUwJ1PFAHj5KhH6EkR1gx9JCTKqsL1ne2a67TSca4 siIu46VH1dMkHi/TPQGG8QXdLRpajtw1 X-Google-Smtp-Source: ABdhPJxjy7h6qUzJMJc1Bbkt8we2RleHBUTUerWb3KW3/8CCh8UqtZ6HhCYhLc5ZOrmlQyHAQHfiTn44GB2u X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a17:906:a04e:: with SMTP id bg14mr6427415ejb.366.1623152726119; Tue, 08 Jun 2021 04:45:26 -0700 (PDT) Date: Tue, 8 Jun 2021 11:45:12 +0000 In-Reply-To: <20210608114518.748712-1-qperret@google.com> Message-Id: <20210608114518.748712-2-qperret@google.com> Mime-Version: 1.0 References: <20210608114518.748712-1-qperret@google.com> X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH v3 1/7] KVM: arm64: Move hyp_pool locking out of refcount helpers From: Quentin Perret To: maz@kernel.org, will@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, catalin.marinas@arm.com, suzuki.poulose@arm.com Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210608_044629_728703_EAA071C0 X-CRM114-Status: GOOD ( 21.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The hyp_page refcount helpers currently rely on the hyp_pool lock for serialization. However, this means the refcounts can't be changed from the buddy allocator core as it already holds the lock, which means pages have to go through odd transient states. For example, when a page is freed, its refcount is set to 0, and the lock is transiently released before the page can be attached to a free list in the buddy tree. This is currently harmless as the allocator checks the list node of each page to see if it is available for allocation or not, but it means the page refcount can't be trusted to represent the state of the page even if the pool lock is held. In order to fix this, remove the pool locking from the refcount helpers, and move all the logic to the buddy allocator. This will simplify the removal of the list node from struct hyp_page in a later patch. Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/gfp.h | 35 ---------------------- arch/arm64/kvm/hyp/nvhe/page_alloc.c | 43 ++++++++++++++++++++------- 2 files changed, 32 insertions(+), 46 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/gfp.h b/arch/arm64/kvm/hyp/include/nvhe/gfp.h index 18a4494337bd..f2c84e4fa40f 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/gfp.h +++ b/arch/arm64/kvm/hyp/include/nvhe/gfp.h @@ -22,41 +22,6 @@ struct hyp_pool { unsigned int max_order; }; -static inline void hyp_page_ref_inc(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - p->refcount++; - hyp_spin_unlock(&pool->lock); -} - -static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - int ret; - - hyp_spin_lock(&pool->lock); - p->refcount--; - ret = (p->refcount == 0); - hyp_spin_unlock(&pool->lock); - - return ret; -} - -static inline void hyp_set_page_refcounted(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - if (p->refcount) { - hyp_spin_unlock(&pool->lock); - BUG(); - } - p->refcount = 1; - hyp_spin_unlock(&pool->lock); -} - /* Allocation */ void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order); void hyp_get_page(void *addr); diff --git a/arch/arm64/kvm/hyp/nvhe/page_alloc.c b/arch/arm64/kvm/hyp/nvhe/page_alloc.c index 237e03bf0cb1..d666f4789e31 100644 --- a/arch/arm64/kvm/hyp/nvhe/page_alloc.c +++ b/arch/arm64/kvm/hyp/nvhe/page_alloc.c @@ -93,15 +93,6 @@ static void __hyp_attach_page(struct hyp_pool *pool, list_add_tail(&p->node, &pool->free_area[order]); } -static void hyp_attach_page(struct hyp_page *p) -{ - struct hyp_pool *pool = hyp_page_to_pool(p); - - hyp_spin_lock(&pool->lock); - __hyp_attach_page(pool, p); - hyp_spin_unlock(&pool->lock); -} - static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, struct hyp_page *p, unsigned int order) @@ -125,19 +116,49 @@ static struct hyp_page *__hyp_extract_page(struct hyp_pool *pool, return p; } +static inline void hyp_page_ref_inc(struct hyp_page *p) +{ + p->refcount++; +} + +static inline int hyp_page_ref_dec_and_test(struct hyp_page *p) +{ + p->refcount--; + return (p->refcount == 0); +} + +static inline void hyp_set_page_refcounted(struct hyp_page *p) +{ + BUG_ON(p->refcount); + p->refcount = 1; +} + +/* + * Changes to the buddy tree and page refcounts must be done with the hyp_pool + * lock held. If a refcount change requires an update to the buddy tree (e.g. + * hyp_put_page()), both operations must be done within the same critical + * section to guarantee transient states (e.g. a page with null refcount but + * not yet attached to a free list) can't be observed by well-behaved readers. + */ void hyp_put_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); if (hyp_page_ref_dec_and_test(p)) - hyp_attach_page(p); + __hyp_attach_page(pool, p); + hyp_spin_unlock(&pool->lock); } void hyp_get_page(void *addr) { struct hyp_page *p = hyp_virt_to_page(addr); + struct hyp_pool *pool = hyp_page_to_pool(p); + hyp_spin_lock(&pool->lock); hyp_page_ref_inc(p); + hyp_spin_unlock(&pool->lock); } void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) @@ -159,8 +180,8 @@ void *hyp_alloc_pages(struct hyp_pool *pool, unsigned int order) p = list_first_entry(&pool->free_area[i], struct hyp_page, node); p = __hyp_extract_page(pool, p, order); - hyp_spin_unlock(&pool->lock); hyp_set_page_refcounted(p); + hyp_spin_unlock(&pool->lock); return hyp_page_to_virt(p); }