From patchwork Tue Feb 20 07:29:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13563529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0ADFCC48BC3 for ; Tue, 20 Feb 2024 07:30:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C0286B0093; Tue, 20 Feb 2024 02:30:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 970196B0095; Tue, 20 Feb 2024 02:30:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 811026B0096; Tue, 20 Feb 2024 02:30:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 708476B0093 for ; Tue, 20 Feb 2024 02:30:46 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1345D8045E for ; Tue, 20 Feb 2024 07:30:46 +0000 (UTC) X-FDA: 81811360092.29.270559F Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2046.outbound.protection.outlook.com [40.107.95.46]) by imf17.hostedemail.com (Postfix) with ESMTP id 211FD40018 for ; Tue, 20 Feb 2024 07:30:42 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=P8dxjSOJ; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf17.hostedemail.com: domain of ankita@nvidia.com designates 40.107.95.46 as permitted sender) smtp.mailfrom=ankita@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708414243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GEUPBUp/mRSrwoBY6hHPxBFvFm1Cd4dE+qw16gP2bsw=; b=6n41FwFNXPGSi4GzpvJJilCP4/03mezfIUtAa2pymqJ1XccClhaEdM1oQDYLMwVuF/lH1k SEXxKvdrbF0DXUeVSLw0H8G7qglHpPFp1WfBEPr/hWrMAyDZH62SBcJrcs+Uo/pgdVNkDU odv5SjHFQKK+keVcCWTjuuOWlIzsbow= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=P8dxjSOJ; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf17.hostedemail.com: domain of ankita@nvidia.com designates 40.107.95.46 as permitted sender) smtp.mailfrom=ankita@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1708414243; a=rsa-sha256; cv=pass; b=VVwdIX06z7jzLR4MF1Oa2AdVepBQqkYc0w3UqxFWBQRi3A25f2H3l3vGWm2lQdr9liqEzm 6VL3wQgQHmEybkiIOJP1ePgRNrH7cEvLyERPRg2Y+BfzafxU+TCyIdRwzBQoyIHCNWwJ7j jvKnF+eXnEOUG7abuONI214km8CEahQ= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=inHGrvtncYNmQA0SSNYxz2O9Zv4pzyhYuqjV5gRztnEyTs4s8y9LUq/iNezPzPBe646uD6A6taJRvUBWZ4uN4uQk6hfVe5830VO2iS15aDSHFoi9TM0jfC+l8WYZ+NxrhZJ46dfxlq3jHCH8m/aiPoESaK5TPuD2Fb3ueF2mqdr5FQHtFxn1Rw+eIgqp+WohIeM8dtQAx7AfwaSAw2WnS8LEhwu/Ncfbcdc68UJOm2sEhWS7cICnuCmIHLzAD10DKsdUixiK0lID+i4NTKspMsHftZhtzMTAwCmVkEv+1IhmXQhEQVRxmc6DU1QxSrL3k+ewsRtzooPI0xF5SY70+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GEUPBUp/mRSrwoBY6hHPxBFvFm1Cd4dE+qw16gP2bsw=; b=hnWmGRlpLQnaapNYoqzg0r1UGJHhLD41vPBwj3HOT9GVfyiVs7uYWggCPy51WHE2Mp1laWHi7SebDiviTdMs6lCs2sbhWV9PVDqSEEyBAPqhKLnU7uzq2bovCwAd0NwdhrhxJ1vF+1Q70/zM40T/3wwOXKgVu5C8rm+wwqzsW8I17S609Qk9yogE7CW8IcbLaRxCTcAnbYvYhnSSuNMPXiUIBNK+JiB6n9UquhdgchvxBUV9gvA3fS9JBuZdbIqxjyeEvXzgKOYH5V/37BopYTaKN0NwLqzcHqpEl7E7plfbdAsVLLPzz+bymtbyt2vpv1S4Rak8TONLbHckJQRPfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GEUPBUp/mRSrwoBY6hHPxBFvFm1Cd4dE+qw16gP2bsw=; b=P8dxjSOJyVuRH67AcNtP8cxTAJ3hb/a574V4QUpha6qAi0g3+Rfa7ox+fOckXoDBSc3tAZdvA6bdLdStehJXj2WpNIu1W9rM1N8KK/juHRssHlS+yqzrtC8yN6T6Yd2kJItC3uiZJJ79FCUbbcqMse/kNwLwJ0UFOj7JYJddZIwkglA6xXkTcxgolnIh9BRjzNNcAVrDojl+SGm/UnD/Fv+H1lzBJ1OxY3ToP+mwh9zt4Qq837b8hG9dBfY1oJOhOplTk7epkEdPOhwb10k8NIUdps1M3ILFmCtDNOLEIBgDRvuqGt5K2dCNnci2Br5UpU+5f2HUgU2WeoN2vWsO0Q== Received: from CY8PR22CA0015.namprd22.prod.outlook.com (2603:10b6:930:45::14) by CY8PR12MB8242.namprd12.prod.outlook.com (2603:10b6:930:77::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.19; Tue, 20 Feb 2024 07:30:40 +0000 Received: from CY4PEPF0000FCBE.namprd03.prod.outlook.com (2603:10b6:930:45:cafe::93) by CY8PR22CA0015.outlook.office365.com (2603:10b6:930:45::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.39 via Frontend Transport; Tue, 20 Feb 2024 07:30:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000FCBE.mail.protection.outlook.com (10.167.242.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.25 via Frontend Transport; Tue, 20 Feb 2024 07:30:40 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 19 Feb 2024 23:30:23 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Mon, 19 Feb 2024 23:30:23 -0800 Received: from sgarnayak-dt.nvidia.com (10.127.8.9) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12 via Frontend Transport; Mon, 19 Feb 2024 23:30:09 -0800 From: To: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , CC: , , , , , , , , , , , , , , Subject: [PATCH v8 3/4] kvm: arm64: set io memory s2 pte as normalnc for vfio pci device Date: Tue, 20 Feb 2024 12:59:25 +0530 Message-ID: <20240220072926.6466-4-ankita@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240220072926.6466-1-ankita@nvidia.com> References: <20240220072926.6466-1-ankita@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBE:EE_|CY8PR12MB8242:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e7e5fe0-ac1e-4e0f-2755-08dc31e5d6ee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qpLJyNWq3sY6o8aGbW1uTCBZ8xMGTbned0cGf7WeEH649MhkBsfFNv0F7bD3Id2MbGCRcYtBt/O+OCgSQvpxiGkEqMNiLFOFC1knnfjSOgo62mubXO0I9QrInBippSz1B+VxQfaAsjeigvx3uIFKUVNhgZTiJjaHFYeoFUfygNbL4nFd/9bmeguhtB7rqwgO10TLXsU1YJOIVXN3uWb0aaovquG3OoyohEkonZdQMe9zYdjwaXRww21Yu+JYCcAiyYHZVVGl5JV+iqfD6F8nZ9pLFpFn4BkSDohG5Omp9ctgFNjTtPRMhm7A0wj0UDpCYHAVO9BECoQQv+IeTfZ/lNUZfvkhxmAzW0p4gIfwuBqlyjnTcX08Jz3aQLh3/fpI9SLO6gltqtUTyonPNq0BqKGi+BVwhPO1Y3gtyk2J06ArR/XtpWt8rjAswBQAOemh4F97qts01urpAXc7VKIm3xckVITWFu8fyjQOUcD3XCXeiCi1RGREAb11otFuINznz48sE7eCvLIqzmfNu60mBWX8mjNtnfyTEHfXIJ1zIcWiSSZ48qW88NgdWSxQl3Ba+K+DAq2Fy5/tm0Q6CE04dCdqlUMcmra8D6QbMakX3QNnso2xWvvB0vdmzGCo+z3jhWezXmRKFRHYfBLUBdYgcHzIYkdqpoQK4QeSK5Hpjek= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(36860700004)(40470700004)(46966006)(921011);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 07:30:40.1721 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e7e5fe0-ac1e-4e0f-2755-08dc31e5d6ee X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBE.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8242 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 211FD40018 X-Stat-Signature: rimrmnncc7ynhdijuhutyo916eztheg4 X-HE-Tag: 1708414242-151140 X-HE-Meta: U2FsdGVkX1/X6GOOs2OhtTOtYakjRWf1gasPwVSvsXmIJpeJ4uDpYEn1wcwfFNM4u+tTfx1BiLTeGKj3SWGNDPOTWwszGpWwSk5gTjTOQnACTFlDEm3qrBhEObHaxSa7EyDlcMFEIgAbB6pUg1xKYioqvi1OgDtwFLCAsJ/4wUC5rW/vTdrIneLsPQaBiwwUlNlPihbf7QIdQjaaa/d9dJAoSNJXqVcKpR3Wo5CgM4ya7X6Lh2/jrHxUarARMEXBhaHO11qZHC6n9F5jDyVUZ+/NmVDYcSfA8K3OlEAwyKxdONYq5EChySux2jlf9QzYcYBDVADvW6awZH/5MHKPZVGwy6hBKVc1vhnUrTZBNj6ANXgWieIxQsWbKJQte411pEt+xYmtK2i2zdJAhjpg+Z7pJUOVDo+aMAR3QTdfwTm1w1BSoRB6I6g3TZrWSrc36C48nfPPWH0qCh0R+81bn+3ehdRu4tLft1V7F5pH9OLDBZl6DERzAJs7oiNLlIAa4s9d4p9l0LS/q35GEVa9Ce7jJMUPhfnZBHI1Z5oHDGJwrRWiyCbGUqfHD+H4OrR0YOZkZ3zRglUi2LpMc0zCD4K/weAY2+CCUbXrqjULMghhlS1m/kpCwOjNXfab4NvdXLnB/WtH/8VayadmI6f5GHYXpnhXE0BPDkc1XpGDBD93P3XKsi+q2LbM1agV6TTyJgEumdR8KZox5kuiNFticjCwi3VYOwBCLmu91MYNOD/fSlQ6hS40TWJVCeRFaaCTkfRFBdfi+VeOUVlzpoQi5AH321V+4MR4qKBHgfwBv9CuUASB5/xExe3eSsJ8aVOWFK4+5Amld6QoNG7Y5uOUuOlU93n2s7ReGj3bluveBUo7Xs3sVvySgBOrPA7GmGOP1NTnhR4UZn+QLsK10QkNsgWeruPMP2yWCtLJtyGea1MyHYYFoLmN9O8Y4ZHQ20CU2Iq8998rdVAHQHI90RR 4phAerjw LkhaogzngsUc4NMSsEe5xamovOh6gS7Srr/cxKDe08HAzGnBp7Nw1h7E4IPHSuKEUekN6SFh4lxzhuHY2+nnlzZEaz2S8/04ENTee14gOFvdzsrr9nLsz1KZ/Bg6SgtfLKs+Elu+Vumi1JvDPkg11AJzmN80TD5vAULE4d2CYucLS8+5oFDcr8/6q75hq0CcjRgEqiPJvnM9ZR/IZ+pxLo6ELVAmFGLf7yaOJa2xiR1pyyMMFofY43TkGdxLumITPXnYdaneLwtZOEv6mwbcYqN6Z6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Ankit Agrawal To provide VM with the ability to get device IO memory with NormalNC property, map device MMIO in KVM for ARM64 at stage2 as NormalNC. Having NormalNC S2 default puts guests in control (based on [1], "Combining stage 1 and stage 2 memory type attributes") of device MMIO regions memory mappings. The rules are summarized below: ([(S1) - stage1], [(S2) - stage 2]) S1 | S2 | Result NORMAL-WB | NORMAL-NC | NORMAL-NC NORMAL-WT | NORMAL-NC | NORMAL-NC NORMAL-NC | NORMAL-NC | NORMAL-NC DEVICE | NORMAL-NC | DEVICE Still this cannot be generalized to non PCI devices such as GICv2. There is insufficient information and uncertainity in the behavior of non PCI driver. A driver must indicate support using the new flag VM_ALLOW_ANY_UNCACHED. Adapt KVM to make use of the flag VM_ALLOW_ANY_UNCACHED as indicator to activate the S2 setting to NormalNc. [1] section D8.5.5 of DDI0487J_a_a-profile_architecture_reference_manual.pdf Suggested-by: Catalin Marinas Acked-by: Jason Gunthorpe Reviewed-by: Catalin Marinas Signed-off-by: Ankit Agrawal --- arch/arm64/kvm/mmu.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d14504821b79..1742fdccb432 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1381,7 +1381,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, int ret = 0; bool write_fault, writable, force_pte = false; bool exec_fault, mte_allowed; - bool device = false; + bool device = false, vfio_allow_any_uc = false; unsigned long mmu_seq; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; @@ -1472,6 +1472,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn = fault_ipa >> PAGE_SHIFT; mte_allowed = kvm_vma_mte_allowed(vma); + vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED; + /* Don't use the VMA after the unlock -- it may have vanished */ vma = NULL; @@ -1557,10 +1559,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault) prot |= KVM_PGTABLE_PROT_X; - if (device) - prot |= KVM_PGTABLE_PROT_DEVICE; - else if (cpus_have_final_cap(ARM64_HAS_CACHE_DIC)) + if (device) { + if (vfio_allow_any_uc) + prot |= KVM_PGTABLE_PROT_NORMAL_NC; + else + prot |= KVM_PGTABLE_PROT_DEVICE; + } else if (cpus_have_final_cap(ARM64_HAS_CACHE_DIC)) { prot |= KVM_PGTABLE_PROT_X; + } /* * Under the premise of getting a FSC_PERM fault, we just need to relax