From patchwork Thu Oct 10 10:31:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Micha=C5=82_Winiarski?= X-Patchwork-Id: 13829904 X-Patchwork-Delegate: bhelgaas@google.com Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 849DB1C1AC1; Thu, 10 Oct 2024 10:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=192.198.163.16 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728556343; cv=fail; b=HW5xLROcNx54nJHwGDPuxSyumGFTLl52eWebnijnE6uijkpqcO6sozad/SQ5lmDAo5N9boA/T3+BRI9heL2bYz5P5j7Omd2DHxW70Qm2GFQ6HxG0UHuX1Q2eRvJV0Ro8FMiVVrie35ke5rR/souDmc9EK6/7jMm4fLTtsFFcs+s= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728556343; c=relaxed/simple; bh=Bs8Jn4giSEzArPZb//LTvOlHynnehCNqHp7FGu65Veg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ZNDRBHNzoKEDXmGKGaIRuMZ6N0Ar0l3Tn9HvA/rs5pBiA9InGznGZcS5GUeYN1oe7TBIaBadJjTXWxE8R9hzXrINoZVv0U/wlQavNwGe/3IpA0hrcMKQ13xGbjUolfhK6OGRbBWiDRHA02kDNNOvxvqFmLuzZFjgao9tc3g41MQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=JeS3FbFU; arc=fail smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="JeS3FbFU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728556341; x=1760092341; h=from:to:cc:subject:date:message-id:in-reply-to: references:content-transfer-encoding:mime-version; bh=Bs8Jn4giSEzArPZb//LTvOlHynnehCNqHp7FGu65Veg=; b=JeS3FbFUE/xtGp/Y4wSTMj+Vh4geg0Tl8jebC+nmupttOkTedrJJzBSB SylaDlKliumcYyjiD4OUQEJ/vX8T1YpTZ+NPTPNdNftjp+d8Fj1NQnDFM T8r+V6LE4LUcq/cGm6Nv+JsUbjADFQxuwASeEB8ERJL+uzX1iGe7AaOvq TyX7TEXvtMcuSxSBObiqsAOgYcwMkpKKyihkFJ4Sj+rk3GmWevde+f2I0 +UUmw8qdk0RgpR20FHUThz0Ukg9OO8MsOdpgM8YoeihqgNSFiiYdcjSd4 a6DZFKYnB2Qe14MlGHyLgiLbJPUdbIM0NdyFJMycRdZiDzWAHCfnzHJZL A==; X-CSE-ConnectionGUID: pEf8QKSfTfmz72F6HLM95A== X-CSE-MsgGUID: se6TxyO8RIyznkrfkFo5Tg== X-IronPort-AV: E=McAfee;i="6700,10204,11220"; a="15528425" X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="15528425" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2024 03:32:20 -0700 X-CSE-ConnectionGUID: v55bOunjSt65Vs2U2noWuA== X-CSE-MsgGUID: Ispt2/E+T2e86earY0XKMg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,192,1725346800"; d="scan'208";a="76464924" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orviesa010.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 10 Oct 2024 03:32:20 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:19 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39 via Frontend Transport; Thu, 10 Oct 2024 03:32:19 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.43) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Thu, 10 Oct 2024 03:32:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kxObkvnTz+CwG+LZrXmCHZzrTAMMIxAjYnGyVF38xc4UMAGEUqrx8KZu9Kc9sLjViPDfz/0VMn0FzmR+AhMJ0saI9s9NdLuEFyMeTKGwHF0m5vdwtcgBHLR+xzQlcO8OFR6alRQLl4utSPrjc5JDAb5mp9ScDoa7CJPNO1B+TUBrCwJuU+0enY3L0ZYwvgRIusVc+XEy+1p7PsVeRG/Rq17Z2owoIfNYMDnKqLUdSBUDuxAJ4MjRDWyHMmxIRjg7F52Kz0H7OXIVypBuqIW7Y57RGloH5JslrAiLzNx2sdBefVhZYS4kLcLNETJna92f9YIAmbI+hveQS8OkIuU8kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QyYEH2qAYR9ofrIMUSnRZ7+9N+tYnnjyVObjHaQr+A8=; b=JAhxDeiBdaGr0aspXtEA0yK+1nDrQUzQ6p84gKd5iJvjmjCkp+l8yrlVrkDaiEI0phNuEt6rm6UCqGxHDDNH83nuHBJwb5NXt0vFpHl+7GNdGH1vlkFUvi+GDt6A+85fvK6vzfBDgCLB9Ixnsi9LsWEEeSbSTWHJluAkl4zEGgm7d5DlBx8/V8gXlKgqa8YO/5KGzDwCeEh/nqVGrd5x/0wMmWRG4km/UWQPOfgyKQpjZaRE6KAWyX2jtKgCbsZstgWlq6GpBucB2a41OTs/YEUweN1W6/t+daVrvw1fD7D+2B7ua44e+J2AHSvz5WXXEOP0bETqMnzRwXErMDYXVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) by MN6PR11MB8103.namprd11.prod.outlook.com (2603:10b6:208:473::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.22; Thu, 10 Oct 2024 10:32:15 +0000 Received: from DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39]) by DM4PR11MB5373.namprd11.prod.outlook.com ([fe80::927a:9c08:26f7:5b39%5]) with mapi id 15.20.8048.013; Thu, 10 Oct 2024 10:32:15 +0000 From: =?utf-8?q?Micha=C5=82_Winiarski?= To: , , , , "Bjorn Helgaas" , =?utf-8?q?Christian_K=C3=B6nig?= , =?utf-8?q?Krzy?= =?utf-8?q?sztof_Wilczy=C5=84ski?= , =?utf-8?q?Ilpo_J=C3=A4rvi?= =?utf-8?q?nen?= CC: Rodrigo Vivi , Michal Wajdeczko , Lucas De Marchi , =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Matt Roper , =?utf-8?q?Micha=C5=82_Winiarski?= Subject: [PATCH v3 1/5] PCI/IOV: Restore VF resizable BAR state after reset Date: Thu, 10 Oct 2024 12:31:59 +0200 Message-ID: <20241010103203.382898-2-michal.winiarski@intel.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241010103203.382898-1-michal.winiarski@intel.com> References: <20241010103203.382898-1-michal.winiarski@intel.com> X-ClientProxiedBy: WA2P291CA0029.POLP291.PROD.OUTLOOK.COM (2603:10a6:1d0:1f::29) To DM4PR11MB5373.namprd11.prod.outlook.com (2603:10b6:5:394::7) Precedence: bulk X-Mailing-List: linux-pci@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR11MB5373:EE_|MN6PR11MB8103:EE_ X-MS-Office365-Filtering-Correlation-Id: 727fef6c-a28e-44ca-f976-08dce916ced5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?61vg0Rv0vmypUKBOYcff610viUTs3o2?= =?utf-8?q?LzppXpGm2YRbEjloReXxPXmlX4jAPQgB5uHji7UVax8S56TD3/4VylV0I5MTP58DM?= =?utf-8?q?n6H+2LQl74ekruG8TbJxDHOIxqdw+AEieDYAWcCRY4AGyj+jz/4IQFooojEU1AgkF?= =?utf-8?q?yNu0Wo7pB0g+K1EdN5lKauYnbCHCrjbnvhYvB5/Kmf7IJFcDcD9Fixk5h5EHGL+KU?= =?utf-8?q?tago9Pi2EuyO7s2rSe11xqOiQcWQ+X0FsQm2If04lENEeh0tuiZer+rhcb6PhV3mE?= =?utf-8?q?G2R9oNGr+jtc3KgLBzGb2Ut6GfUYELY7n+S/fs3hJMCuXC6NZNyR5G2YdIP8WjjYG?= =?utf-8?q?s9uYWzM2ebUeSSPTSZmlzwCZc7LQtwt0guM9ONqkBI/a1LrTgc4DbxsZD+Ec0xY3k?= =?utf-8?q?EbtoFTFBFT0KNcZeHBcL4C0MbeSXy7y4R+1X5Kbl4dUcQsAfAvcZUZrjfZh/wRzmk?= =?utf-8?q?JXlBXkEE8rjdk3yIxPvuCX5freVPWp/wkS6V5TT8a78bVQar4a/1o1XL/g3vO6SxC?= =?utf-8?q?sKV1QkzjyEgNTNxpZI9LzV+WSvzUOeqcLCoCs1CyGrZT0fUbvpL9Rb7xwS/CcloMA?= =?utf-8?q?Hs2e5rvblRKpPwhZ+oledhTLn5UnmrTWmOJVw8V5ECD158T9s6K1VMN1Z0QLicwn9?= =?utf-8?q?2AOPPAvke7kQiWEhb32tEUCqM+v/HxTZeZ62gFNjoshgn7BLL5tUVo5GCMj6AqrHo?= =?utf-8?q?ZeYI2EWxZkzjDv5aEssp+PZPYFnAUbY+5BgMHA7+G2KhUKQgPOE3GvMWG8TVK4iCy?= =?utf-8?q?E1RM84Si8QoQMwyVKJ3qU8m+lUiqTRO4tzl5rtQD7jscG6g7+uqXTdMl8+vMSGNlC?= =?utf-8?q?5A6h5UCj0EqS0uP2CGyOKHeVH3EfoM4asXuGp+gFkIaC1+BvquVPuBHfStg7MhzPl?= =?utf-8?q?ai+hs8LninjUPf6d1PfuBvFmhvZ/JsRoWrhR6jOlc8TzTyBhzSlbbcjE33tVpuoKD?= =?utf-8?q?x0+zi/hYdybppYddu/G47sT5MPiqkVTWGVRbW59TugW0wHEodPWFnINHc+1Twuaje?= =?utf-8?q?mmUf2sqgrpoUXpaOzUtSRhUvs4Ea0JCbb0VQR4KUYXVI3i7Ukv76Mx/eNo3ue2v08?= =?utf-8?q?tCwjqXu7YmRI6/kfRHu49pwOdutEzP7yJ22LlCwZQNLJBz1n5v+WnkH6VC75UKeES?= =?utf-8?q?vIWIR8elNn/dIaaaf5LRlhneNOJ08ssCMa2wNIzMzniIj+7v/qLK/MMKt+paXOTJ4?= =?utf-8?q?+mboCrV/Js1saF+DdtvDCyPehIqKnWesqD3A=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR11MB5373.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(7416014)(1800799024)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?L3qRzr08pgIzWyJIu1FNUI210Jn1?= =?utf-8?q?wsEj9d7TTsV5FhPqUr3pUgWPZ5vzUNCuIWI9wOL4BHNH4YcldLlHvEP/8xLwy06rl?= =?utf-8?q?r8YzrB/yWjO2imFUKFDcpE50f7yRObPgiwFgipwg8jdD9JuwbjssVknPep7GH78wF?= =?utf-8?q?faZa7GRWaAzdZUbNiwe0SaOawzXPfwYIHv09PVTzLnihY2wL59GNNLieiXKlpZIsw?= =?utf-8?q?ESkBG4Dmmx4AdKvqcWIigfIskS62tKGkMl+2gFGSqMxQt3CSU5zwporEJObSmYMOr?= =?utf-8?q?SnDx5A6rLbYNYklt4PmWxh/IBoBKlgRRiXBuHE/NUcH2BkyDhvR1X5s7j4nEWCw2p?= =?utf-8?q?352qxKvwbWogDmGJXWrGJTpTZQoyyxlYAG7fG44J0cqf//XMn4RxO7tCWR3yXgUCG?= =?utf-8?q?cFnvtaJYooeH3OqPZLVNIGyksH1It4Pss8bwsAs4a0Q4DqktaTuhOVflG9x47EwXV?= =?utf-8?q?xlCQco0lvuklQEKEA6KatIF22l5OjhvEQkdSoUizAiTOGEMIFqkG4wuaBQpGVZVj3?= =?utf-8?q?T4Rn7WeWYBNbbw2BOnIP/oJx84mMQccsigLFr7anYUHDPnq3zAItVfHyk3I1Lgcf1?= =?utf-8?q?Dgmw4CXYC3FhAe9xfUC+Dwf0UNjwp0N2IA5vBt8Uf6g/HVBYK6Ci83KHg+gnEVLhN?= =?utf-8?q?ecSbcl1xfttynAsFnn1uWBuUdkG7CGvgqSa8RwOf0ZVV6W5nz044wjv+BHPSzgPlJ?= =?utf-8?q?LbyYgrhT4JrzCHL5ClA9ZGvCAfLXrGsvuJxoX0KhR//BRc2eZ1VhxU+eeoHHMWNd5?= =?utf-8?q?6HbD/CiPXCoe8IaBQcaOLb0ZfxuCdYGe/50jmg3xoDwdJV6MNa2b6XUO6mNNWu+1/?= =?utf-8?q?1lTg94TBpnXaDuhKNjrXYFl5EXSS5jyPAn360cSGPWjeWgrxa5gIX7SaswlFA62Dd?= =?utf-8?q?1c/v2IlI0N1Icz6gM+gV3rUf3Ow4oSPagQtFnzFX/eax1H9ov3xb/Cdfn477gaFux?= =?utf-8?q?RmSVmREiBkQprEQ+LMXRtzcw55gKx+s7V/1fFqertyZNgPjitNAKJu9rkjXk5X6+k?= =?utf-8?q?TVpOzW9BL7nrC+75okGIFl3MkL5mKYt4XMZW8ng/3btJRRwL3h4n3P8rVdxfWaY5L?= =?utf-8?q?UjdOQBmhyS2OvDzuaJaWP6UTi9NCu2TFvA3RsThcpurhI+/Hd5GyU89Vb0/6BGsdh?= =?utf-8?q?M6f4Mpsjtrfa1HkDhk5w1qBt+hUGgCcCjm9ldjbBx+5aJdrmLZPwt4+WoyTJ6cQL0?= =?utf-8?q?8ydq90w2GJbwnHM4Tm8M8USYOo3vfa8QHh2wDBj+lQSmbz7PQW3cSNpHvNlpqPpfT?= =?utf-8?q?DOh5BAaDvqAVvBd/skZeRf35wyoB7Mc1tZA5yZp5WXcSY7skUnx0IEohTiVeHCLcV?= =?utf-8?q?lFQkTcY69dkBegweGil9JxZG+aWA6I8dRjLGS/dp7m8lkfhEwSn5v0NTxnkRMHkv7?= =?utf-8?q?KDx6y2QHc+U8EhnUzt+DJuSnJmWArv7zjXnJyuVnqzbyTJb50r8ihVXfW18rOmU+P?= =?utf-8?q?9BoamG0AlqKBeN3r49gRxSXCdEjJmcjtNwVh3DxJkA1duDHBUKPHAuHjLLwMeCfo/?= =?utf-8?q?ZPiQftKmKgEbPjz9hwUZiFUaGRz7n1oB3w=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 727fef6c-a28e-44ca-f976-08dce916ced5 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2024 10:32:14.9701 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vVXEKXCCClLRcvJhHTc5FqsYOxG5yD6sk7fa5GfU3VLM02bNLxQvHh2XuaMQFntp92y03M4nHCY+XHdhb4+S0xwY40qw0E5u6J2IuRwlSn8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8103 X-OriginatorOrg: intel.com Similar to regular resizable BAR, VF BAR can also be resized, e.g. by the system firmware, or the PCI subsystem itself. Add the capability ID and restore it as a part of IOV state. See PCIe r4.0, sec 9.3.7.4. Signed-off-by: Michał Winiarski Reviewed-by: Ilpo Järvinen --- drivers/pci/iov.c | 29 ++++++++++++++++++++++++++++- include/uapi/linux/pci_regs.h | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index aaa33e8dc4c97..fd5c059b29c13 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -7,6 +7,7 @@ * Copyright (C) 2009 Intel Corporation, Yu Zhao */ +#include #include #include #include @@ -862,6 +863,30 @@ static void sriov_release(struct pci_dev *dev) dev->sriov = NULL; } +static void sriov_restore_vf_rebar_state(struct pci_dev *dev) +{ + unsigned int pos, nbars, i; + u32 ctrl; + + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_VF_REBAR); + if (!pos) + return; + + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); + nbars = FIELD_GET(PCI_REBAR_CTRL_NBAR_MASK, ctrl); + + for (i = 0; i < nbars; i++, pos += 8) { + int bar_idx, size; + + pci_read_config_dword(dev, pos + PCI_REBAR_CTRL, &ctrl); + bar_idx = ctrl & PCI_REBAR_CTRL_BAR_IDX; + size = pci_rebar_bytes_to_size(dev->sriov->barsz[bar_idx]); + ctrl &= ~PCI_REBAR_CTRL_BAR_SIZE; + ctrl |= FIELD_PREP(PCI_REBAR_CTRL_BAR_SIZE, size); + pci_write_config_dword(dev, pos + PCI_REBAR_CTRL, ctrl); + } +} + static void sriov_restore_state(struct pci_dev *dev) { int i; @@ -1021,8 +1046,10 @@ resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno) */ void pci_restore_iov_state(struct pci_dev *dev) { - if (dev->is_physfn) + if (dev->is_physfn) { + sriov_restore_vf_rebar_state(dev); sriov_restore_state(dev); + } } /** diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 12323b3334a9c..a0cf701c4c3af 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -740,6 +740,7 @@ #define PCI_EXT_CAP_ID_L1SS 0x1E /* L1 PM Substates */ #define PCI_EXT_CAP_ID_PTM 0x1F /* Precision Time Measurement */ #define PCI_EXT_CAP_ID_DVSEC 0x23 /* Designated Vendor-Specific */ +#define PCI_EXT_CAP_ID_VF_REBAR 0x24 /* VF Resizable BAR */ #define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26 /* Physical Layer 16.0 GT/s */ #define PCI_EXT_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */