From patchwork Fri Oct 14 00:19:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gregory Price X-Patchwork-Id: 13006574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DFCD2C433FE for ; Fri, 14 Oct 2022 00:20:55 +0000 (UTC) Received: from localhost ([::1]:38250 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oj8Ra-0007jX-Qf for qemu-devel@archiver.kernel.org; Thu, 13 Oct 2022 20:20:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39240) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oj8Qp-00072l-L7 for qemu-devel@nongnu.org; Thu, 13 Oct 2022 20:20:07 -0400 Received: from mail-dm6nam10on2064.outbound.protection.outlook.com ([40.107.93.64]:23303 helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oj8Qm-0002Dx-Hb for qemu-devel@nongnu.org; Thu, 13 Oct 2022 20:20:07 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RjMLqdIyesnfHrUcB0Rr9MuTj08E7Rkj+0hgnf+tTc/Oro9FDB1qX4v2GUq3fm2w/XW7O3qJKsxtt7NV+4FAteP3gKJEQ5ULcjHKiFt3buvjH87hSECBRTGm/tLl/QnLC2YmFdbM5Cd3yAfCPWjILKFSFSYJAlFGB6mmN+kl3BqNMhO/qjQ83XpPBpi+KuWQa6g2rALl1k5neW14kQeCvNix26EbWMrmBcriMKBu7M+9xx++kSNtq7HdO8YbEo97C+dGJPoG6sG9WcmKrsGYoTihq1NcpxemhZ5fVlYk2jyumh/tAc3DiAxBwfVxru92CX17WGnuZXiXTFpW6GG+Ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iiFeUCBLecNzFyn5JMprLQQCTHKWzg0uPUMg4B1xfCg=; b=cj7Q95rBJl1/g3BjKMIKOFES5IbpbgVO6focFYEpSWbkwTQGYAyOS5uywAtUXUnKxGJpLd1xkBJJ94iu3/qJ61vULngxbQ53MnM7kR1rGIyrftQvW5piXcl3oiY9BHOwgiEBBPT9/wzwXbZc0xMx4Aaez21C6+cQSUV9b8AFpYetJTkJF4JtXuAqY80N8OThBxhu/qZ4afQHHh34LX7lfeoCZ4IAlU0Cu0+4sPn0uKLrqvO57idfqic8GXeIm12vSfDs21cvVDjE7uL8LI4yYEcfl+m8waU+/qfu6o6qFZWGLsfIBlKbWw1TbaV3pPeLE9IMMArceNUN5ywiJmVA7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iiFeUCBLecNzFyn5JMprLQQCTHKWzg0uPUMg4B1xfCg=; b=rgGRgECODjuwKQdwGgpoGuqL3VWg+NTja8HgcrSdKrbpg0RSRfSsaieVkcc4TBuIMHh1t6qVIgTciFBDszfhfcK05q+vV2+OCdsZhDgkl89NxawJPV9CbbKtxkj9p/SB1+f6oXfdv0CXYhdUwg4xg6EGKxJyNRX20bYNm18/trE= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=memverge.com; Received: from BN6PR17MB3121.namprd17.prod.outlook.com (2603:10b6:405:7c::19) by DM4PR17MB5897.namprd17.prod.outlook.com (2603:10b6:8:4a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.32; Fri, 14 Oct 2022 00:19:58 +0000 Received: from BN6PR17MB3121.namprd17.prod.outlook.com ([fe80::655f:bfa9:e33a:5af4]) by BN6PR17MB3121.namprd17.prod.outlook.com ([fe80::655f:bfa9:e33a:5af4%5]) with mapi id 15.20.5723.026; Fri, 14 Oct 2022 00:19:56 +0000 Date: Thu, 13 Oct 2022 20:19:56 -0400 From: Gregory Price To: Jonathan Cameron Cc: Gregory Price , qemu-devel@nongnu.org, linux-cxl@vger.kernel.org, Alison Schofield , Davidlohr Bueso , "a.manzanares@samsung.com" , Ben Widawsky Subject: [BUG] cxl,i386: e820 mappings may not be correct for cxl Message-ID: Content-Disposition: inline X-ClientProxiedBy: BL0PR03CA0010.namprd03.prod.outlook.com (2603:10b6:208:2d::23) To BN6PR17MB3121.namprd17.prod.outlook.com (2603:10b6:405:7c::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR17MB3121:EE_|DM4PR17MB5897:EE_ X-MS-Office365-Filtering-Correlation-Id: a48af3cb-2a6a-457e-d4e2-08daad79d2f1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1IXileS//ZnFcQY5HwHjJUg3FXmnoSSDSCnbjAK+EovgYxA+hlQVS1A9M8bdNF5tHGCwnSZytBC6+c1MIjQkxhYW0DFYrPDVG6mHTURytlIFVpzVtEY+CcrrlLxvIJ4vz3IcL6FVMXfR+42gae4EixJYdDERtiUqWNGYG5gAY44uBIj66vL4t/3YHi8hkrVC9sQuJtuRzfcr3T9VPTyZbvwj7+3HIUax1GfuJg7LJd77e+LSCqUjI+/t9dgJwjLa1NVMwvuOU2J/0mecQvM0pmwsYcAh+/eW2pcHGjYq/2PG3EiryF9l8q+1l+jMfqNJX5DuEnRK/NUhkzfvODQeCvgVPU/ZyWzUdLwfTR572tmawac5s0RdBLz1P3l3MXx06JYhQZbqjGfrEUNMWrbiLQejxC9ujuSKBHp3bwJWv04j6f2nAaIzvuH9tyE9Lz60IXaKS4YZgtJ3rKEt7Sb3v8HhCxPlMyX9Cm+dxdxoO0gtxwZmHWYg4S68+yxbEzaZa+rv1CFHyJpbRl8mbkUmPUU4pE5YUsEx53MoKDI/L0wzEXSz1aKUHGYl+LbopA8/OdZJo3vQsHaFc5lIVLqOsmUPFZWi3XZTDo/9Hf6GlmGr27lGHk7LCNu34bv51Nt92jUt9ovyjYXJkwphoeACLh61lBSZb2C+8J2sJvttKXEnkX7X0oAzDPvnXfk+ddykxKmULRa1J0GVP9RdTQYIc5SEVfLDoLBiLbdtGCQrbuU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR17MB3121.namprd17.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(346002)(396003)(39840400004)(366004)(136003)(376002)(451199015)(2906002)(41300700001)(316002)(66946007)(8676002)(8936002)(478600001)(6512007)(66556008)(4326008)(44832011)(6486002)(966005)(5660300002)(6916009)(54906003)(26005)(66476007)(86362001)(36756003)(2616005)(83380400001)(186003)(6506007)(38100700002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0ZmoUZbn2QGb4oiI2CgeW3P9iovObK4l75Td6dbLWqrpTytZNMkHXQHlnk+e/fzBgf4BfLY9jLE9MSdhhGHPQUzFxQFfpf/kVaGHbht0zodTmq1FnkrMi0YR3+nFCtcoO4pvnO2JqEyV3GTEK2GzWfHrY/m/jzm+ZthgGaUOYHWKJMreOrK2pgPq7sDZ0fV5busvn5CDeTJ6p718eaegRVLn1S3fQvzeGM1rZUQl++bgEpfY2nxowRJMTrMZ0xOh/iZLoV5PnmjEDbMexIP+hL4t7N/CeID8BELxJtNvAlOQwXkddci/yFHiCoDwRDZe6ZpjbOKrAYde2HSGgfGFEeo5FJ0q0aeOXLVWvYqlj4tHtmbyxISf6u5C34pHNp8fyapWi6wmDhoIrCg/wPkOiYbGOzF5fqNBAQhfzEZ6U0wKlmFFpA5JOGCk7ehyvNsFMQZEnDEV8CKCHlMoTy23xVbTJkGSNU4EV1BZ5I4B+Gp0uQ2DcrfDnKCuD8dDffhXXM0fgAhcwjMIZt9U9Uwp92m4E4GluZqVSk16BRvTfO0Cqxiv11eZYKamENMK9SHfWtFbq4dPlTEqiA7VleCfXKI1xOl5gCtS7Uoba7sKtbLfg3LmUZGuSjA+kjiH5HklV92vVadW53oLStO1c/H9k3ZeJkR69p2mvqTyo9h7P9jlizCD3VApkbPwCta/Mn93XeBiO0ZG/z9MDppijlRmfas4GcZdcw18BWFp6yqY8iBCk9TTKG9OBFLj3P1PbxwuAH9L67uZcbfBd1yGzv1K6IwgRKm2C7rcmYsOChT9B6n83ih3F+QMmJhLxiV1q6rbSVrTjEJT2kzjnAWIObxSboYM4GgDweQobTnvJWbF1+HPqHX84SouNUXZFqd9SdGRKoOfgvOLpynV+ppOmrMB2UWoZKCrOWRTHzavf8UP1dmOBxah6mn17VPeAYJnkfQ2LRw/2A0usRR9bYtrvZs8LsAmu6V3Ap7Nmf/ZA35rsj6kX94G631dJ3UV6UOYUz9gPToFcYF++SbdoccM2TuhMygJ7tqzh13ISI075zgP3g9Xz/lLc934JCNxMDTikCEWH+GYlcXyUfEbTlUtwswWRAxg+GhBOpsmbDjtkBuEISAdN0+WskT4Y8t38jG5LlT6zPgCnFWWx8xANV0V5brgQhpoNMeloXukqIfIjiOVL2PPVRNuCtc3j2BzG6GgjryAG0wfRN5VBd7ggWbfc1peRuspaRl784wEdgRrBAoSfqQKZijTe1MA0DqJiVWwvs8BWaZUZWXKr9NlVteBeBo0eestqNTjZKY+SERrzkHHv3puR+j4IqMcdSB8Z41Zv3FyV+zi/57lz/ggeYJnxCrUUXAuE7V+7YEFgdMntZV5DGZVvutnZe0I9NZ5d0weNH9M5FOnZpBZCuvsn0sY6hjrnzSgdeE9Ta4h+PcimPlKJ/y94iIaV+SfT2xTqxTR1QPteg9TIn+KaEnMY4+3XlVDVHjXoUGcGfIIgQEiACL4RcUPGVKbbjbbjUqWHjlXeogDppIIzkTc0kOV4+9elVi8ahEDKoh+fgwB6oloiYMiiOoEgA0wJJtr3kPF+QeH7d82EDX6RAj7ar9KQPmHD/Z+SA== X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: a48af3cb-2a6a-457e-d4e2-08daad79d2f1 X-MS-Exchange-CrossTenant-AuthSource: BN6PR17MB3121.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2022 00:19:56.9152 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bnwKrwTobpG9xTEEX+4vDtaB/RFQT0xfis6jxOBPGPd8tXGEuuH9nwyzz7Ly4ywbvAK7Ild1e5K3/CHfkoDdaveyO6tZs794AvHTIPn1WyU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR17MB5897 Received-SPF: pass client-ip=40.107.93.64; envelope-from=gregory.price@memverge.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Context included below from prior discussion - `cxl create-region` would fail on inability to allocate memory - traced this down to the memory region being marked RESERVED - E820 map marks the CXL fixed memory window as RESERVED Re: x86 errors, I found that region worked with this patch. (I also added the SRAT patches the Davidlohr posted, but I do not think they are relevant). I don't think this is correct, and setting this to E820_RAM causes the system to fail to boot at all, but with this change `cxl create-region` succeeds, which suggests our e820 mappings in the i386 machine are incorrect. Anyone who can help or have an idea as to what e820 should actually be doing with this region, or if this is correct and something else is failing, please help! On Mon, Oct 10, 2022 at 05:32:42PM +0100, Jonathan Cameron wrote: > > > > but i'm not sure of what to do with this info. We have some proof > > > that real hardware works with this no problem, and the only difference > > > is that the EFI/bios/firmware is setting the memory regions as `usable` > > > or `soft reserved`, which would imply the EDK2 is the blocker here > > > regardless of the OS driver status. > > > > > > But I'd seen elsewhere you had gotten some of this working, and I'm > > > failing to get anything working at the moment. If you have any input i > > > would greatly appreciate the help. > > > > > > QEMU config: > > > > > > /opt/qemu-cxl2/bin/qemu-system-x86_64 \ > > > -drive file=/var/lib/libvirt/images/cxl.qcow2,format=qcow2,index=0,media=d\ > > > -m 2G,slots=4,maxmem=4G \ > > > -smp 4 \ > > > -machine type=q35,accel=kvm,cxl=on \ > > > -enable-kvm \ > > > -nographic \ > > > -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 \ > > > -device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 \ > > > -object memory-backend-file,id=cxl-mem0,mem-path=/tmp/cxl-mem0,size=256M \ > > > -object memory-backend-file,id=lsa0,mem-path=/tmp/cxl-lsa0,size=256M \ > > > -device cxl-type3,bus=rp0,pmem=true,memdev=cxl-mem0,lsa=lsa0,id=cxl-pmem0 \ > > > -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=256M > > > > > > I'd seen on the lists that you had seen issues with single-rp setups, > > > but no combination of configuration I've tried (including all the ones > > > in the docs and tests) lead to a successful region creation with > > > `cxl create-region` > > > > Hmm. Let me have a play. I've not run x86 tests for a while so > > perhaps something is missing there. > > > > I'm carrying a patch to override check_last_peer() in > > cxl_port_setup_targets() as that is wrong for some combinations, > > but that doesn't look like it's related to what you are seeing. > > I'm not sure if it's relevant, but turned out I'd forgotten I'm carrying 3 > patches that aren't upstream (and one is a horrible hack). > > Hack: https://lore.kernel.org/linux-cxl/20220819094655.000005ed@huawei.com/ > Shouldn't affect a simple case like this... > > https://lore.kernel.org/linux-cxl/20220819093133.00006c22@huawei.com/T/#t > (Dan's version) > > https://lore.kernel.org/linux-cxl/20220815154044.24733-1-Jonathan.Cameron@huawei.com/T/#t > > For writes to work you will currently need two rps (nothing on the second is fine) > as we still haven't resolved if the kernel should support an HDM decoder on > a host bridge with one port. I think it should (Spec allows it), others unconvinced. > > Note I haven't shifted over to x86 yet so may still be something different from > arm64. > > Jonathan > > diff --git a/hw/i386/pc.c b/hw/i386/pc.c index 566accf7e6..a5e688a742 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -1077,7 +1077,7 @@ void pc_memory_init(PCMachineState *pcms, memory_region_init_io(&fw->mr, OBJECT(machine), &cfmws_ops, fw, "cxl-fixed-memory-region", fw->size); memory_region_add_subregion(system_memory, fw->base, &fw->mr); - e820_add_entry(fw->base, fw->size, E820_RESERVED); + e820_add_entry(fw->base, fw->size, E820_NVS); cxl_fmw_base += fw->size; cxl_resv_end = cxl_fmw_base; }