From patchwork Thu Sep 12 18:03:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Jin X-Patchwork-Id: 11143619 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE79416B1 for ; Thu, 12 Sep 2019 18:04:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CD2002084D for ; Thu, 12 Sep 2019 18:04:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="XHvtafrL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD2002084D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i8TR9-0007Oy-M0; Thu, 12 Sep 2019 18:03:19 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1i8TR8-0007Ot-D2 for xen-devel@lists.xenproject.org; Thu, 12 Sep 2019 18:03:18 +0000 X-Inumbo-ID: 98f7f626-d587-11e9-978d-bc764e2007e4 Received: from aserp2120.oracle.com (unknown [141.146.126.78]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 98f7f626-d587-11e9-978d-bc764e2007e4; Thu, 12 Sep 2019 18:03:17 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x8CHxEFx155240; Thu, 12 Sep 2019 18:03:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=to : from : subject : message-id : date : mime-version : content-type : content-transfer-encoding; s=corp-2019-08-05; bh=eF5ZvNeDNNFBCR3/VhAcPHWKX7FFqyhFN5sQiitqf9Y=; b=XHvtafrLUoNxMB9KQ/UTPz/XLKFyF+wwjz03klmhmYxhOULzMb+ulKvlb7jFkFxVxv7N FIufNx3wyMhWUGBqXIA73mpzGNRMkVQuonO24mw3mqZQJLACsXb4jbciLqwO3UJinOPw PwpLkLfCR3zYzs6Oy6de/mCd3Iuuw5cTqZA79oneCoI33mV4TjJA6Rm/5lmIZj0uD/Y2 xhu+ioTnK02Kx8Bwc2ZbG2U9yxdjFJW/hPyKw1bT/tA0mf3czvIGmPeS3bNqtNxwAjfq idIpBIwKj+58WcTahHDi622Ls3+5/rK0yQuGXQPhMmkGUW7Akn9WPouyNg/IfngsHWAJ zA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2120.oracle.com with ESMTP id 2uytd306d1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Sep 2019 18:03:16 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x8CHx30J045449; Thu, 12 Sep 2019 18:03:16 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3030.oracle.com with ESMTP id 2uytdnsbfm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 12 Sep 2019 18:03:16 +0000 Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x8CI3Fh0016854; Thu, 12 Sep 2019 18:03:15 GMT Received: from [10.211.46.60] (/10.211.46.60) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 12 Sep 2019 11:03:14 -0700 To: xen-devel@lists.xenproject.org, Jan Beulich , Boris Ostrovsky , Joao Martins , "DONGLI.ZHANG" From: Joe Jin Message-ID: <70457d4e-068f-0160-532b-e00dd295b3b1@oracle.com> Date: Thu, 12 Sep 2019 11:03:14 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 Content-Language: en-US X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9378 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=759 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1909120189 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9378 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=820 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1909120189 Subject: [Xen-devel] [PATCH RFC] pass-through: sync pir to irr after msix vector been updated X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" With below testcase, guest kernel reported "No irq handler for vector": 1). Passthrough mlx ib VF to 2 pvhvm guests. 2). Start rds-stress between 2 guests. 3). Scale down 2 guests vcpu from 32 to 6 at the same time. Repeat above test several iteration, guest kernel reported "No irq handler for vector", and IB traffic downed to zero which caused by interrupt lost. When vcpu offline, kernel disabled local IRQ, migrate IRQ to other cpu, update MSI-X table, enable IRQ. If any new interrupt arrived after local IRQ disabled also before MSI-X table been updated, interrupt still used old vector and dest cpu info, and when local IRQ enabled again, interrupt been sent to wrong cpu and vector. Looks sync PIR to IRR after MSI-X been updated is help for this issue. BTW, I could not reproduced this issue if I disabled apicv. Signed-off-by: Joe Jin --- xen/drivers/passthrough/io.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c index 4290c7c710..10c5b5d1e1 100644 --- a/xen/drivers/passthrough/io.c +++ b/xen/drivers/passthrough/io.c @@ -412,6 +412,9 @@ int pt_irq_create_bind( pirq_dpci->gmsi.gvec = pt_irq_bind->u.msi.gvec; pirq_dpci->gmsi.gflags = gflags; } + + if ( hvm_funcs.sync_pir_to_irr ) + hvm_funcs.sync_pir_to_irr(d->vcpu[pirq_dpci->gmsi.dest_vcpu_id]); } /* Calculate dest_vcpu_id for MSI-type pirq migration. */ dest = MASK_EXTR(pirq_dpci->gmsi.gflags,