From patchwork Mon Jun 10 23:38:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 10985667 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B25DF14B6 for ; Mon, 10 Jun 2019 23:38:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A272D28174 for ; Mon, 10 Jun 2019 23:38:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9653A28694; Mon, 10 Jun 2019 23:38:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15B9128174 for ; Mon, 10 Jun 2019 23:38:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390831AbfFJXiq (ORCPT ); Mon, 10 Jun 2019 19:38:46 -0400 Received: from mail-eopbgr20066.outbound.protection.outlook.com ([40.107.2.66]:22416 "EHLO EUR02-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2390524AbfFJXiq (ORCPT ); Mon, 10 Jun 2019 19:38:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z6BGDuE3gp1NMtEuDaRXieL4loJT0eX3ND/Pb4Ioi7M=; b=aNZEcy+5cpH8iPjt8K6fV/l/xdA80ufqnHOVH3gSuO/hJfFjd9GkInS4J6nvAZMu0Y65E+rL+tyC/94dQWINq5JIm4//BI00nb7KOpyW30H7H7SRKdOCdNmMtzGGxlb9JGSgUK6clM2+XLp5iJhtegJF+knfz4Yp0m+tCdiAWyc= Received: from DB6PR0501MB2759.eurprd05.prod.outlook.com (10.172.227.7) by DB6PR0501MB2166.eurprd05.prod.outlook.com (10.168.55.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1965.12; Mon, 10 Jun 2019 23:38:29 +0000 Received: from DB6PR0501MB2759.eurprd05.prod.outlook.com ([fe80::3b:cb20:88ed:30bf]) by DB6PR0501MB2759.eurprd05.prod.outlook.com ([fe80::3b:cb20:88ed:30bf%5]) with mapi id 15.20.1965.017; Mon, 10 Jun 2019 23:38:29 +0000 From: Saeed Mahameed To: Saeed Mahameed , Leon Romanovsky CC: "linux-rdma@vger.kernel.org" , "netdev@vger.kernel.org" , Yuval Avnery , Parav Pandit Subject: [PATCH mlx5-next 09/16] net/mlx5: Move IRQ rmap creation to IRQ allocation phase Thread-Topic: [PATCH mlx5-next 09/16] net/mlx5: Move IRQ rmap creation to IRQ allocation phase Thread-Index: AQHVH+WbOT2GJNW4EEOGWRXxQUbxcQ== Date: Mon, 10 Jun 2019 23:38:28 +0000 Message-ID: <20190610233733.12155-10-saeedm@mellanox.com> References: <20190610233733.12155-1-saeedm@mellanox.com> In-Reply-To: <20190610233733.12155-1-saeedm@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-originating-ip: [209.116.155.178] x-clientproxiedby: BYAPR01CA0015.prod.exchangelabs.com (2603:10b6:a02:80::28) To DB6PR0501MB2759.eurprd05.prod.outlook.com (2603:10a6:4:84::7) authentication-results: spf=none (sender IP is ) smtp.mailfrom=saeedm@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ad13eeff-1381-4dce-b163-08d6edfcbd75 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020);SRVR:DB6PR0501MB2166; x-ms-traffictypediagnostic: DB6PR0501MB2166: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:514; x-forefront-prvs: 0064B3273C x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(366004)(136003)(376002)(39860400002)(346002)(396003)(189003)(199004)(450100002)(85306007)(53936002)(6512007)(14454004)(50226002)(2616005)(186003)(256004)(81166006)(486006)(6436002)(11346002)(8676002)(8936002)(25786009)(476003)(14444005)(52116002)(446003)(2906002)(478600001)(99286004)(81156014)(4326008)(6486002)(107886003)(71200400001)(5660300002)(66446008)(64756008)(305945005)(66946007)(386003)(6506007)(7736002)(26005)(76176011)(71190400001)(102836004)(66476007)(86362001)(73956011)(66556008)(110136005)(6636002)(36756003)(54906003)(3846002)(6116002)(316002)(1076003)(66066001)(68736007);DIR:OUT;SFP:1101;SCL:1;SRVR:DB6PR0501MB2166;H:DB6PR0501MB2759.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: UOKIlLbGp3XIhCbPIzhdVPxgtmhnAemAqKY4ICbG59PAZ8+7pitDbwOuL0sUyJaV4H7fmOyiPAKSfwxEBA/k/KoBkNRVOnPgYbj27bn1r7Iq2FIOiT7cGsQBQfqxmSAESMlH8T+HL2Su0J5uRuz+78e2gBisBhieAmBsMbA3Co3/Sgr4ZQPO6KRXWxkAtVOJoeRE8qXB6eOuMlxJKzkR52kImzxA3CEokfv78Jh00zOEgtHAiIhbCLjf6jjKBhILs6YIYReFNA3LnS+Ykuyfg/H5zJnQsLsDUYVdqn3kjxB99H4iKDdtH0s4Qg7T+FkBduHrJJpo9jngNSjtJjB04i9S4AO3j1X+EFjJX9XhPih8/03vo23baKaimIUAuQmtnhjt3oqBr7ZDy6kcsxnVZavDdrfe2GVSzQMI7DHNjD0= MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad13eeff-1381-4dce-b163-08d6edfcbd75 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jun 2019 23:38:28.9445 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: saeedm@mellanox.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0501MB2166 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Yuval Avnery Rmap creation/deletion is part of the IRQ life-cycle. Signed-off-by: Yuval Avnery Reviewed-by: Parav Pandit Signed-off-by: Saeed Mahameed --- drivers/net/ethernet/mellanox/mlx5/core/eq.c | 82 ++++++++++++++------ 1 file changed, 57 insertions(+), 25 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c index cdfa35ec02fa..1ea983c1ec05 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c @@ -216,6 +216,49 @@ static int request_irqs(struct mlx5_core_dev *dev, int nvec) return err; } +static void irq_clear_rmap(struct mlx5_core_dev *dev) +{ +#ifdef CONFIG_RFS_ACCEL + struct mlx5_irq_table *irq_table = dev->priv.irq_table; + + free_irq_cpu_rmap(irq_table->rmap); +#endif +} + +static int irq_set_rmap(struct mlx5_core_dev *mdev) +{ + int err = 0; +#ifdef CONFIG_RFS_ACCEL + struct mlx5_irq_table *irq_table = mdev->priv.irq_table; + int num_affinity_vec; + int vecidx; + + num_affinity_vec = mlx5_irq_get_num_comp(irq_table); + irq_table->rmap = alloc_irq_cpu_rmap(num_affinity_vec); + if (!irq_table->rmap) { + err = -ENOMEM; + mlx5_core_err(mdev, "failed to allocate cpu_rmap. err %d", err); + goto err_out; + } + + vecidx = MLX5_EQ_VEC_COMP_BASE; + for (; vecidx < irq_table->nvec; vecidx++) { + err = irq_cpu_rmap_add(irq_table->rmap, + pci_irq_vector(mdev->pdev, vecidx)); + if (err) { + mlx5_core_err(mdev, "irq_cpu_rmap_add failed. err %d", err); + goto err_irq_cpu_rmap_add; + } + } + return 0; + +err_irq_cpu_rmap_add: + irq_clear_rmap(mdev); +err_out: +#endif + return err; +} + static int mlx5_cmd_destroy_eq(struct mlx5_core_dev *dev, u8 eqn) { u32 out[MLX5_ST_SZ_DW(destroy_eq_out)] = {0}; @@ -893,12 +936,6 @@ static void destroy_comp_eqs(struct mlx5_core_dev *dev) clear_comp_irqs_affinity_hints(dev); -#ifdef CONFIG_RFS_ACCEL - if (table->irq_table->rmap) { - free_irq_cpu_rmap(table->irq_table->rmap); - table->irq_table->rmap = NULL; - } -#endif list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) { list_del(&eq->list); if (destroy_unmap_eq(dev, &eq->core)) @@ -921,11 +958,6 @@ static int create_comp_eqs(struct mlx5_core_dev *dev) INIT_LIST_HEAD(&table->comp_eqs_list); ncomp_eqs = table->num_comp_eqs; nent = MLX5_COMP_EQ_SIZE; -#ifdef CONFIG_RFS_ACCEL - table->irq_table->rmap = alloc_irq_cpu_rmap(ncomp_eqs); - if (!table->irq_table->rmap) - return -ENOMEM; -#endif for (i = 0; i < ncomp_eqs; i++) { int vecidx = i + MLX5_EQ_VEC_COMP_BASE; struct mlx5_eq_param param = {}; @@ -942,10 +974,6 @@ static int create_comp_eqs(struct mlx5_core_dev *dev) tasklet_init(&eq->tasklet_ctx.task, mlx5_cq_tasklet_cb, (unsigned long)&eq->tasklet_ctx); -#ifdef CONFIG_RFS_ACCEL - irq_cpu_rmap_add(table->irq_table->rmap, - pci_irq_vector(dev->pdev, vecidx)); -#endif eq->irq_nb.notifier_call = mlx5_eq_comp_int; param = (struct mlx5_eq_param) { .index = vecidx, @@ -1039,14 +1067,7 @@ void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev) int i, max_eqs; clear_comp_irqs_affinity_hints(dev); - -#ifdef CONFIG_RFS_ACCEL - if (table->irq_table->rmap) { - free_irq_cpu_rmap(table->irq_table->rmap); - table->irq_table->rmap = NULL; - } -#endif - + irq_clear_rmap(dev); mutex_lock(&table->lock); /* sync with create/destroy_async_eq */ max_eqs = table->num_comp_eqs + MLX5_EQ_VEC_COMP_BASE; for (i = max_eqs - 1; i >= 0; i--) { @@ -1086,13 +1107,19 @@ static int alloc_irq_vectors(struct mlx5_core_dev *dev) table->nvec = nvec; + err = irq_set_rmap(dev); + if (err) + goto err_set_rmap; + err = request_irqs(dev, nvec); if (err) - goto err_free_irqs; + goto err_request_irqs; return 0; -err_free_irqs: +err_request_irqs: + irq_clear_rmap(dev); +err_set_rmap: pci_free_irq_vectors(dev->pdev); err_free_irq_info: kfree(table->irq_info); @@ -1104,6 +1131,11 @@ static void free_irq_vectors(struct mlx5_core_dev *dev) struct mlx5_irq_table *table = dev->priv.irq_table; int i; + /* free_irq requires that affinity and rmap will be cleared + * before calling it. This is why there is asymmetry with set_rmap + * which should be called after alloc_irq but before request_irq. + */ + irq_clear_rmap(dev); for (i = 0; i < table->nvec; i++) free_irq(pci_irq_vector(dev->pdev, i), &mlx5_irq_get(dev, i)->nh);