From patchwork Tue Oct 1 15:38:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11169113 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A15371599 for ; Tue, 1 Oct 2019 15:38:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7FE8020842 for ; Tue, 1 Oct 2019 15:38:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="CUOnmnmR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727372AbfJAPii (ORCPT ); Tue, 1 Oct 2019 11:38:38 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:34818 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727246AbfJAPii (ORCPT ); Tue, 1 Oct 2019 11:38:38 -0400 Received: by mail-qt1-f196.google.com with SMTP id m15so22200597qtq.2 for ; Tue, 01 Oct 2019 08:38:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M8TytSI9W0EaNiTDJIBpTTQUheltq6fvtn/bWPVhJh0=; b=CUOnmnmRrfKeY+HsmVV4VVhxeYuNximkEJE5jSrM9tqwXpfiavH3PngsSBmgcjgyX5 p67h75WBmrfgWWO1sLmIvW93iONURYVqUgJhbPH7l/Ag4+JmomYBXz8kou5b7faYszpj TK8JYyT+BQG3kbfWa6KgZ5QuZ1ja2w6lhOry7eJEJ/HSwKS9NdLY3UMKEE9VLU0ZImKS D1ilq98f/zWf1BIKCcGBERvJwfxT0EzX81H9SGq54rccgv7axkht9/zCXedWtn4vG5KG yI+Zjl+wTmfn75vHNk/8cT+a8YghI+Mt3VCxZWcEyGMNnWWvPKnYjt7Q/IXS0oT/FZ3K Dvsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M8TytSI9W0EaNiTDJIBpTTQUheltq6fvtn/bWPVhJh0=; b=p6jMFo6HQ/FN2SJDLMseIo6OsXWK3QcLV7A2xASXrxq9uwxq7pryIOYQEVLORJc8cS o5NQuBx0p6pLkRoci2d3irlFibdKzV/s/gVbFBNdaQfc/6H//ERW/SBdYzAxqNKFHU9G 6W0FLGXTceJ2okw0umLAtoP613Zo8w2AS3OiWXi2qATJjfrSH7gFfKvoULo+6GYBabdN N+Usg3+dJG406Ca1q5B20KR3t2JsuovZvjPjlKEjk4YXXPL+ynNF1kt5EGjHXRaFRoB+ OwQR/B89pw/aNcFbhUNurW82bsuAvBGUEvpIRTUjAzWeYZrCmElK80eesmMWFydzaqiL MlPA== X-Gm-Message-State: APjAAAUJEj8Wy5h9Z8B3yRYyXgByQXw+ZOcivAenOAy1K0BSspnm56cs G4gLOC8OPweEh9RDy/yFqnoxgF4WKaE= X-Google-Smtp-Source: APXvYqzcYNSQxIxUsYDdzrcUalJ9ckmjQ8vaqWqtpx/04voydhrI2uHxOt6U/e//et8okaJjcDxb+w== X-Received: by 2002:ac8:2f81:: with SMTP id l1mr31693007qta.269.1569944317534; Tue, 01 Oct 2019 08:38:37 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id g45sm9073129qtc.9.2019.10.01.08.38.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Oct 2019 08:38:35 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iFKEU-0006EX-0e; Tue, 01 Oct 2019 12:38:34 -0300 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Artemy Kovalyov Subject: [PATCH 1/6] RDMA/mlx5: Do not allow rereg of a ODP MR Date: Tue, 1 Oct 2019 12:38:16 -0300 Message-Id: <20191001153821.23621-2-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191001153821.23621-1-jgg@ziepe.ca> References: <20191001153821.23621-1-jgg@ziepe.ca> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jason Gunthorpe This code is completely broken, the umem of a ODP MR simply cannot be discarded without a lot more locking, nor can an ODP mkey be blithely destroyed via destroy_mkey(). Fixes: 6aec21f6a832 ("IB/mlx5: Page faults handling infrastructure") Reviewed-by: Artemy Kovalyov Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/mr.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 1eff031ef04842..e7f840f306e46a 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1441,6 +1441,9 @@ int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start, if (!mr->umem) return -EINVAL; + if (is_odp_mr(mr)) + return -EOPNOTSUPP; + if (flags & IB_MR_REREG_TRANS) { addr = virt_addr; len = length; @@ -1486,8 +1489,6 @@ int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start, } mr->allocated_from_cache = 0; - if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) - mr->live = 1; } else { /* * Send a UMR WQE @@ -1516,7 +1517,6 @@ int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start, set_mr_fields(dev, mr, npages, len, access_flags); - update_odp_mr(mr); return 0; err: From patchwork Tue Oct 1 15:38:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11169107 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01AA41599 for ; Tue, 1 Oct 2019 15:38:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D42A42133F for ; Tue, 1 Oct 2019 15:38:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="FXHq4hOe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727312AbfJAPih (ORCPT ); Tue, 1 Oct 2019 11:38:37 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:33112 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727280AbfJAPih (ORCPT ); Tue, 1 Oct 2019 11:38:37 -0400 Received: by mail-qk1-f196.google.com with SMTP id x134so11692954qkb.0 for ; Tue, 01 Oct 2019 08:38:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s8IPBG3dgj2MA8wIPbTYtDxNXlAPNZNoDy7J6NM8bRQ=; b=FXHq4hOeiYiYpL8i2etxTsNt5Z+cjnttDjILRPclKw4ytfydwQO19NkaZqZmVFSx4t 60CK5IwCyOJ4zlHpT7W40Y37I7WJNlEaRLY0RDUyswXFLEq2cK8VCR+2kn+rC1BS1duF u9EKJffL+sO6QXnyeK/YsJyEH4DXsylVd6BkMlHpAYswvnTIEDRsYkcbU0Yr2bJYOt15 qqfJmyIJ13vBw+tn6PRUhiP9lEpuKTjWFGhpIqy0tBnn9S1r9IhJ+xuHTmGipDYYS2kl aCJZqkf1YyL2g/xC8finzEichRaX2HprDdC3KiPlPDAdyB7fkW2PpWp0EUDsTtgqAgcs kbtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s8IPBG3dgj2MA8wIPbTYtDxNXlAPNZNoDy7J6NM8bRQ=; b=imQQwvZu688mHNL1/OSttvfUyH9P3pXJKiEqJcudPs3GXaGmV5nX7QAUSFJaTy1ENP fc0QDAM6+DXhXg86mCbYwAZJriyBqTimZyKhupA/WuUAGvuGZoq6j3KYgx5bgNOwxbBv jarJwMjWCATXVAFZ7X7HZ9mtuX/u/ECdhodly9uB0n0vq3WWyghIa4Ut4vOyfl4MrKzV 8VcL8MsmcqNFdty+QSvZq5lenQB+pfOKeWngspMg1Dwm8r4X67ICZFkq8zPn7WZc6wqd XsQT7urpxy6VBgz2oq4q2E0S+OvnkhZ3SP7jQGa3hrT4S578v8zubCZEeUlk7mx0PQSI L/cw== X-Gm-Message-State: APjAAAWB/uKIEVojCuF36Q2oCu3/1iO8lHq4+Zk6DibNU6XbziL2LBMY XsqlhFa8t/mptHfC/PJ8/wh8KGpANuM= X-Google-Smtp-Source: APXvYqzIs39si8HJ7MjjT0NVAxrhbR6lILeTrTWwe9Y92gprg7fsZI1qwJhLLHuC+SE5FLyJMgE7ng== X-Received: by 2002:a37:4a85:: with SMTP id x127mr6545943qka.362.1569944315715; Tue, 01 Oct 2019 08:38:35 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id 29sm7681743qkp.86.2019.10.01.08.38.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Oct 2019 08:38:34 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iFKEU-0006Ed-1f; Tue, 01 Oct 2019 12:38:34 -0300 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Artemy Kovalyov Subject: [PATCH 2/6] RDMA/mlx5: Fix a race with mlx5_ib_update_xlt on an implicit MR Date: Tue, 1 Oct 2019 12:38:17 -0300 Message-Id: <20191001153821.23621-3-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191001153821.23621-1-jgg@ziepe.ca> References: <20191001153821.23621-1-jgg@ziepe.ca> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jason Gunthorpe mlx5_ib_update_xlt() must be protected against parallel free of the MR it is accessing, also it must be called single threaded while updating the HW. Otherwise we can have races of the form: CPU0 CPU1 mlx5_ib_update_xlt() mlx5_odp_populate_klm() odp_lookup() == NULL pklm = ZAP implicit_mr_get_data() implicit_mr_alloc() mlx5_ib_update_xlt mlx5_odp_populate_klm() odp_lookup() != NULL pklm = VALID mlx5_ib_post_send_wait() mlx5_ib_post_send_wait() // Replaces VALID with ZAP This can be solved by putting both the SRCU and the umem_mutex lock around every call to mlx5_ib_update_xlt(). This ensures that the content of the interval tree relavent to mlx5_odp_populate_klm() (ie mr->parent == mr) will not change while it is running, and thus the posted WRs to update the KLM will always reflect the correct information. The race above will resolve by either having CPU1 wait till CPU0 completes the ZAP or CPU0 will run after the add and instead store VALID. The pagefault path adding children already holds the umem_mutex and SRCU, so the only missed lock is during MR destruction. Fixes: 81713d3788d2 ("IB/mlx5: Add implicit MR support") Reviewed-by: Artemy Kovalyov Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/odp.c | 34 ++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 2e9b4306179745..3401c06b7e54f5 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -178,6 +178,29 @@ void mlx5_odp_populate_klm(struct mlx5_klm *pklm, size_t offset, return; } + /* + * The locking here is pretty subtle. Ideally the implicit children + * list would be protected by the umem_mutex, however that is not + * possible. Instead this uses a weaker update-then-lock pattern: + * + * srcu_read_lock() + * + * mutex_lock(umem_mutex) + * mlx5_ib_update_xlt() + * mutex_unlock(umem_mutex) + * destroy lkey + * + * ie any change the children list must be followed by the locked + * update_xlt before destroying. + * + * The umem_mutex provides the acquire/release semantic needed to make + * the children list visible to a racing thread. While SRCU is not + * technically required, using it gives consistent use of the SRCU + * locking around the children list. + */ + lockdep_assert_held(&to_ib_umem_odp(mr->umem)->umem_mutex); + lockdep_assert_held(&mr->dev->mr_srcu); + odp = odp_lookup(offset * MLX5_IMR_MTT_SIZE, nentries * MLX5_IMR_MTT_SIZE, mr); @@ -202,15 +225,22 @@ static void mr_leaf_free_action(struct work_struct *work) struct ib_umem_odp *odp = container_of(work, struct ib_umem_odp, work); int idx = ib_umem_start(odp) >> MLX5_IMR_MTT_SHIFT; struct mlx5_ib_mr *mr = odp->private, *imr = mr->parent; + struct ib_umem_odp *odp_imr = to_ib_umem_odp(imr->umem); + int srcu_key; mr->parent = NULL; synchronize_srcu(&mr->dev->mr_srcu); - ib_umem_odp_release(odp); - if (imr->live) + if (imr->live) { + srcu_key = srcu_read_lock(&mr->dev->mr_srcu); + mutex_lock(&odp_imr->umem_mutex); mlx5_ib_update_xlt(imr, idx, 1, 0, MLX5_IB_UPD_XLT_INDIRECT | MLX5_IB_UPD_XLT_ATOMIC); + mutex_unlock(&odp_imr->umem_mutex); + srcu_read_unlock(&mr->dev->mr_srcu, srcu_key); + } + ib_umem_odp_release(odp); mlx5_mr_cache_free(mr->dev, mr); if (atomic_dec_and_test(&imr->num_leaf_free)) From patchwork Tue Oct 1 15:38:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11169109 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 97DC117EE for ; Tue, 1 Oct 2019 15:38:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6CCA52133F for ; Tue, 1 Oct 2019 15:38:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="PyNMT3WB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727332AbfJAPih (ORCPT ); Tue, 1 Oct 2019 11:38:37 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:41394 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727246AbfJAPih (ORCPT ); Tue, 1 Oct 2019 11:38:37 -0400 Received: by mail-qt1-f193.google.com with SMTP id d16so1558018qtq.8 for ; Tue, 01 Oct 2019 08:38:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VjSimegOVv1mcHQrf53Oma0EmyHc2zoY77haYmlOcRw=; b=PyNMT3WBMuzQrCaWKCocLuFA42oL/lFhDZz2ZKJ+DN4FWOtA7GA4YABlo/yUeDORxe HgS1geoXhpVYtqXTsM3YkAdXBW9LIGghhnQgVC8qPheM0sn1thmbUO2mx5wbMFK/cXN4 gMO15CwTYKPgUimOai7JciKkKHqorXrxHxrLrxyYf+PHZAX4nHitJzgy22GjESNj0FDH Eo5WjoV4CE5CTNJAq9cV3yCq5llZis2YApsNO8YjIrjj94ww2PJctCCm4b4TS7FnUlk3 W9D55QauWJ9PvWLAKl7L8mC3kLg792jTJRTCT0jcGlVLRWnsJCCW1EIhK5yEVPXFGMf+ 6J3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VjSimegOVv1mcHQrf53Oma0EmyHc2zoY77haYmlOcRw=; b=nAqUSpDD6+xhjroERf3k4DuQ76jYmqIp4nhP+EOfj2vJjl4DTsqk9yZjlxqJWhcKXR /OZ8LxbAbJD829V87Qyav6QxyaBLCgVBOVggCNncFCC2mgBdTICTcNSDbwb/Ty0/J8R3 ZeTU82aU9bSFymaoZYXxmaXRwMsqjHhKwBEkLbd4fOy9i84YGO8nfv/MBAPf0r/CmUZ7 BNiZKlB7hBd6RsuwSMQ0pEXE7vmUrNW4hri3Ay5oJ+6p8qJZ5xfAZVpLpQ+yck6OAWuw qvDn8b0wYOEJ4/J0M8JQPHtOJclg3PHXvXw2Qnq1vhSaaJMRQDU/4ZCCfJN0vS3YRsKd NjnQ== X-Gm-Message-State: APjAAAVOPKu0g5JijmyR8ZYLbg2AtF6HN61XzoDulV1e1Y1YFZv++Xrd mJtUuvQg4SAFGXKoBTRw5O+Lz+W7uE0= X-Google-Smtp-Source: APXvYqyKs13ZaPqNEFDqX83BTo/jcQ5zBDKEno2PoY8HZjUtGtSnWob3Pc0YclC5jSZnICXS5FtA1w== X-Received: by 2002:a0c:eb89:: with SMTP id x9mr25228054qvo.18.1569944316517; Tue, 01 Oct 2019 08:38:36 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id 199sm7850089qkk.112.2019.10.01.08.38.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Oct 2019 08:38:34 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iFKEU-0006Ej-2j; Tue, 01 Oct 2019 12:38:34 -0300 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Artemy Kovalyov Subject: [PATCH 3/6] RDMA/odp: Lift umem_mutex out of ib_umem_odp_unmap_dma_pages() Date: Tue, 1 Oct 2019 12:38:18 -0300 Message-Id: <20191001153821.23621-4-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191001153821.23621-1-jgg@ziepe.ca> References: <20191001153821.23621-1-jgg@ziepe.ca> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jason Gunthorpe This fixes a race of the form: CPU0 CPU1 mlx5_ib_invalidate_range() mlx5_ib_invalidate_range() // This one actually makes npages == 0 ib_umem_odp_unmap_dma_pages() if (npages == 0 && !dying) // This one does nothing ib_umem_odp_unmap_dma_pages() if (npages == 0 && !dying) dying = 1; dying = 1; schedule_work(&umem_odp->work); // Double schedule of the same work schedule_work(&umem_odp->work); // BOOM npages and dying must be read and written under the umem_mutex lock. Since whenever ib_umem_odp_unmap_dma_pages() is called mlx5 must also call mlx5_ib_update_xlt, and both need to be done in the same locking region, hoist the lock out of unmap. This avoids an expensive double critical section in mlx5_ib_invalidate_range(). Fixes: 81713d3788d2 ("IB/mlx5: Add implicit MR support") Reviewed-by: Artemy Kovalyov Signed-off-by: Jason Gunthorpe --- drivers/infiniband/core/umem_odp.c | 6 ++++-- drivers/infiniband/hw/mlx5/odp.c | 12 ++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index f67a30fda1ed9a..163ff7ba92b7f1 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -451,8 +451,10 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) * that the hardware will not attempt to access the MR any more. */ if (!umem_odp->is_implicit_odp) { + mutex_lock(&umem_odp->umem_mutex); ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), ib_umem_end(umem_odp)); + mutex_unlock(&umem_odp->umem_mutex); kvfree(umem_odp->dma_list); kvfree(umem_odp->page_list); } @@ -719,6 +721,8 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, u64 addr; struct ib_device *dev = umem_odp->umem.ibdev; + lockdep_assert_held(&umem_odp->umem_mutex); + virt = max_t(u64, virt, ib_umem_start(umem_odp)); bound = min_t(u64, bound, ib_umem_end(umem_odp)); /* Note that during the run of this function, the @@ -726,7 +730,6 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, * faults from completion. We might be racing with other * invalidations, so we must make sure we free each page only * once. */ - mutex_lock(&umem_odp->umem_mutex); for (addr = virt; addr < bound; addr += BIT(umem_odp->page_shift)) { idx = (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; if (umem_odp->page_list[idx]) { @@ -757,7 +760,6 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, umem_odp->npages--; } } - mutex_unlock(&umem_odp->umem_mutex); } EXPORT_SYMBOL(ib_umem_odp_unmap_dma_pages); diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 3401c06b7e54f5..1930d78c3091cb 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -308,7 +308,6 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start, idx - blk_start_idx + 1, 0, MLX5_IB_UPD_XLT_ZAP | MLX5_IB_UPD_XLT_ATOMIC); - mutex_unlock(&umem_odp->umem_mutex); /* * We are now sure that the device will not access the * memory. We can safely unmap it, and mark it as dirty if @@ -319,10 +318,11 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start, if (unlikely(!umem_odp->npages && mr->parent && !umem_odp->dying)) { - WRITE_ONCE(umem_odp->dying, 1); + umem_odp->dying = 1; atomic_inc(&mr->parent->num_leaf_free); schedule_work(&umem_odp->work); } + mutex_unlock(&umem_odp->umem_mutex); } void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev) @@ -585,15 +585,19 @@ void mlx5_ib_free_implicit_mr(struct mlx5_ib_mr *imr) if (mr->parent != imr) continue; + mutex_lock(&umem_odp->umem_mutex); ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), ib_umem_end(umem_odp)); - if (umem_odp->dying) + if (umem_odp->dying) { + mutex_unlock(&umem_odp->umem_mutex); continue; + } - WRITE_ONCE(umem_odp->dying, 1); + umem_odp->dying = 1; atomic_inc(&imr->num_leaf_free); schedule_work(&umem_odp->work); + mutex_unlock(&umem_odp->umem_mutex); } up_read(&per_mm->umem_rwsem); From patchwork Tue Oct 1 15:38:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11169111 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19A78912 for ; Tue, 1 Oct 2019 15:38:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EB1C42133F for ; Tue, 1 Oct 2019 15:38:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="dEAHwIYg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727367AbfJAPii (ORCPT ); Tue, 1 Oct 2019 11:38:38 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:46719 "EHLO mail-qk1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727280AbfJAPii (ORCPT ); Tue, 1 Oct 2019 11:38:38 -0400 Received: by mail-qk1-f193.google.com with SMTP id 201so11626715qkd.13 for ; Tue, 01 Oct 2019 08:38:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iMTlO+AEgeDLsL8YAEtLh+P+HOtJZNPl0oq3/hQrOh8=; b=dEAHwIYgJ9V2/7IkEdYXM8at8DrKQqaErQMfF4CVDErlim9qX2xYBGr6qUyd6MMCzg kW3Zf+jlQaitXaaYc1+Kg7RGhq89tmtXhhuFWlH5efnlW8iVOeHroJwySekqgU7MZFDa 9Cg1wqEYgei1DxpohcB+bB1cjgZHsmha/8/3sK1VPGJcKah4wdo09A5UvhPI2cudGweg 2uwe5rFzmg4wDOA8infVKmX7y6JF7kXpYjS7hLzevfRxeF/vDbp0/EOjFSzwxrEJ/lBl qT/GtMXcWlQ3ehcApm55GVNFiX+IVq+ro+jlShpwGXQU9gfM1dDHpdMA4hZlk5129Cq3 Gfzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iMTlO+AEgeDLsL8YAEtLh+P+HOtJZNPl0oq3/hQrOh8=; b=UdHfNW8bB4odA1g3OLyXOI1lxkv4RJNGqH+WU/SgROHszvqUE3VRB0Rf3J3lXLFkid yNjvqrzpyz+0oaX3B3AWcRJhnThc56vC5pFPL3ASTuToOEkZHNrm16enMxbXexZlKaTH /7zTEOxJKkdslVebG9t2u5nSxDgpkfCfY3CtfFPkEgwZ5knMcyD1zwNF5Q4u3hAa9Mzi RMwhANJRUOxf6k4yNgaK8f0rmXqJaIvA9aycuukGFR80f0TvOH/AYcaT6aIdJYeSa2Xj DkmbqGka31JqSDBwZ80fzAtQ3dpZM/clmgHKXMiDYT1klAOBnC11Dardv2rl+Rrl8IPI tRdw== X-Gm-Message-State: APjAAAWmKi45/e5TiVEIvoJGHHMyyWJAJS7tyNV7DSzuVTAQlywQZgNX 7LXpz+/op27qlRFNV2uQKpwN0/PG6XI= X-Google-Smtp-Source: APXvYqyOS2+JlI4Pk18PHLLI0E0a7U1k5yd5/f7rUJ4sB9CDFRN6MeCvtesUM7X4VdZUGCljCAOXDA== X-Received: by 2002:a37:c204:: with SMTP id i4mr6541380qkm.282.1569944317138; Tue, 01 Oct 2019 08:38:37 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id u132sm7901728qka.50.2019.10.01.08.38.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Oct 2019 08:38:35 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iFKEU-0006Ep-3x; Tue, 01 Oct 2019 12:38:34 -0300 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Artemy Kovalyov Subject: [PATCH 4/6] RDMA/mlx5: Order num_pending_prefetch properly with synchronize_srcu Date: Tue, 1 Oct 2019 12:38:19 -0300 Message-Id: <20191001153821.23621-5-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191001153821.23621-1-jgg@ziepe.ca> References: <20191001153821.23621-1-jgg@ziepe.ca> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jason Gunthorpe During destroy setting live = 0 and then synchronize_srcu() prevents num_pending_prefetch from incrementing, and also, ensures that all work holding that count is queued on the WQ. Testing before causes races of the form: CPU0 CPU1 dereg_mr() mlx5_ib_advise_mr_prefetch() srcu_read_lock() num_pending_prefetch_inc() if (!live) live = 0 atomic_read() == 0 // skip flush_workqueue() atomic_inc() queue_work(); srcu_read_unlock() WARN_ON(atomic_read()) // Fails Swap the order so that the synchronize_srcu() prevents this. Fixes: a6bc3875f176 ("IB/mlx5: Protect against prefetch of invalid MR") Reviewed-by: Artemy Kovalyov Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/mr.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index e7f840f306e46a..0ee8fa01177fc9 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1609,13 +1609,14 @@ static void dereg_mr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr) */ mr->live = 0; + /* Wait for all running page-fault handlers to finish. */ + synchronize_srcu(&dev->mr_srcu); + /* dequeue pending prefetch requests for the mr */ if (atomic_read(&mr->num_pending_prefetch)) flush_workqueue(system_unbound_wq); WARN_ON(atomic_read(&mr->num_pending_prefetch)); - /* Wait for all running page-fault handlers to finish. */ - synchronize_srcu(&dev->mr_srcu); /* Destroy all page mappings */ if (!umem_odp->is_implicit_odp) mlx5_ib_invalidate_range(umem_odp, From patchwork Tue Oct 1 15:38:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11169115 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F8BD1599 for ; Tue, 1 Oct 2019 15:38:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D8B2D2133F for ; Tue, 1 Oct 2019 15:38:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="FGP4OZaR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732670AbfJAPik (ORCPT ); Tue, 1 Oct 2019 11:38:40 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:46806 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727280AbfJAPij (ORCPT ); Tue, 1 Oct 2019 11:38:39 -0400 Received: by mail-qt1-f194.google.com with SMTP id u22so22117555qtq.13 for ; Tue, 01 Oct 2019 08:38:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YRwvhHAb6cW4dr0SmjhZbNWKzHjgG12t0Qxkx60jldw=; b=FGP4OZaRY3iuuNJ4438Ei+B7Mq/xt38bLk1Y7MVhNEHPL2oR63sM5EQqd6itoQI0jU SjtFLXLJqAuiq2kGZtWeMhKorf6pVk4xXgif1TjaeRUMCNE/od7VZ3v2wd+v7Fg2GxSr 4QzrYvQsVMDDYKg/zMctBV/tlpu2erB1cle3wiL/Z/9HRD+jZOs4PPLTJ8p5j1Jn8Nb2 KWyW4VSBVf4MEO8dEkLCKS7n+oqN/k3UErKjg7iFUsQKD75/d6g8iqq5lHUhJ0rJxL1b R+Rsu840ZrjTWkomndpBmIOUzlNJSTCB6rOCWMXHuxXmiRdxkn3t9xorrQnfQXDIpt5j YezA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YRwvhHAb6cW4dr0SmjhZbNWKzHjgG12t0Qxkx60jldw=; b=c4UB+n1jXmksVdiwKgEBJ3qZPCxwz4sYth31g+hiexY7s3xrCiC5ZdaVnyC0Bw3lGh yHn3s3XVO1Nf1uc0YlYQEbnrXH8NNnBR567ym4z4kWPZQth8n3lTAFjh7+lNqRhy1pXY 7OhTQMG7+Rzx+7n0oH8d6Tmpz7M/kq6949sxT7ITHBqI0JVsuhfnapusLIslbH6px5L/ Jebs5KgVAgmTNE8v+hKnqz7Ivxwhxb+wuT2Iig4x/Na1+kdUIRrZPfJglnvWd52Bsaay //gtK/p++VsyvjWDCwyGuCHqT2zcNQUcwh7uJ/tqKkOHI4Dx3vGKEDUWKaSPpY7wdPmC QqIg== X-Gm-Message-State: APjAAAWUEXqWWidls19/TcFVbVX28+ua1yoOADen9tKlMVaR7E6qfrxT nEgTP6Ra3yqamKftokPU7fqXrFoUt2o= X-Google-Smtp-Source: APXvYqzyFGhbLzik3l5S7ApX+U2D8E5f6lIFQSEvTWpFUh7tXREQMAl71o4kbtddrm26Tir10PKcSw== X-Received: by 2002:ac8:4148:: with SMTP id e8mr31175582qtm.227.1569944318144; Tue, 01 Oct 2019 08:38:38 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id 20sm7942480qkn.4.2019.10.01.08.38.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Oct 2019 08:38:35 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iFKEU-0006Ev-5b; Tue, 01 Oct 2019 12:38:34 -0300 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Artemy Kovalyov Subject: [PATCH 5/6] RDMA/mlx5: Put live in the correct place for ODP MRs Date: Tue, 1 Oct 2019 12:38:20 -0300 Message-Id: <20191001153821.23621-6-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191001153821.23621-1-jgg@ziepe.ca> References: <20191001153821.23621-1-jgg@ziepe.ca> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jason Gunthorpe live is used to signal to the pagefault thread that the MR is initialized and ready for use. It should be after the umem is assigned and all other setup is completed. This prevents races (at least) of the form: CPU0 CPU1 mlx5_ib_alloc_implicit_mr() implicit_mr_alloc() live = 1 imr->umem = umem num_pending_prefetch_inc() if (live) atomic_inc(num_pending_prefetch) atomic_set(num_pending_prefetch,0) // Overwrites other thread's store Further, live is being used with SRCU as the 'update' in an acquire/release fashion, so it can not be read and written raw. Move all live = 1's to after MR initialization is completed and use smp_store_release/smp_load_acquire() for manipulating it. Add a missing live = 0 when an implicit MR child is deleted, before queuing work to do synchronize_srcu(). The barriers in update_odp_mr() were some broken attempt to create a acquire/release, but were not even applied consistently and missed the point, delete it as well. Fixes: 6aec21f6a832 ("IB/mlx5: Page faults handling infrastructure") Reviewed-by: Artemy Kovalyov Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/mr.c | 36 ++++------------------------ drivers/infiniband/hw/mlx5/odp.c | 14 ++++++----- 3 files changed, 14 insertions(+), 38 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 2ceaef3ea3fb92..15e42825cc976e 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -606,7 +606,7 @@ struct mlx5_ib_mr { struct mlx5_ib_dev *dev; u32 out[MLX5_ST_SZ_DW(create_mkey_out)]; struct mlx5_core_sig_ctx *sig; - int live; + unsigned int live; void *descs_alloc; int access_flags; /* Needed for rereg MR */ diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 0ee8fa01177fc9..3a27bddfcf31f5 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -84,32 +84,6 @@ static bool use_umr_mtt_update(struct mlx5_ib_mr *mr, u64 start, u64 length) length + (start & (MLX5_ADAPTER_PAGE_SIZE - 1)); } -static void update_odp_mr(struct mlx5_ib_mr *mr) -{ - if (is_odp_mr(mr)) { - /* - * This barrier prevents the compiler from moving the - * setting of umem->odp_data->private to point to our - * MR, before reg_umr finished, to ensure that the MR - * initialization have finished before starting to - * handle invalidations. - */ - smp_wmb(); - to_ib_umem_odp(mr->umem)->private = mr; - /* - * Make sure we will see the new - * umem->odp_data->private value in the invalidation - * routines, before we can get page faults on the - * MR. Page faults can happen once we put the MR in - * the tree, below this line. Without the barrier, - * there can be a fault handling and an invalidation - * before umem->odp_data->private == mr is visible to - * the invalidation handler. - */ - smp_wmb(); - } -} - static void reg_mr_callback(int status, struct mlx5_async_work *context) { struct mlx5_ib_mr *mr = @@ -1346,8 +1320,6 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, mr->umem = umem; set_mr_fields(dev, mr, npages, length, access_flags); - update_odp_mr(mr); - if (use_umr) { int update_xlt_flags = MLX5_IB_UPD_XLT_ENABLE; @@ -1363,10 +1335,12 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, } } - if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) { - mr->live = 1; + if (is_odp_mr(mr)) { + to_ib_umem_odp(mr->umem)->private = mr; atomic_set(&mr->num_pending_prefetch, 0); } + if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) + smp_store_release(&mr->live, 1); return &mr->ibmr; error: @@ -1607,7 +1581,7 @@ static void dereg_mr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr) /* Prevent new page faults and * prefetch requests from succeeding */ - mr->live = 0; + WRITE_ONCE(mr->live, 0); /* Wait for all running page-fault handlers to finish. */ synchronize_srcu(&dev->mr_srcu); diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 1930d78c3091cb..3f9478d1937668 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -231,7 +231,7 @@ static void mr_leaf_free_action(struct work_struct *work) mr->parent = NULL; synchronize_srcu(&mr->dev->mr_srcu); - if (imr->live) { + if (smp_load_acquire(&imr->live)) { srcu_key = srcu_read_lock(&mr->dev->mr_srcu); mutex_lock(&odp_imr->umem_mutex); mlx5_ib_update_xlt(imr, idx, 1, 0, @@ -318,6 +318,7 @@ void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start, if (unlikely(!umem_odp->npages && mr->parent && !umem_odp->dying)) { + WRITE_ONCE(mr->live, 0); umem_odp->dying = 1; atomic_inc(&mr->parent->num_leaf_free); schedule_work(&umem_odp->work); @@ -459,8 +460,6 @@ static struct mlx5_ib_mr *implicit_mr_alloc(struct ib_pd *pd, mr->ibmr.lkey = mr->mmkey.key; mr->ibmr.rkey = mr->mmkey.key; - mr->live = 1; - mlx5_ib_dbg(dev, "key %x dev %p mr %p\n", mr->mmkey.key, dev->mdev, mr); @@ -514,6 +513,8 @@ static struct ib_umem_odp *implicit_mr_get_data(struct mlx5_ib_mr *mr, mtt->parent = mr; INIT_WORK(&odp->work, mr_leaf_free_action); + smp_store_release(&mtt->live, 1); + if (!nentries) start_idx = addr >> MLX5_IMR_MTT_SHIFT; nentries++; @@ -566,6 +567,7 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd, init_waitqueue_head(&imr->q_leaf_free); atomic_set(&imr->num_leaf_free, 0); atomic_set(&imr->num_pending_prefetch, 0); + smp_store_release(&imr->live, 1); return imr; } @@ -807,7 +809,7 @@ static int pagefault_single_data_segment(struct mlx5_ib_dev *dev, switch (mmkey->type) { case MLX5_MKEY_MR: mr = container_of(mmkey, struct mlx5_ib_mr, mmkey); - if (!mr->live || !mr->ibmr.pd) { + if (!smp_load_acquire(&mr->live) || !mr->ibmr.pd) { mlx5_ib_dbg(dev, "got dead MR\n"); ret = -EFAULT; goto srcu_unlock; @@ -1675,12 +1677,12 @@ static bool num_pending_prefetch_inc(struct ib_pd *pd, mr = container_of(mmkey, struct mlx5_ib_mr, mmkey); - if (mr->ibmr.pd != pd) { + if (!smp_load_acquire(&mr->live)) { ret = false; break; } - if (!mr->live) { + if (mr->ibmr.pd != pd) { ret = false; break; } From patchwork Tue Oct 1 15:38:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11169117 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 44778912 for ; Tue, 1 Oct 2019 15:38:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 231D12133F for ; Tue, 1 Oct 2019 15:38:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="Ckgnye/s" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727280AbfJAPik (ORCPT ); Tue, 1 Oct 2019 11:38:40 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:38597 "EHLO mail-qk1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727246AbfJAPik (ORCPT ); Tue, 1 Oct 2019 11:38:40 -0400 Received: by mail-qk1-f193.google.com with SMTP id u186so11656802qkc.5 for ; Tue, 01 Oct 2019 08:38:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KKdHuEV0TcgXHbIA+nzfMHJx5W8bR5+RCJ7HA5Fsyp8=; b=Ckgnye/spviyMCGY5BzcjAjVOC8Edeb/NOLsMXMa8uYmoahbA/2ZAKbsaKvWd0w+Zq vGWgCo/UcLvQImMFjF0oNsa1xY0v4eYqatRHYo0A29L5e9SdjqbolasFgKBmy4TpSkNQ kiGyYrb7tf9jW+p+SvD0N+s/DKJEGU1psiRvz3y9YDfq96Njt5JUcELRDPWIuPYzG178 8cBtAf93ubYLIonJZtPNO61lGBU0hykorIb8aWXRy3Oloiv9GynezQsND785rPZIpYvF jH4t6uEDWS+zVc8tSrubLGZgKZK2YJUIo+Lg8zmYOGZqLL4jnU71BN56rqOgADYFMuXX r4ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KKdHuEV0TcgXHbIA+nzfMHJx5W8bR5+RCJ7HA5Fsyp8=; b=GZK98UpUClACRezA8XVA/hGvKDC9bMSeZv/6j6EpmK9de7ySU3srUmKAe6Xf3lzkvC +fgKjzCpQmQgOz0vqeyUeYzDvHpRmoXumpyI2IBga47nMClclccZ8HdzRCPSo6jKx/Oq MWsLJRHEEnKXV4OBERSiFXaM6HzLSFg+Drzlr+CdMESMgrhdQvqYh8WW5d5LYZojkOZN AF7VgEJDjJurn8jK71hTxM5V6O1At2tSdKpQ+/GG9EX5AJZZ/Y/PWiW5+gCKSpm9aUyy IY/8qsaU5YqYN/HS2mRbO5td/yVoWBGUJtxjnbcynDKS+/xmIft4jP2cEbwc4wusDJRc LNwg== X-Gm-Message-State: APjAAAXf0iyklAChaskQBb6byRwTiFg04YgnSrnF+jpwKNBmGYH6dIo5 UekJVMXxz93NrHDY3q6OPoEO0BSPePg= X-Google-Smtp-Source: APXvYqwkWxJDVIU8/2SL3hSYNNB8vSwQDnZJqGeB47kGkPp/Iym973yd3aj9vHaOrcfdGgCVDvCg8g== X-Received: by 2002:a37:6156:: with SMTP id v83mr6785365qkb.80.1569944318920; Tue, 01 Oct 2019 08:38:38 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id v4sm7487811qkj.28.2019.10.01.08.38.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Oct 2019 08:38:36 -0700 (PDT) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iFKEU-0006F1-6d; Tue, 01 Oct 2019 12:38:34 -0300 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Jason Gunthorpe , Artemy Kovalyov Subject: [PATCH 6/6] RDMA/mlx5: Add missing synchronize_srcu() for MW cases Date: Tue, 1 Oct 2019 12:38:21 -0300 Message-Id: <20191001153821.23621-7-jgg@ziepe.ca> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191001153821.23621-1-jgg@ziepe.ca> References: <20191001153821.23621-1-jgg@ziepe.ca> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Jason Gunthorpe While MR uses live as the SRCU 'update', the MW case uses the xarray directly, xa_erase() causes the MW to become inaccessible to the pagefault thread. Thus whenever a MW is removed from the xarray we must synchronize_srcu() before freeing it. This must be done before freeing the mkey as re-use of the mkey while the pagefault thread is using the stale mkey is undesirable. Add the missing synchronizes to MW and DEVX indirect mkey and delete the bogus protection against double destroy in mlx5_core_destroy_mkey() Fixes: 534fd7aac56a ("IB/mlx5: Manage indirection mkey upon DEVX flow for ODP") Fixes: 6aec21f6a832 ("IB/mlx5: Page faults handling infrastructure") Reviewed-by: Artemy Kovalyov Signed-off-by: Jason Gunthorpe --- drivers/infiniband/hw/mlx5/devx.c | 58 ++++++-------------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 - drivers/infiniband/hw/mlx5/mr.c | 21 +++++-- drivers/net/ethernet/mellanox/mlx5/core/mr.c | 8 +-- 4 files changed, 33 insertions(+), 55 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c index 59022b7441448f..d609f4659afb7a 100644 --- a/drivers/infiniband/hw/mlx5/devx.c +++ b/drivers/infiniband/hw/mlx5/devx.c @@ -1298,29 +1298,6 @@ static int devx_handle_mkey_create(struct mlx5_ib_dev *dev, return 0; } -static void devx_free_indirect_mkey(struct rcu_head *rcu) -{ - kfree(container_of(rcu, struct devx_obj, devx_mr.rcu)); -} - -/* This function to delete from the radix tree needs to be called before - * destroying the underlying mkey. Otherwise a race might occur in case that - * other thread will get the same mkey before this one will be deleted, - * in that case it will fail via inserting to the tree its own data. - * - * Note: - * An error in the destroy is not expected unless there is some other indirect - * mkey which points to this one. In a kernel cleanup flow it will be just - * destroyed in the iterative destruction call. In a user flow, in case - * the application didn't close in the expected order it's its own problem, - * the mkey won't be part of the tree, in both cases the kernel is safe. - */ -static void devx_cleanup_mkey(struct devx_obj *obj) -{ - xa_erase(&obj->ib_dev->mdev->priv.mkey_table, - mlx5_base_mkey(obj->devx_mr.mmkey.key)); -} - static void devx_cleanup_subscription(struct mlx5_ib_dev *dev, struct devx_event_subscription *sub) { @@ -1362,8 +1339,16 @@ static int devx_obj_cleanup(struct ib_uobject *uobject, int ret; dev = mlx5_udata_to_mdev(&attrs->driver_udata); - if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) - devx_cleanup_mkey(obj); + if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) { + /* + * The pagefault_single_data_segment() does commands against + * the mmkey, we must wait for that to stop before freeing the + * mkey, as another allocation could get the same mkey #. + */ + xa_erase(&obj->ib_dev->mdev->priv.mkey_table, + mlx5_base_mkey(obj->devx_mr.mmkey.key)); + synchronize_srcu(&dev->mr_srcu); + } if (obj->flags & DEVX_OBJ_FLAGS_DCT) ret = mlx5_core_destroy_dct(obj->ib_dev->mdev, &obj->core_dct); @@ -1382,12 +1367,6 @@ static int devx_obj_cleanup(struct ib_uobject *uobject, devx_cleanup_subscription(dev, sub_entry); mutex_unlock(&devx_event_table->event_xa_lock); - if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) { - call_srcu(&dev->mr_srcu, &obj->devx_mr.rcu, - devx_free_indirect_mkey); - return ret; - } - kfree(obj); return ret; } @@ -1491,26 +1470,21 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_OBJ_CREATE)( &obj_id); WARN_ON(obj->dinlen > MLX5_MAX_DESTROY_INBOX_SIZE_DW * sizeof(u32)); - if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) { - err = devx_handle_mkey_indirect(obj, dev, cmd_in, cmd_out); - if (err) - goto obj_destroy; - } - err = uverbs_copy_to(attrs, MLX5_IB_ATTR_DEVX_OBJ_CREATE_CMD_OUT, cmd_out, cmd_out_len); if (err) - goto err_copy; + goto obj_destroy; if (opcode == MLX5_CMD_OP_CREATE_GENERAL_OBJECT) obj_type = MLX5_GET(general_obj_in_cmd_hdr, cmd_in, obj_type); - obj->obj_id = get_enc_obj_id(opcode | obj_type << 16, obj_id); + if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) { + err = devx_handle_mkey_indirect(obj, dev, cmd_in, cmd_out); + if (err) + goto obj_destroy; + } return 0; -err_copy: - if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) - devx_cleanup_mkey(obj); obj_destroy: if (obj->flags & DEVX_OBJ_FLAGS_DCT) mlx5_core_destroy_dct(obj->ib_dev->mdev, &obj->core_dct); diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 15e42825cc976e..1a98ee2e01c4b9 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -639,7 +639,6 @@ struct mlx5_ib_mw { struct mlx5_ib_devx_mr { struct mlx5_core_mkey mmkey; int ndescs; - struct rcu_head rcu; }; struct mlx5_ib_umr_context { diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 3a27bddfcf31f5..630599311586ec 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1962,14 +1962,25 @@ struct ib_mw *mlx5_ib_alloc_mw(struct ib_pd *pd, enum ib_mw_type type, int mlx5_ib_dealloc_mw(struct ib_mw *mw) { + struct mlx5_ib_dev *dev = to_mdev(mw->device); struct mlx5_ib_mw *mmw = to_mmw(mw); int err; - err = mlx5_core_destroy_mkey((to_mdev(mw->device))->mdev, - &mmw->mmkey); - if (!err) - kfree(mmw); - return err; + if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) { + xa_erase(&dev->mdev->priv.mkey_table, + mlx5_base_mkey(mmw->mmkey.key)); + /* + * pagefault_single_data_segment() may be accessing mmw under + * SRCU if the user bound an ODP MR to this MW. + */ + synchronize_srcu(&dev->mr_srcu); + } + + err = mlx5_core_destroy_mkey(dev->mdev, &mmw->mmkey); + if (err) + return err; + kfree(mmw); + return 0; } int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mr.c b/drivers/net/ethernet/mellanox/mlx5/core/mr.c index 9231b39d18b21c..c501bf2a025210 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mr.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/mr.c @@ -112,17 +112,11 @@ int mlx5_core_destroy_mkey(struct mlx5_core_dev *dev, u32 out[MLX5_ST_SZ_DW(destroy_mkey_out)] = {0}; u32 in[MLX5_ST_SZ_DW(destroy_mkey_in)] = {0}; struct xarray *mkeys = &dev->priv.mkey_table; - struct mlx5_core_mkey *deleted_mkey; unsigned long flags; xa_lock_irqsave(mkeys, flags); - deleted_mkey = __xa_erase(mkeys, mlx5_base_mkey(mkey->key)); + __xa_erase(mkeys, mlx5_base_mkey(mkey->key)); xa_unlock_irqrestore(mkeys, flags); - if (!deleted_mkey) { - mlx5_core_dbg(dev, "failed xarray delete of mkey 0x%x\n", - mlx5_base_mkey(mkey->key)); - return -ENOENT; - } MLX5_SET(destroy_mkey_in, in, opcode, MLX5_CMD_OP_DESTROY_MKEY); MLX5_SET(destroy_mkey_in, in, mkey_index, mlx5_mkey_to_idx(mkey->key));