From patchwork Tue Nov 12 20:22:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 11240023 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3BB3614ED for ; Tue, 12 Nov 2019 20:22:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED80E21A49 for ; Tue, 12 Nov 2019 20:22:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="SdP44dA/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED80E21A49 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1655E6B0006; Tue, 12 Nov 2019 15:22:50 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0D58A6B0007; Tue, 12 Nov 2019 15:22:50 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1EB96B0006; Tue, 12 Nov 2019 15:22:49 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0194.hostedemail.com [216.40.44.194]) by kanga.kvack.org (Postfix) with ESMTP id D94D36B0003 for ; Tue, 12 Nov 2019 15:22:49 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 9A14D180AD820 for ; Tue, 12 Nov 2019 20:22:49 +0000 (UTC) X-FDA: 76148748858.06.cause49_189e80be2bf3b X-Spam-Summary: 2,0,0,feb48f4979b8d673,d41d8cd98f00b204,jgg@ziepe.ca,::jglisse@redhat.com:rcampbell@nvidia.com:jhubbard@nvidia.com:felix.kuehling@amd.com:linux-rdma@vger.kernel.org:dri-devel@lists.freedesktop.org:amd-gfx@lists.freedesktop.org:alexander.deucher@amd.com:bskeggs@redhat.com:boris.ostrovsky@oracle.com:christian.koenig@amd.com:david1.zhou@amd.com:dennis.dalessandro@intel.com:jgross@suse.com:mike.marciniszyn@intel.com:oleksandr_andrushchenko@epam.com:petrcvekcz@gmail.com:sstabellini@kernel.org:nouveau@lists.freedesktop.org:xen-devel@lists.xenproject.org:hch@infradead.org:jgg@mellanox.com,RULES_HIT:2:41:69:152:355:379:541:800:960:967:968:973:988:989:1260:1277:1311:1313:1314:1345:1437:1515:1516:1518:1535:1593:1594:1605:1606:1730:1747:1777:1792:1801:2194:2199:2393:2525:2559:2565:2570:2682:2685:2693:2703:2859:2895:2901:2933:2937:2939:2942:2945:2947:2951:2954:3022:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4119:4250:4321:436 2:4605:5 X-HE-Tag: cause49_189e80be2bf3b X-Filterd-Recvd-Size: 8802 Received: from mail-qv1-f68.google.com (mail-qv1-f68.google.com [209.85.219.68]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 12 Nov 2019 20:22:49 +0000 (UTC) Received: by mail-qv1-f68.google.com with SMTP id x14so6962838qvu.0 for ; Tue, 12 Nov 2019 12:22:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=GWMSOcigcFkvS1mXDN4UZKv/Sz4qJBz/Xw1txpllC58=; b=SdP44dA/qt3Cafkmp5tDYwUfTKua1RYFbjfub38WMqp1Vetd5HDAMxGgKpbA4Ao+oK gsEL1oHkgBYSL5wAFUvGW4WL3U/pvufnqzGsum5CGpKAJYB+qH5Upt6hHPp0oyUiUVZQ luFJjPDkpYMu/LEwn7P4i1KvQc3sG0kQARArCClpXMObiDqr/9YyrAyqL0dxb++bMYDb XbTfpT4OBOBeoefNdIfWJ+fDQI8fjcFqTYyS9qXYt67Hx6qKRVE8+RXhoDoz1lpBwD/V slDxBVQ6lEnBCWWDRENPVOvt6nSoUCSoIGfYhgVkElAgGx2oznQ/Ekwnr6eHH2zAzayA jOxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=GWMSOcigcFkvS1mXDN4UZKv/Sz4qJBz/Xw1txpllC58=; b=Jsl+aHFUD1MXsHZAliXb+rsukj1CCH+glYpITgbH7pTjjVG44yeuOwxT6j9FbN4/UX J//rrqKTJyg4JuZ5r2iXzHmcN1+yWpryq4d+zqk0Mpbk7mDgtVnzj0ppNfO99toL7SGD XmJjVx9Cek44dmjNnfG1Pk03AmJVIVt6LZ0bbjmaENJrmVRGVKjP35zBdH0gkBWNRQOU G+/gvJqXht6goqwbX3/E0XD0enGvfX4vhVRCK7g9gk0S+zFNrbeZncwN4SE4Xau8az+M Ezp2ZTxY4xyC0R/14EM+vbkC79uuqWdaI6NTfhZxvUAQY9zf5YnAKmiZs4XkqwvsNqqr 2n7A== X-Gm-Message-State: APjAAAULvHtUqroUnTUTgPhmwFRb8mkEAvroVYDGJRC+zUpsv3BEJAKZ Iu0gm40eJql4jtm5QeWi4R0fPJAvJnI= X-Google-Smtp-Source: APXvYqyZGwIPop6xwsfhsv3DLyXXAOZa8Eq5KSgCu6w55310PeMbie/nmCz5Pj6znsHarv3wD6pCCQ== X-Received: by 2002:a05:6214:14b2:: with SMTP id bo18mr30642652qvb.72.1573590168105; Tue, 12 Nov 2019 12:22:48 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-162-113-180.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.180]) by smtp.gmail.com with ESMTPSA id j71sm10881994qke.90.2019.11.12.12.22.47 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 12 Nov 2019 12:22:47 -0800 (PST) Received: from jgg by mlx.ziepe.ca with local (Exim 4.90_1) (envelope-from ) id 1iUcgZ-0003jU-35; Tue, 12 Nov 2019 16:22:47 -0400 From: Jason Gunthorpe To: linux-mm@kvack.org, Jerome Glisse , Ralph Campbell , John Hubbard , Felix.Kuehling@amd.com Cc: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, Alex Deucher , Ben Skeggs , Boris Ostrovsky , =?utf-8?q?Christian_K=C3=B6ni?= =?utf-8?q?g?= , David Zhou , Dennis Dalessandro , Juergen Gross , Mike Marciniszyn , Oleksandr Andrushchenko , Petr Cvek , Stefano Stabellini , nouveau@lists.freedesktop.org, xen-devel@lists.xenproject.org, Christoph Hellwig , Jason Gunthorpe Subject: [PATCH hmm v3 00/14] Consolidate the mmu notifier interval_tree and locking Date: Tue, 12 Nov 2019 16:22:17 -0400 Message-Id: <20191112202231.3856-1-jgg@ziepe.ca> X-Mailer: git-send-email 2.24.0 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jason Gunthorpe 8 of the mmu_notifier using drivers (i915_gem, radeon_mn, umem_odp, hfi1, scif_dma, vhost, gntdev, hmm) drivers are using a common pattern where they only use invalidate_range_start/end and immediately check the invalidating range against some driver data structure to tell if the driver is interested. Half of them use an interval_tree, the others are simple linear search lists. Of the ones I checked they largely seem to have various kinds of races, bugs and poor implementation. This is a result of the complexity in how the notifier interacts with get_user_pages(). It is extremely difficult to use it correctly. Consolidate all of this code together into the core mmu_notifier and provide a locking scheme similar to hmm_mirror that allows the user to safely use get_user_pages() and reliably know if the page list still matches the mm. This new arrangment plays nicely with the !blockable mode for OOM. Scanning the interval tree is done such that the intersection test will always succeed, and since there is no invalidate_range_end exposed to drivers the scheme safely allows multiple drivers to be subscribed. Four places are converted as an example of how the new API is used. Four are left for future patches: - i915_gem has complex locking around destruction of a registration, needs more study - hfi1 (2nd user) needs access to the rbtree - scif_dma has a complicated logic flow - vhost's mmu notifiers are already being rewritten This is already in linux-next, a git tree is available here: https://github.com/jgunthorpe/linux/commits/mmu_notifier v3: - Rename mmu_range_notifier to mmu_interval_notifier for clarity Avoids confusion with struct mmu_notifier_range - Fix bugs in odp, amdgpu and xen gntdev from testing - Make ops an argument to mmu_interval_notifier_insert() to make it harder to misuse - Update many comments - Add testing of mm_count during insertion v2: https://lore.kernel.org/r/20191028201032.6352-1-jgg@ziepe.ca v1: https://lore.kernel.org/r/20191015181242.8343-1-jgg@ziepe.ca Absent any new discussion I think this will go to Linus at the next merge window. Thanks to everyone to helped! Jason Gunthorpe (14): mm/mmu_notifier: define the header pre-processor parts even if disabled mm/mmu_notifier: add an interval tree notifier mm/hmm: allow hmm_range to be used with a mmu_interval_notifier or hmm_mirror mm/hmm: define the pre-processor related parts of hmm.h even if disabled RDMA/odp: Use mmu_interval_notifier_insert() RDMA/hfi1: Use mmu_interval_notifier_insert for user_exp_rcv drm/radeon: use mmu_interval_notifier_insert nouveau: use mmu_notifier directly for invalidate_range_start nouveau: use mmu_interval_notifier instead of hmm_mirror drm/amdgpu: Call find_vma under mmap_sem drm/amdgpu: Use mmu_interval_insert instead of hmm_mirror drm/amdgpu: Use mmu_interval_notifier instead of hmm_mirror mm/hmm: remove hmm_mirror and related xen/gntdev: use mmu_interval_notifier_insert Documentation/vm/hmm.rst | 105 +--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 + .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 9 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 14 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 443 ++------------ drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 53 -- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 13 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 145 +++-- drivers/gpu/drm/nouveau/nouveau_svm.c | 230 ++++--- drivers/gpu/drm/radeon/radeon.h | 9 +- drivers/gpu/drm/radeon/radeon_mn.c | 218 ++----- drivers/infiniband/core/device.c | 1 - drivers/infiniband/core/umem_odp.c | 303 ++-------- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/hfi1/hfi.h | 2 +- drivers/infiniband/hw/hfi1/user_exp_rcv.c | 146 ++--- drivers/infiniband/hw/hfi1/user_exp_rcv.h | 3 +- drivers/infiniband/hw/mlx5/mlx5_ib.h | 7 +- drivers/infiniband/hw/mlx5/mr.c | 3 +- drivers/infiniband/hw/mlx5/odp.c | 50 +- drivers/xen/gntdev-common.h | 8 +- drivers/xen/gntdev.c | 179 ++---- include/linux/hmm.h | 195 +----- include/linux/mmu_notifier.h | 147 ++++- include/rdma/ib_umem_odp.h | 68 +-- include/rdma/ib_verbs.h | 2 - kernel/fork.c | 1 - mm/Kconfig | 2 +- mm/hmm.c | 276 +-------- mm/mmu_notifier.c | 565 +++++++++++++++++- 31 files changed, 1271 insertions(+), 1931 deletions(-)