From patchwork Wed Feb 15 16:13:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Thomas Hellstrom X-Patchwork-Id: 13141842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D57AEC636D4 for ; Wed, 15 Feb 2023 16:14:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 407C46B0072; Wed, 15 Feb 2023 11:14:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B7B36B0073; Wed, 15 Feb 2023 11:14:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 231C46B0074; Wed, 15 Feb 2023 11:14:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0CFAD6B0072 for ; Wed, 15 Feb 2023 11:14:34 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CA624AB4B0 for ; Wed, 15 Feb 2023 16:14:33 +0000 (UTC) X-FDA: 80470024026.07.0054F3F Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf09.hostedemail.com (Postfix) with ESMTP id A9010140025 for ; Wed, 15 Feb 2023 16:14:29 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=L24Xp9xL; spf=none (imf09.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676477670; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=7zmszHVJAA32GamOEGtobUJ5VinTxr4lT3kV6MGTGao=; b=g0fyB9l3v36fdIcWnhop7LFf9FbxKIgRmMZWeJJsW8bIm1T32u+0wnzETBYFIrO/gA0b2H S+7ut8HPSzHWKBOCfTdZSMFVdso5C30OFYpw4wM9RxZEdzNxV0lgWeWNUnoljE4c58nXKS e3rZqCHCmuodtdfxGO3ocs5DicqZEWM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=L24Xp9xL; spf=none (imf09.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 134.134.136.31) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676477670; a=rsa-sha256; cv=none; b=4zVNbSPwEon5IzwqY2KtLqDv4J0HmEm7Y4OhPnhsB9GQhjYUCA8t082Vi1QeVLgW3X0gEE MNqcuC2i7vfeZrkT98lHUn628Jc/VMBrhIhOHXSe3ZF0cYCcLbSRD3NwZ6wcxxPPoHGw9h Xf2gmYwonnNgaoJbCVmBpgrUxHXlpNw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676477669; x=1708013669; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=63jEmIdqFGbeZAGy9F/2+7ZHTEwokckWrPhHLYhSYF4=; b=L24Xp9xLlQSSEcUl9xZacQQ8PVg0WRmEZabI7ag6lKYn3jazIYaYjztR XnD7YswPf/4np3d/wOOHNYRrXOkxVtieqK3PSVGuGotTFD+6DXbpit3Il 5oKmLDwA5y12UlI4saE21DBYgDdaCrqFnoCHe/IVKDPOm5QYiqVBi+0u2 88c7+sPmhVpLzlbQApxjVIPhn7eUp7e/7m3AUtVMBa1uYK6RMqd3AyEpb ltz+ev5XdZk8ySd0M4HdNLDNUIhYSz7hXL7NB6IxCGHjVsB13yxUX8ZL7 MeSw/7fnShnpNHBTwLBegvZ/WHn5NRNmrxyiYLN+4YiLwaUrUzuGRtwjQ g==; X-IronPort-AV: E=McAfee;i="6500,9779,10622"; a="393870617" X-IronPort-AV: E=Sophos;i="5.97,300,1669104000"; d="scan'208";a="393870617" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2023 08:14:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10622"; a="758471611" X-IronPort-AV: E=Sophos;i="5.97,300,1669104000"; d="scan'208";a="758471611" Received: from auliel-mobl1.ger.corp.intel.com (HELO thellstr-mobl1.intel.com) ([10.249.254.14]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2023 08:14:22 -0800 From: =?utf-8?q?Thomas_Hellstr=C3=B6m?= To: dri-devel@lists.freedesktop.org Cc: =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Andrew Morton , "Matthew Wilcox (Oracle)" , Miaohe Lin , David Hildenbrand , Johannes Weiner , Peter Xu , NeilBrown , Daniel Vetter , Christian Koenig , Dave Airlie , linux-graphics-maintainer@vmware.com, linux-mm@kvack.org, intel-gfx@lists.freedesktop.org, Dave Hansen , Matthew Auld Subject: [RFC PATCH 00/16] Add a TTM shrinker Date: Wed, 15 Feb 2023 17:13:49 +0100 Message-Id: <20230215161405.187368-1-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.38.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A9010140025 X-Stat-Signature: qtg7bpoe6kg5pe7qf31a1oogzurieash X-HE-Tag: 1676477669-371457 X-HE-Meta: U2FsdGVkX1+Gnm3cnv+kY5E1gpRwcr8YC+CYbIDA8BxVjI10M6gqUuUg8ychVFNXgZr2XJLWfGOnpoU1C71Ye9h3IAGtjycqlCnZ17N8h48et/nZAjv4TWqMIp/SuowPy7kiVu5boTa7Mj7U58iF+ijVAQnp/OXlSD3rCmqHVhaUeMfnh9St58gSIHPkbiziyKYWxwyLjHNqZZfZu/RWLuMs7H4jw2dEdwZSnEjDGtvEZ4ey0+WQcr1Ug6YuFEf7MyyqXa28THRHXqku0vhE3rCYs65COub8eIzqpavssgLzYm4yA4pT8/ZBA1btlOHtkNYMgrk5fiWXrwhf8SsQmwLIsSVVWNbgpqK41ts1RVl5SeK9O2m/io9OY7rlAqJBRBRU7rlqRSx3gIMcy5R7sIToaYokNAg/84VtzoOrElODqmATdWk79EKKcVItW/p3YKAOxO9n/1HwDa53Pylbg3EvBadQwi+wn2nn3Zk2+8SGiyiLqSmy8qZagQrRIezpqYTMBssfen6yDi0CmfsfSlq9DvLtikCmqIEkaAkVbON6H96SvljoOmwbMmNI3IDZV2/bIF+nZ9YmV2UZpaA/0/k9ljVOBiRqaroFO4EyAzLBi6eXzxggJ1CwHJllbI8IfFQJxJZ6p330SpfCDkEwRNa6oKk8Xb0XA0d79wlBlB74eUcx4/OrRVZUb+pJEZEc0gnMdPIGGN40V8dE9YZnl/3BvttSZtyjAIHz9qegLes+Qky+FqkJnlZhvpny6pkYcPef+T9q2CP6vRKSUGmvPzC3Wy3I6UKMN4Z2rCvWy7J3d+dV78HcVUltwk+lR8JMsaBJlfQyys73nQMEjHQydwmnG8Qf6DRfU8Otjl7HZF7JXHTrOurKR9sxBBnpAFIWst4qK3PIVTkkNPuLqnELdSBH9q/wR9IPr++SSY+e/GJcdezrZ+yCImY4mE9qcb3mxV/YuSo6zSLcB2zMs94 mNShmBIm EyD/eBP/toeGoEa0JJBaPMRpvK50GTMrCeOxC7mMsh7iMxoVprYGxilY7AlVNSbxSk79kB/YhuSyawUDV+FcfT/DKFBMNdOcMH1BczGdw3dBXX13hzWXc8seleu2s5QUC2yOiF1U0IvFf3GqRFdfLRkzM6m35IlNSQXHXGspKUIdVAr6i1b9jvPh3WAod7iNK+jJJ8RS/xX4gt2OKtUcsPo0IOXfp3tLQ5hHq3OCXJzQJCZ4c1rhZEosTD6WzinDye/O10q7GsMG8ejfcYGMEA88vr+TsGamlAGuRag2vlFY6LboesVXw37cgOdO1gVtXg6Bd9RbVQgv26xD+96kL0lpR7ifAmHmaRLigZXCXuXp0fBuhK5iNBj5wOHJkbXmYl2rA0zm0aT5hhj7h7Nul4Jpulf3GqA/hOd5lsrLh/Qt9qgqptcz6co6VlicNg/LdmMN35xwVkacfaNxEUMt51gezLZd4mh2hbcx8TAwXtrt5a1nvcsJeo7HOD0D5VHt2+PKagHLo0xtd9AtmgVvjMIdHwsK0dT+7ckucFgEu8mUvm8eAMtPPZExNuA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This series introduces a TTM shrinker. Currently the TTM subsystem allows a certain watermark fraction of system memory to be pinned by GPUs. Any allocation beyond that will cause TTM to attempt to copy memory to shmem objects for possible later swapout so that that fraction is fulfilled. That unnecessarily happens also on systems where swapping is not available, but still works reasonably well in many cases. However there is no way for the system to swap out all of graphics memory even in situatons where graphics processes are suspended. So add a TTM shrinker capable of moving graphics memory pages to the swap cache for later laundring and free, and, in the case there is no swap available, freeing graphics memory that is kept around for caching purposes. For devices where the shrinker is active, the watermark fraction is disabled, but for devices not (yet) supporting shrinking or using dma_alloced memory which we can't insert into the swap-cache, keep it around. Each driver needs to implement a callback to enable the shrinker for its devices. Enable it for i915 as a POC. Will also be used by the new Intel xe driver if accepted. The parts of the series mostly needing consideration and feecback is *) The mm part, inserting pages into the swap-cache. Is it acceptable and, if so, correct? It *might* be possible we can do without this part, but then we'd have to be able to call read_mapping_page() and trylock_page() on non-isolated shmem pages from reclaim context, and need to be able to recover from failures. *) The TTM driver callback for shrinking *) The additional TTM functions to mark buffer-objects as not needed, but good to have around for caching purposes. *) Swapin doesn't lose content on error and is also interruptible or at least killable ATM. This complicates helpers. Should we drop this and just drop content on error, and wait for swapin uninterruptible? The TTM pool code could indeed do without additional complication... *) Is there a better way to do shrink throttling to avoid filling the swap-cache completely. *) Is it good enough for real-world workloads? The series has been tested using the i915 driver with a 4GiB VRAM DG1 on a system with 14GiB system memory and 16GiB SSD Swap, and using an old igt-gpu-tools version, 8c0bb07b7b4d, of gem_lmem_swapping which overcommits system memory quite extensively Patch walkthrough: Initial bugfixes, could be decoupled from the series. drm/ttm: Fix a NULL pointer dereference. drm/ttm/pool: Fix ttm_pool_alloc error path. Cleanups and restructuring: drm/ttm: Use the BIT macro for the TTM_TT_FLAGs drm/ttm, drm/vmwgfx: Update the TTM swapout interface drm/ttm: Unexport ttm_global_swapout() Adding shrinker without enabling it: drm/ttm: Don't use watermark accounting on shrinkable pools drm/ttm: Reduce the number of used allocation orders for TTM pages drm/ttm: Add a shrinker and shrinker accounting drm/ttm: Introduce shrink throttling drm/ttm: Remove pinned bos from shrinkable accounting drm/ttm: Add a simple api to set/ clear purgeable ttm_tt content Adding the core mm part to insert and read-back pages from the swap-cache: mm: Add interfaces to back up and recover folio contents using swap. TTM helpers for shrinking: drm/ttm: Make the call to ttm_tt_populate() interruptible when faulting. drm/ttm: Provide helpers for shrinking. drm/ttm: Use fault-injection to test error paths. Enable i915: drm/i915, drm/ttm: Use the TTM shrinker rather than the external shmem pool Any feedback greatly appreciated. Thomas Cc: Andrew Morton Cc: "Matthew Wilcox (Oracle)" Cc: Miaohe Lin Cc: David Hildenbrand Cc: Johannes Weiner Cc: Peter Xu Cc: NeilBrown Cc: Daniel Vetter Cc: Christian Koenig Cc: Dave Airlie Cc: Cc: Cc: Thomas Hellström (16): drm/ttm: Fix a NULL pointer dereference drm/ttm/pool: Fix ttm_pool_alloc error path drm/ttm: Use the BIT macro for the TTM_TT_FLAGs drm/ttm, drm/vmwgfx: Update the TTM swapout interface drm/ttm: Unexport ttm_global_swapout() drm/ttm: Don't use watermark accounting on shrinkable pools drm/ttm: Reduce the number of used allocation orders for TTM pages drm/ttm: Add a shrinker and shrinker accounting drm/ttm: Introduce shrink throttling. drm/ttm: Remove pinned bos from shrinkable accounting drm/ttm: Add a simple api to set / clear purgeable ttm_tt content mm: Add interfaces to back up and recover folio contents using swap drm/ttm: Make the call to ttm_tt_populate() interruptible when faulting drm/ttm: Provide helpers for shrinking drm/ttm: Use fault-injection to test error paths drm/i915, drm/ttm: Use the TTM shrinker rather than the external shmem pool drivers/gpu/drm/Kconfig | 11 + drivers/gpu/drm/i915/gem/i915_gem_object.h | 6 - .../gpu/drm/i915/gem/i915_gem_object_types.h | 6 - drivers/gpu/drm/i915/gem/i915_gem_pages.c | 5 +- drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 273 ++------- drivers/gpu/drm/i915/i915_gem.c | 3 +- drivers/gpu/drm/ttm/ttm_bo.c | 45 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 19 +- drivers/gpu/drm/ttm/ttm_device.c | 85 ++- drivers/gpu/drm/ttm/ttm_pool.c | 522 ++++++++++++++++-- drivers/gpu/drm/ttm/ttm_tt.c | 336 +++++++++-- drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 3 +- include/drm/ttm/ttm_bo.h | 4 +- include/drm/ttm/ttm_device.h | 36 +- include/drm/ttm/ttm_pool.h | 19 + include/drm/ttm/ttm_tt.h | 57 +- include/linux/swap.h | 10 + mm/Kconfig | 18 + mm/Makefile | 2 + mm/swap_backup_folio.c | 178 ++++++ mm/swap_backup_folio_test.c | 111 ++++ 21 files changed, 1361 insertions(+), 388 deletions(-) create mode 100644 mm/swap_backup_folio.c create mode 100644 mm/swap_backup_folio_test.c