From patchwork Tue Jan 7 22:22:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13929632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92D86E77199 for ; Tue, 7 Jan 2025 22:22:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 154586B008C; Tue, 7 Jan 2025 17:22:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1059C6B0095; Tue, 7 Jan 2025 17:22:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0F136B0096; Tue, 7 Jan 2025 17:22:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CA6CB6B008C for ; Tue, 7 Jan 2025 17:22:44 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 17A471A0E03 for ; Tue, 7 Jan 2025 22:22:44 +0000 (UTC) X-FDA: 82982081448.18.4ECD4ED Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 399F820010 for ; Tue, 7 Jan 2025 22:22:42 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Dy8jD6Mf; spf=pass (imf13.hostedemail.com: domain of 3MKl9ZwoKCLszptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3MKl9ZwoKCLszptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736288562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8OEPHYADGiebrrUIJmwP++YfNdezEkFFL7taBOiv4vY=; b=kfkaRgOh77y7ImvDBU81MwjB4pNonUJ2ghelsBQTsvVESK9q4YiuJG0szexXUT4C4M3j9G h+vKS9mpLYkCuyD0biT6HyYd4zijSt1MrJGOeZf9e2fXrs47XuM7TzjmB4YC8lTIzFFzOI L9Rpwm/Kw6SNvnEQ93B1wB55Vh18m6w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736288562; a=rsa-sha256; cv=none; b=ScL8v5Lp7lWHJeeBkdUE2c5JPaNBQrg8GQKGBSQI4R6pQKxw27RC07Xt9zduKkqMCvRXWk ekaW6KmOgiwTQZgfpFQ+vbGYF6YheMpwoQjRM6uueV/YjVAArRAaI6Aq5GHhnByKUWkCpR GTbycbp3CDAvXR7fb/k4gMhZhLa1AQE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Dy8jD6Mf; spf=pass (imf13.hostedemail.com: domain of 3MKl9ZwoKCLszptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3MKl9ZwoKCLszptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2166f9f52fbso363339525ad.2 for ; Tue, 07 Jan 2025 14:22:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736288561; x=1736893361; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8OEPHYADGiebrrUIJmwP++YfNdezEkFFL7taBOiv4vY=; b=Dy8jD6MfKOoEquQVVlAdv1c7ePfk8be1Nmz+sDwB0DVt1CSFMbWp47qs4Bw0srpLX4 SqKDDYUnSpnAuoTfGMVV3BRy7anaiEq9e2g3VitWtq9n7qPXItR8Y0QfdhxvZ7LRzvCY CD7iJXZi8IFfs9Jyi/ozoxLJZ8LkH5OPb7RdVsEuwXHJEMR4TBnTc6ECPMvObL1FWKrH qeQTp9/iazuU8I1IWZL+cKHUcjBJXIhd6sBjjLzsN6r5om69q7OQefJ1DZge1CY7IYpS Yh/hggbGVUuNnOvUHVjLsdFs+O52WCHP7NaHldegXytUeT/yKoepcDIMUPEgAXIWTH89 tFWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736288561; x=1736893361; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8OEPHYADGiebrrUIJmwP++YfNdezEkFFL7taBOiv4vY=; b=MFhJx+yck9JlAEhfv74lX4l/Te7PvuLdFXvRt0ml10M4hn8ptfmFF/L0+YA51jla2J uGcgdIhV0mMa/UdHmpt8X+klly6NzIvxe+yyayt4YVWx9MsQO6ml1kqCsLlbprOdcVMr K2LiMIqkk5sgzuruzNebtiqHuaEkHsTS2NaVwbRJtUMoSwFNR2EeuI0p2O1Ee7KCNjQ0 sLLlVVkryubYYi/S9G7ddmZxuCxEKA/xi1Z1SYQlpZmrehBnT/+ZW0QGGH6KXQozlBFD CRp1mYXsLDg4skaWH1YUlkqdKRHRgy2/TR5EMQgie6p96bZvJjNHGNyGRDbNAyW/ww92 rT3g== X-Forwarded-Encrypted: i=1; AJvYcCUp2d1CKnYKc2C5EKB1F1DjFJY0t7OPKrjf0t3ayhsbVzynLBummkpHVB+YRK2qwGYIcF7Q/vf9Aw==@kvack.org X-Gm-Message-State: AOJu0Yz3CaLr182OMHpWvwZ9aZ0usXrNjWrxMSeaDWC0aS++UP3H9ZNt n5FmgdVWY6Ohql5XLKoH+ezhKPiLdNfTC8rkXeA+R4N2gsZrCeHc64O5XLod/YOZzbXGlKCkfiG NUpy2UmkPw+wumwK+pQ== X-Google-Smtp-Source: AGHT+IEgs5IL10xqkJJftn7yhmclPLhI6k7/GpQT4G68VnNLJlk+dMe1GT3gZ8d2wJgSYVs0Bs7gZdOCLhcukz9L X-Received: from pfwo11.prod.google.com ([2002:a05:6a00:1bcb:b0:725:f045:4714]) (user=yosryahmed job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:c681:b0:1e6:44b4:78ab with SMTP id adf61e73a8af0-1e88d0edb56mr1529448637.8.1736288560944; Tue, 07 Jan 2025 14:22:40 -0800 (PST) Date: Tue, 7 Jan 2025 22:22:35 +0000 In-Reply-To: <20250107222236.2715883-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20250107222236.2715883-1-yosryahmed@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250107222236.2715883-2-yosryahmed@google.com> Subject: [PATCH v2 2/2] mm: zswap: disable migration while using per-CPU acomp_ctx From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , Vitaly Wool , Barry Song , Sam Sun , Kanchana P Sridhar , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , stable@vger.kernel.org X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 399F820010 X-Stat-Signature: cptaneaaam3n8xqysoop7srfn3d4m5bp X-Rspam-User: X-HE-Tag: 1736288562-995300 X-HE-Meta: U2FsdGVkX19f5qmiuVbhXnlhJ2jPOa0+YvfuNhz3yUSGyz9qeB6Rmr0WaRnvLK/uSTQDk/neYgcgaqyWffXnVhHfn984gFtn+Im/wkBJhIBTJbD62RSIb5aOd0a7cmxmfKvz0hktNCnSEHtrKQzkL2MRAA0BP6+mWhczVAfbailSohzjtL6JyHCdesC64IqxJiJ8Mq5xXUNSJt+8UaOrxe4T1x++CDyjX1UiwHryElCOnDfvK8SwChxCo5abs7XZVDFlCqXJuxLsBlRqx/zkrEvvcn0nPBUavDcBFB3ad7/a/efBRIR++uds1G+yQrFpGb1ET1ftfaMYpCWNPuxkIP79/H2cdng9IOV+9IWNsT8Uc2PwDyjEu0EjnQO9t1+QUmYegEBkZW3g/rj3UJabd4h5n+6gvoxLMWBxhJM34m7vdWHbVaqhH5FoRNvG+iIGPz0QYIGbIMcgKuUD2zQiq0dizJUXD8FcBdGj+S1rfbzYHgr8tfifjtb9RwgK7A0SopMm0v4+CST6A9AOrlvR8wieks1toMQGV8ZKHXBH7K+b2mgk1ys1XZQ+6Xozrsc2iNw/+7aCxm+LylHAQ6CdV7oq/2J3XNLpOEnngh/iSoPOf3TTnqx8+ZcOWdJkcD8budXw/aOYzbScSY3rx1sdXqUZmFqrKBtPIHkt8ZATPmKXi+HKbUfjmZgoL/VifexrSVyZjbU+LVsSeMDRZTTYr5TVzMsSnp1kXmvOiOno1B/hnV7Y9UWwPqJSM7lk9FSheQksrIAc+ntUILv9BmlAAP5UE9bDVoDBOo39+ipD9YCzkYSdiWIjuk0q+9gN2rxx9EGPIgZEZ4KUZx8wXqgiGrZc2s5cSd4ave30/woJA/RyRk1D5eQN93ehUXMcLN+aT/QjYBMSxQ2vYTjidNcSzwHBTgIJhn+0eI7hKHBbzOU9TRt+AT08+zdhaXz0Sx/VOuucE/p4Bdx8TVN2meb Wv58EIDI WHeUk7ZhRjSL0Y5Q526quB2Sw+hLSjxGnTyfXm97IIWf8SqTEztdz6IFn4RnMOarHKgYXDnElZQOGIF2j+53X59CiwxojyY7KGPrRPBGEcjW1NnQwex7Sne5ydQhN2LYX38HUl262Guh97qF1cFOiyNPk+MIrMwxKOZsEhyxxiTnlXw90L0asxCE9ph9CPDqWlVE4Ax2VDaTbyv6EsbLiOYLpH77Ier5E6Ebg8CBiANdDAChsKPvJMMEA9kC8sHrMVlf+tgwO0YYNcwiaGMKMz6PKyf21Vvl7q96+TBgdTAqzQNTuqT8u/0ONrYoIAexKAcaNbTHoo/bH+GTewrY0DL31GEXtA00MdioJDI50jxFR+2jxHFmSIyEiy4iUV6Er5ZS4AMozMZjlt71mGOXkJTsekLADufQzTiKMI0R4wtEBxyoPvDqNsWwcVwcI++S54iuBMv3kYSsYCSxyiA6p+SzD3B211Y0dM2Xk6INDfnxh2T1pjFNpF6sdGo6Fh+7QkgICCLS87P5OhdngZhP2IikKUxBeEWU/FIWoNIjFG+hvHmHtVCySyBeIT3y+OLlnleCtUwDu0lEGsRVf/CZbW10eaRB1xOKsEjBYHDofwHyxyrE8zwFOcG4RsZzZmBZxji7jtlwlAJv2WW5XUcDZxdNES9chnwhAjpsnhOFXfX9MPKfGBA+TSNEPVNStPa60kXOBQ5Klm4xvhooH7IcKn9CSZI2027OUn+YI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In zswap_compress() and zswap_decompress(), the per-CPU acomp_ctx of the current CPU at the beginning of the operation is retrieved and used throughout. However, since neither preemption nor migration are disabled, it is possible that the operation continues on a different CPU. If the original CPU is hotunplugged while the acomp_ctx is still in use, we run into a UAF bug as the resources attached to the acomp_ctx are freed during hotunplug in zswap_cpu_comp_dead(). The problem was introduced in commit 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") when the switch to the crypto_acomp API was made. Prior to that, the per-CPU crypto_comp was retrieved using get_cpu_ptr() which disables preemption and makes sure the CPU cannot go away from under us. Preemption cannot be disabled with the crypto_acomp API as a sleepable context is needed. Commit 8ba2f844f050 ("mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx") increased the UAF surface area by making the per-CPU buffers dynamic, adding yet another resource that can be freed from under zswap compression/decompression by CPU hotunplug. This cannot be fixed by holding cpus_read_lock(), as it is possible for code already holding the lock to fall into reclaim and enter zswap (causing a deadlock). It also cannot be fixed by wrapping the usage of acomp_ctx in an SRCU critical section and using synchronize_srcu() in zswap_cpu_comp_dead(), because synchronize_srcu() is not allowed in CPU-hotplug notifiers (see Documentation/RCU/Design/Requirements/Requirements.rst). This can be fixed by refcounting the acomp_ctx, but it involves complexity in handling the race between the refcount dropping to zero in zswap_[de]compress() and the refcount being re-initialized when the CPU is onlined. Keep things simple for now and just disable migration while using the per-CPU acomp_ctx to block CPU hotunplug until the usage is over. Fixes: 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") Cc: Signed-off-by: Yosry Ahmed Reported-by: Johannes Weiner Closes: https://lore.kernel.org/lkml/20241113213007.GB1564047@cmpxchg.org/ Reported-by: Sam Sun Closes: https://lore.kernel.org/lkml/CAEkJfYMtSdM5HceNsXUDf5haghD5+o2e7Qv4OcuruL4tPg6OaQ@mail.gmail.com/ --- mm/zswap.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb236..ecd86153e8a32 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -880,6 +880,18 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } +/* Remain on the CPU while using its acomp_ctx to stop it from going offline */ +static struct crypto_acomp_ctx *acomp_ctx_get_cpu(struct crypto_acomp_ctx __percpu *acomp_ctx) +{ + migrate_disable(); + return raw_cpu_ptr(acomp_ctx); +} + +static void acomp_ctx_put_cpu(void) +{ + migrate_enable(); +} + static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -893,8 +905,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, gfp_t gfp; u8 *dst; - acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - + acomp_ctx = acomp_ctx_get_cpu(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); dst = acomp_ctx->buffer; @@ -950,6 +961,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, zswap_reject_alloc_fail++; mutex_unlock(&acomp_ctx->mutex); + acomp_ctx_put_cpu(); return comp_ret == 0 && alloc_ret == 0; } @@ -960,7 +972,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) struct crypto_acomp_ctx *acomp_ctx; u8 *src; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + acomp_ctx = acomp_ctx_get_cpu(entry->pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); @@ -990,6 +1002,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) if (src != acomp_ctx->buffer) zpool_unmap_handle(zpool, entry->handle); + acomp_ctx_put_cpu(); } /*********************************