From patchwork Thu Jul 27 16:22:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13330365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60CA7C04FDF for ; Thu, 27 Jul 2023 16:23:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 640BF6B0080; Thu, 27 Jul 2023 12:23:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EFAC6B0081; Thu, 27 Jul 2023 12:23:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4698C6B0082; Thu, 27 Jul 2023 12:23:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2E7826B0080 for ; Thu, 27 Jul 2023 12:23:53 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 09874B168B for ; Thu, 27 Jul 2023 16:23:53 +0000 (UTC) X-FDA: 81057913146.30.E993541 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by imf24.hostedemail.com (Postfix) with ESMTP id 1D8D3180026 for ; Thu, 27 Jul 2023 16:23:50 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ZM6siFmc; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690475031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9xKvUZAC/PcwLq4Mn3vj+fKTV8KIinkTpyaNVs2EYco=; b=US2PttDwDnG1tjr3UtfwHQhjzJTTpsEUxUN5OIk3CGbYBvILCsGx0Jj2C/9Mr1F9Lbd0LG fRL+gI81pxIw5wKE4sK9hB0g671PVJvHQIoh0146+NeQUwfEifKNXAZ1IMCNQNwNGKFw6G ggNjF2xprDoq320xWvdTVBwwA3UPgI4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690475031; a=rsa-sha256; cv=none; b=W4aZWMtW92EP4JS8aHez39rmRjYxTSC+BuEG0e1sq2u6ZnKSo2TBEi58Y102aKEUAlDqS0 c8n7gXHGLknE7xm56M2aEiWXpPi+f7J6OPzWjKBgH6FScGiCbeiFnl94QVnyTXvsqv3NVx XtH+0zaR8l44Se6AxVOnxAxeeQqaiko= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ZM6siFmc; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.45 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-1bb5ce75e43so808469fac.3 for ; Thu, 27 Jul 2023 09:23:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1690475030; x=1691079830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9xKvUZAC/PcwLq4Mn3vj+fKTV8KIinkTpyaNVs2EYco=; b=ZM6siFmcEpT22TCKTBC20dzc2m3blvxe+fV7xBWQ5V/W+B8u2sR51MOrUVn0oe9H9r p3SLhh1gZWJjZ41HzedjMK816QkKQbpYiYtUKKn82b4C9atpUvnedHQf+0jDaBf3j77Q tJOmmfcGqfxttLoXBrJCFjVGF8sW0Q572CZ16Ht4fa8chGgIb2Jw6/K1cT/xLpr/JkM5 r3iilsyakH1SBZfuqYO5hwJQP1WMnOayvcj1j6HQ6bVFu643j76ZwjEXMY25LUTF4q1y UHe6Ly/6TR285I3CJYLC6dr76PLwRPjFwQFb634smNKQW1NJjIxSKuDdZhMfZRorHggK kV/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690475030; x=1691079830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9xKvUZAC/PcwLq4Mn3vj+fKTV8KIinkTpyaNVs2EYco=; b=IxtsnzLbTdTVmbEmnMkSLc6jdRoBycV3YEiEhs/8BESNnKTdllK69fyC4jtc0jyUeU j8YE03emO8tNdkg4TmzpOWL6b5ZpAxIPMa6ZOROO0martR6utN3p6+7wIDOvey5RRcm0 sc2xZta8aIm8Q1b+vhDl0Q4Y6UntQaqmdZ1QL2l5Ir4m+ut88gmcrK8h9DQb47ko77AU T9wxOd0VyR3CJ1XoPvX+DiCwN3PmctTs9CPh2xhbocLwNU+ev9ZB14wJxiUj06Fc9sls INREZ0qfiOzH6w2YJdBt+VTPQv6bWVU+0zXFaMwuHmXkEg31cpZGNBBitLYHeBy1OQlE tzXA== X-Gm-Message-State: ABy/qLZ5FlI0T63L9zaxgXeeo16LQJ41GeelhBoZRilWSxufefQKwyLJ PzkTDhX79hAeyF5vIMG6kwsLNA== X-Google-Smtp-Source: APBJJlGe/1LwJSfSBsFM0E1SehZPQyy8thiIa1GfuOSx8pATP8cfV6nW9jnsPOBTG2/9/yJ+wCDfCQ== X-Received: by 2002:a05:6871:593:b0:1bb:5ef8:d606 with SMTP id u19-20020a056871059300b001bb5ef8d606mr3430099oan.41.1690475029900; Thu, 27 Jul 2023 09:23:49 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:ad06]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b004055106ee80sm519047qtj.44.2023.07.27.09.23.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 09:23:49 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Yosry Ahmed , Nhat Pham , Domenico Cerasuolo , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] mm: zswap: kill zswap_get_swap_cache_page() Date: Thu, 27 Jul 2023 12:22:25 -0400 Message-ID: <20230727162343.1415598-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230727162343.1415598-1-hannes@cmpxchg.org> References: <20230727162343.1415598-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1D8D3180026 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 5pzz5zi3yciuzw3oa6d1eq94u3hd7fhd X-HE-Tag: 1690475030-755713 X-HE-Meta: U2FsdGVkX1/iSt2PRGvjUHbV3qD5DjkhAW4kuSoaVm/bFGcbKkJCa92I0IVPbupnt/i6CUB4rns+s77DEG+R6xbgunOaKG1I68F7WpfvRdt0h8Px/hsofTDdlZoBVjiHAwEesEwEzzrcnEDnj8rbI9gzOEFRs65AumnnJXT5p1WWGOKFYhbG9kFxAcIDE2k9npgr7fWawifEX33Q6hxPfEme4pyjmlTPipz/9U4+E1dL9JlMvjaWLw5/ORx8MWSDqGW91lACgWbZtCRyOPqazmi1g3vU4aSrm58ZQpJDVt/uYSbXbVB200VpWcaSD+vo+7prIGTgGcCC9S/kKaydIwyEMGd7xGh/9hd6+/g0NxSfi2EnzbhZ5DLQYThySAlAJGslfVZNHWtUrjDvN5SFPM9VCeOk/hEnf2Mmz/7FLfLU1WDpg74CLfKrzrVhXNs8CsNtzp5Z7EBUVlo857uBI+jEM+7NJFw5WmafBzS7Nfy2VlXTZW7EIEnJ5n0mDWaHyyosLIZ1T2LHEV55GbEtx1z+toCfBhUxPXMsuGaTvHVa+nkevO6LD4jxsJxE3zA5tg2+Mj9N3WBCYTLC38tcl0ouDeqCs3l/Jny9ehiEbWR8eCrOc3oYa+WRGgzJxf0v+robc6Zt7o/1FaK/3OlFVsACPX7rVIe3omyFtgP/RTi1VLMs1dUeif2/l2EoevnCCIOLpz2FR0rs0hisBQVYazd2d42KwQQfYwrbLRLpEiL+mC/pu+4LH7Z0rVKUiG0tuHJ7gaJuaCt1fCeKDErVwp19ufZxlZsKlXkjlCBDsu5hMgV0AtJeE9L+mAaImmWAFOFGJmtUDA4oTLvF6oi3vmXFMV9ikolmpqxXohLpBbtipdeqFxtoEJlT3c/Z24VPsgGTEpReRYbt90tRq3tChWIA6gexvOjz1YkzswGEsRlEu5A8IBgWu3rv6823oFpuH3pZFQ4TDNMnJGlW+h4 +9P7S0Yu /Pa2DDK8s0McloSGMIbOkYBLU4KGYdCWmrUK7/TNyG8j9ui6pGdJLOTY7rVWGkEnaNggFZeisP/dGFFQT+okzAmnu6f7lc2luknCDAYs5m2WZGAhvgMzwn9BPKcswaB8DzaOOmujPGPx2aRewVFfYjFpS8HrjcOJaY54vXJVQlUoZ7WXWJ4+qCLM1pLC77U6OHyky8QzgOMIDSkTZ2Tj7cx+K57g4rMlPlftjITbeC39dWnK01gk+YoerIUSbgfw8McK8JdGv8bZJM1F1pH3irGuZnTVFu0cfFZKXdmNX9jwNuHOSN/NPFR0+qGqs1WDjujI8jBZC0QLLffPVnlR0YpqeCK7c+V0PIGyufT91HHSlqgEBVcp30NA1oLV7ScEFSGWRlJCkayJxvL3N4KfpvKJZ0XjUO9mb3uYZQRV5nphOnVQ20pYsPQHVxbj44PLzlZDQUvaL7wh1QfFcZfVkvN6uxKfWcbr7yVBeTeCMBW5GAE99bmg7m7JbElIr13ZxW6Pl2j3RK8CNvSllfruiDxrfdeAJmVpZFIeW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The __read_swap_cache_async() interface isn't more difficult to understand than what the helper abstracts. Save the indirection and a level of indentation for the primary work of the writeback func. Signed-off-by: Johannes Weiner Reviewed-by: Yosry Ahmed --- mm/zswap.c | 142 ++++++++++++++++++++--------------------------------- 1 file changed, 53 insertions(+), 89 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index e34ac89e6098..bba4ead672be 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1016,43 +1016,6 @@ static int zswap_enabled_param_set(const char *val, /********************************* * writeback code **********************************/ -/* return enum for zswap_get_swap_cache_page */ -enum zswap_get_swap_ret { - ZSWAP_SWAPCACHE_NEW, - ZSWAP_SWAPCACHE_EXIST, - ZSWAP_SWAPCACHE_FAIL, -}; - -/* - * zswap_get_swap_cache_page - * - * This is an adaption of read_swap_cache_async() - * - * This function tries to find a page with the given swap entry - * in the swapper_space address space (the swap cache). If the page - * is found, it is returned in retpage. Otherwise, a page is allocated, - * added to the swap cache, and returned in retpage. - * - * If success, the swap cache page is returned in retpage - * Returns ZSWAP_SWAPCACHE_EXIST if page was already in the swap cache - * Returns ZSWAP_SWAPCACHE_NEW if the new page needs to be populated, - * the new page is added to swapcache and locked - * Returns ZSWAP_SWAPCACHE_FAIL on error - */ -static int zswap_get_swap_cache_page(swp_entry_t entry, - struct page **retpage) -{ - bool page_was_allocated; - - *retpage = __read_swap_cache_async(entry, GFP_KERNEL, - NULL, 0, &page_was_allocated); - if (page_was_allocated) - return ZSWAP_SWAPCACHE_NEW; - if (!*retpage) - return ZSWAP_SWAPCACHE_FAIL; - return ZSWAP_SWAPCACHE_EXIST; -} - /* * Attempts to free an entry by adding a page to the swap cache, * decompressing the entry data into the page, and issuing a @@ -1073,7 +1036,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; struct zpool *pool = entry->pool->zpool; - + bool page_was_allocated; u8 *src, *tmp = NULL; unsigned int dlen; int ret; @@ -1088,65 +1051,66 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } /* try to allocate swap cache page */ - switch (zswap_get_swap_cache_page(swpentry, &page)) { - case ZSWAP_SWAPCACHE_FAIL: /* no memory or invalidate happened */ + page = __read_swap_cache_async(swpentry, GFP_KERNEL, NULL, 0, + &page_was_allocated); + if (!page) { ret = -ENOMEM; goto fail; + } - case ZSWAP_SWAPCACHE_EXIST: - /* page is already in the swap cache, ignore for now */ + /* Found an existing page, we raced with load/swapin */ + if (!page_was_allocated) { put_page(page); ret = -EEXIST; goto fail; + } - case ZSWAP_SWAPCACHE_NEW: /* page is locked */ - /* - * Having a local reference to the zswap entry doesn't exclude - * swapping from invalidating and recycling the swap slot. Once - * the swapcache is secured against concurrent swapping to and - * from the slot, recheck that the entry is still current before - * writing. - */ - spin_lock(&tree->lock); - if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { - spin_unlock(&tree->lock); - delete_from_swap_cache(page_folio(page)); - ret = -ENOMEM; - goto fail; - } + /* + * Page is locked, and the swapcache is now secured against + * concurrent swapping to and from the slot. Verify that the + * swap entry hasn't been invalidated and recycled behind our + * backs (our zswap_entry reference doesn't prevent that), to + * avoid overwriting a new swap page with old compressed data. + */ + spin_lock(&tree->lock); + if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); + delete_from_swap_cache(page_folio(page)); + ret = -ENOMEM; + goto fail; + } + spin_unlock(&tree->lock); - /* decompress */ - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); - dlen = PAGE_SIZE; + /* decompress */ + acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + dlen = PAGE_SIZE; - src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; - zpool_unmap_handle(pool, entry->handle); - } + src = zpool_map_handle(pool, entry->handle, ZPOOL_MM_RO); + if (!zpool_can_sleep_mapped(pool)) { + memcpy(tmp, src, entry->length); + src = tmp; + zpool_unmap_handle(pool, entry->handle); + } - mutex_lock(acomp_ctx->mutex); - sg_init_one(&input, src, entry->length); - sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); - ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; - mutex_unlock(acomp_ctx->mutex); - - if (!zpool_can_sleep_mapped(pool)) - kfree(tmp); - else - zpool_unmap_handle(pool, entry->handle); + mutex_lock(acomp_ctx->mutex); + sg_init_one(&input, src, entry->length); + sg_init_table(&output, 1); + sg_set_page(&output, page, PAGE_SIZE, 0); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); + ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + mutex_unlock(acomp_ctx->mutex); + + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); + else + zpool_unmap_handle(pool, entry->handle); - BUG_ON(ret); - BUG_ON(dlen != PAGE_SIZE); + BUG_ON(ret); + BUG_ON(dlen != PAGE_SIZE); - /* page is up to date */ - SetPageUptodate(page); - } + /* page is up to date */ + SetPageUptodate(page); /* move it to the tail of the inactive list after end_writeback */ SetPageReclaim(page); @@ -1157,16 +1121,16 @@ static int zswap_writeback_entry(struct zswap_entry *entry, zswap_written_back_pages++; return ret; + fail: if (!zpool_can_sleep_mapped(pool)) kfree(tmp); /* - * if we get here due to ZSWAP_SWAPCACHE_EXIST - * a load may be happening concurrently. - * it is safe and okay to not free the entry. - * it is also okay to return !0 - */ + * If we get here because the page is already in swapcache, a + * load may be happening concurrently. It is safe and okay to + * not free the entry. It is also okay to return !0. + */ return ret; }