From patchwork Tue Jan 2 17:53:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70FECC46CD2 for ; Tue, 2 Jan 2024 17:54:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1366E6B0087; Tue, 2 Jan 2024 12:54:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E8196B00A9; Tue, 2 Jan 2024 12:54:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7AEE6B0087; Tue, 2 Jan 2024 12:53:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D0B946B0081 for ; Tue, 2 Jan 2024 12:53:59 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A5F4C1C1024 for ; Tue, 2 Jan 2024 17:53:59 +0000 (UTC) X-FDA: 81635119398.07.1EF73EB Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf15.hostedemail.com (Postfix) with ESMTP id BDE82A001A for ; Tue, 2 Jan 2024 17:53:57 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hMoaZ1W1; spf=pass (imf15.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218037; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HOgiTUumllZGmGcBS/5D1WFbQ4vDZwA6J8hbWSvm0VQ=; b=7f02MEcw238tlpmwbPf3RR8nfY7bq2YCqPIUPeBilgkMXx3/R86QbspXSAVlGYloWWkO/R 515dBY9/Kn8mxRPA2TKVWNaUOo3mRsw5X8eWJRs/YYmjdcDPSSzSWDc7yXCszFrAJVvL08 la9N8NmVm2mbdf8IP9E1z+GhTR9TJQw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218037; a=rsa-sha256; cv=none; b=y2ZlM2RDlMoVM1JzBG9HZh2I1yAMIUSboOsF4iReM84H81SZAFHzBcIBNimeREUQE18jRk HyrsFy6MButeVu6ffnv2KugBufsjd47WbO4bZ6jqyHm3U5VqQv9stWKeXuISqsrOYHVg9X sN1y7n8hWkOnnm/0pb4hd14/9Cuo6GI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hMoaZ1W1; spf=pass (imf15.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1d3e6c86868so74364545ad.1 for ; Tue, 02 Jan 2024 09:53:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218035; x=1704822835; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=HOgiTUumllZGmGcBS/5D1WFbQ4vDZwA6J8hbWSvm0VQ=; b=hMoaZ1W1w7JlMOnzSmZHLaVEEerIzQ/Gv1jAl4OeQPV52ZSpftG7XpFGSGgClVojkq 9XO7kzr2iBvVu6BhbsdHg5eV2l1iMAwaRioyawL8SsJEYFbjO6Dc22GnDNU53d9REVFw cUDYYyZELe1DVhOrxPrj2KeR04CHxr4OeEHoHg5waJqi0eAcl+PLgjzjZXpjsFPKFJTQ d4kcsD0+2y2jbjJJCdhLXgCllgWCbY8pO7H6QfL9AZzDyXKnIAV08Cy18B5Y9OlE9XhO xfJSxSCp3GUSrY6wsXmTgEX+rqi+35tvJSEjc+rM2XOm/wqoHXNgDG7sqMB2GhoeybwS 2E4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218035; x=1704822835; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HOgiTUumllZGmGcBS/5D1WFbQ4vDZwA6J8hbWSvm0VQ=; b=vNrkCbWlDdH3+DZ1Tmy/kjRyBZVZ41bkLwRUAc1XaCK96Lt2PNAk6YRBJT5e+QsPGM rg/8RLITLymj3cNCU6mXc+PWQkxg/M0nsR+4drPBzL/2uXyvxl+YN/VfgY3ThJ4S95v3 7BpuSdCJy4vn2kFnAQqKFa7InhUTicKY/tHB2WVBvU1BGBEWiLNEE9bd03225s/v1L2S 2lsore7IObtKg6wKA6WCzeBu14PIspYobKzZt9cxs+NVEJ2v0LIHzSSob90iBACizpgW ygCf3C0IIy7aaA3WwtxTuOLhhwYv/b+6OSXyiTETiTuw9ZKkcCM6oQbRfrarvI5vWKcI cnYw== X-Gm-Message-State: AOJu0Ywt02wi74aokwgKKnApg7MDMtvV0YK6b9IDqAfxMW9OCs23rDnJ uS90CplWds4dlamoGO1XvOKPRHKSPkNAJr81 X-Google-Smtp-Source: AGHT+IE9UyXqSDCkq4STavfMkchWm/4gFSM+hqb3udvJPylkrwaXHIW+zWlXliWIUXub6vHeHy9JRA== X-Received: by 2002:a17:902:c409:b0:1d4:cca4:7e81 with SMTP id k9-20020a170902c40900b001d4cca47e81mr724379plk.64.1704218035614; Tue, 02 Jan 2024 09:53:55 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.53.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:53:55 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 1/9] mm/swapfile.c: add back some comment Date: Wed, 3 Jan 2024 01:53:30 +0800 Message-ID: <20240102175338.62012-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: BDE82A001A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: apbzdsp7ok7jhw8c4mumcop3tggps3w4 X-HE-Tag: 1704218037-950747 X-HE-Meta: U2FsdGVkX18PKKHdYAYwiFJzRvCagMXEoX4rexb6g3iKO8vLnWtwtTBeHKwZ1bvyi1xh7+PyqkQtxAQUHIIhvEa2e8KiEL3rbrAmpKgQ9Etn6d8Qk7LJs142MaUBz11qOvfhzv2GP17eg7ArK9BWCcCdVVQcIAP+NAAG+YbdpQ63IIiuhr5R5tWAc4aicXcqYeqHmB8Czu/YXugfpUXirAX3RTA4zhabnO4dmunp6c7Tq5IiEH8nANa4FQNNsJ6tumqy466TH0b6nb7+aIlMXkCGT3/vWwHLYaHe2lnBxfV4IJLD0TYtU5XEgX9CLNjdLGbUXk59uGjNa6bxR0ILcCmtgNjUmeVoxuh7Vl50Ga4cHpjTpvH4J0P3MZ2btBPH9DSrF4FDGIq+Jh7QuQ7qb6sAYlMVSnUtPbidTjvbO0xQDraoHiCe/yKHsIsC3dYXLQTW6ikCS/hu6C3AnQ9cSuPIOXQvk1/b9VTbfOnGEaLXy2TrGkKWGNaU9JBIcKnnQIDnPVqRbwWJbZwgZQdJefBs6nqGF9JBRgSn19bYvsXuW6e+G95mu2MMkhNSGx+5LYVWq8pK8CrLUyQ0JIacyWfl2VzWZtBuX7O2pBJStpBOpr++Pe4/sI5ewD25HnzQbf1mdEHxXIPuy/kbtlvDhrlsqNw76pGCDrnpfaMdKwO75l0zcdZW6ACidHKYBp2kMQjE/nxysM3v+AM0PsWBOsVHKTZdqwfCA0mqoE/fNPVqJdWj73aDO6PGqO0PeftIUIqRehHLLISXBE/M64C6/umJhX8n0ALky9KGcjUByImaRxdU2rs7umWIO6ZdMOa39roQdG8SaH77TO80zF9fOasRzkP1JQ9m5NPs6Z2WyYVjfWmakfxAP8d4hQbarK5SOtLgPPGcKVWBKV1ddaiQPQoGg37w9BNtYiZ6G7ItN/a75ZOcOs2qldRDt+cRAyA/+ihdE/rjUEyutdDiLyv LXjT6A3N c43kEqJdZmcscF/JtSOkWaOERLWG3JvvpkkyhFWhYsbzE/ciIkuRz3GyRsuSbEKYe+ySuOfdK1vseSCTnnvirwPE5QTS7JGRkEzm5bIS2Aj1cx3QMDBI6OzfZeDn2CHoNECDHktKiTQe6ff53vXgVmcVDU/tbI06E4M0qJ4Cd58CMD6NfKkpjR3kpWXWFLJM5VCkXGlYsf8C71vy2NvVfcl1fX788NEZrlNH+Umz/fkaUAqtlqt4c7j/H2Y74oDhIpSQ814KZ2IBLjBtY5nuEExdKZW90G0XW5YMW6zX7ALMdb9ykH+M+q3zLfyw/KhHLIfF76vLVgCzIz2IR2s6K0zq1WxkpTLY+LE4GhLSqfc8Z5FW2sCs8JpGu4xvGNoU4g6+/2hNaCQjliEVhdyxQM5TdqXkO2aaGsZ+TK2WocSVMuj9TD8wOEy4+CJIiRHI0mFzOrtY0tujnvmtGKszlB4Axu0TmsEl/a8udm0kzPSf78ATz6flKfDtbEcclxUX1SK0jl7m0ft2XmcbiL+9hOQKNeO8vfaMyD3lcLl6+y004OGTN+CohUCnXWCWfsNFGV/Qo9tuk4ndxP3LwE/U2l75p5ohatGwrPvVQ2JSrlqOMu9Ord5vMFzvE8b+phUgdzcdK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Some useful comments were dropped in commit b56a2d8af914 ('mm: rid swapoff of quadratic complexity'), add them back. Signed-off-by: Kairui Song --- mm/swapfile.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/swapfile.c b/mm/swapfile.c index 3eec686484ef..f7271504aa0a 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1880,6 +1880,17 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio = page_folio(page); } if (!folio) { + /* + * The entry could have been freed, and will not + * be reused since swapoff() already disabled + * allocation from here, or alloc_page() failed. + * + * We don't hold lock here, so the swap entry could be + * SWAP_MAP_BAD (when the cluster is discarding). + * Instead of fail out, We can just skip the swap + * entry because swapoff will wait for discarding + * finish anyway. + */ swp_count = READ_ONCE(si->swap_map[offset]); if (swp_count == 0 || swp_count == SWAP_MAP_BAD) continue; From patchwork Tue Jan 2 17:53:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9450EC46CD2 for ; Tue, 2 Jan 2024 17:54:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30C566B012C; Tue, 2 Jan 2024 12:54:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BAFA6B012D; Tue, 2 Jan 2024 12:54:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 134486B0131; Tue, 2 Jan 2024 12:54:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 005236B012C for ; Tue, 2 Jan 2024 12:54:02 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C2BACA0864 for ; Tue, 2 Jan 2024 17:54:02 +0000 (UTC) X-FDA: 81635119524.08.765095B Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf16.hostedemail.com (Postfix) with ESMTP id D4E1D18001E for ; Tue, 2 Jan 2024 17:54:00 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iiPuvqDm; spf=pass (imf16.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218040; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=leW6kIEfUqfY3XAn3Ki1C0WiRu5w1qcwcxw36bp08sM=; b=GWlnkUHgMyqHzVzkkjOSZ1+z5O29dUIrAyUibHsW+LXY++PunXu2DK3WRnQ8jZHKc610k8 fhbfGYfJF/m6aI9/WgdrwbqONI2mMfFT+3kQbm/RQuHT34e8r12x0yLaUeax9hDwq/W/vf pjM5Fuzi7LirXKScKYi0vomlQFP1XQw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218040; a=rsa-sha256; cv=none; b=hGTVcHe82nA+cJxCYFGURwRXhitZMp14sjaljjOdcXh4OmAtzet9AZ65p4HIdjpcsrDpFp y44CJMVuCFVtaDA7fDJQ1wDzmsnDHzjkW/InNlEUriikhHfe5OnpkE+3ADaoH7paNXot+t hGHKlK6fEnKRhCRHJTIz40aFwMvU8nk= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iiPuvqDm; spf=pass (imf16.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d3ea5cc137so74551455ad.0 for ; Tue, 02 Jan 2024 09:54:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218039; x=1704822839; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=leW6kIEfUqfY3XAn3Ki1C0WiRu5w1qcwcxw36bp08sM=; b=iiPuvqDmVfJALMtlSoDAD1hK+FsTrhm53QKic8ZGuXJC5xXvSeXgJ5B2Khq0PTotLD XHqfHM4Ex9gknoyLfNzCafUNKo1L/Vh9Ei8ZAm/9c1o7L63EH1Werte20zH0PWeycdj6 QX7AneLykeK9s9jL6tpx1sP5YxnXdOJ0Lwaa8xc/2UkQngUI445ziEDoa0khcixztj+W r/miCpwtCjs7D4o9ZElad5z4coBy6AFjPcZCDA+X762DgzTlV8BaXI24fHrVlB0OE85E FBsp9LvilN+GedMNwdaIMjXGBdXwhvLOuJjl9oKMxRM4WVvtIPrjgPyKADpUcx+QIf1S ts1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218039; x=1704822839; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=leW6kIEfUqfY3XAn3Ki1C0WiRu5w1qcwcxw36bp08sM=; b=dKjeGhvPJ7cj9G5rN0DN3gTElF2oT+eKVUMQ0GezDhrOnToHTjmxHM6WmdXYcCKPPJ bjX1T1TQiVkwEIMe8cdvCucCrzee140+lA05KjwBfDW9Uogc6zVVFAHPAZ4lFH8AARZM RBSk9ueWZ9VU+C/pyJhyNbMggQGx3ZSApMr3gfIe1buqk3M3HWqCz2e00K9c1J7qvk9x W38cTXGxW3te4cEAQacQIHgbMCdvgix3A7py4hxN08hVMIUooFST1nqHExBu0baSzD5q hshMCTMCK8lGKzYOQSNsikVtZ9M4VrQ9HrJ7XIjZ7hWE9r6P0iJCLd+CGl0oXhFlDHww LIcw== X-Gm-Message-State: AOJu0Yz37FjXt/BglEXsQHm0AoMflbIh93QudEx4EuYVX5hUFOOXQ01v gY4ousvKus82v5Vj86PYZHvvpRpaAN9HmCfa X-Google-Smtp-Source: AGHT+IHbiz8m7h29JKdOV+VIY3InvdMjldzD40RepmLTENYklXxpKThd4lbQBZ1H/dFstp+Ax17T1A== X-Received: by 2002:a17:902:a585:b0:1d0:6ffe:a28 with SMTP id az5-20020a170902a58500b001d06ffe0a28mr17151404plb.134.1704218039237; Tue, 02 Jan 2024 09:53:59 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.53.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:53:58 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 2/9] mm/swap: move no readahead swapin code to a stand-alone helper Date: Wed, 3 Jan 2024 01:53:31 +0800 Message-ID: <20240102175338.62012-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: D4E1D18001E X-Rspam-User: X-Stat-Signature: 6zg7etpqq1zst4pxk4fx1o7sd79zix4i X-Rspamd-Server: rspam03 X-HE-Tag: 1704218040-494312 X-HE-Meta: U2FsdGVkX18WmY4rQVd0TN8vc7Ibw59rhbiZoYP7FNU0TAl/odo3pdSkE73OaKIWX1o/pOpfYA4T9BmIVO5YHwPbmNnDmeP+kq2/2BYnKffZtsThdD3Z6C7alJI+Oz8vL/emUKrpHbmAEt2s6sRkt7JREZdP3rIZTSmHQdjSE+RCF4w6HKmyAXdo6kWpO+HV4bMzW+C2UFbRCXgXltZXk4k+sDxRFJMFQqOI13NG/1cXIpSUTO/9t7vAzUqIt9j6vuP3OrJmrPMXnXfaqkKNySd8cFG5Gz3gp/BEoVoQsiYjLl2Ux202cfFFcZxCdAsBxAWgSN53YtKZ92LIiMwRWCiv/J/uTUdo7giJvIxyZ2myONefeCP9yjHuewvmW/E+hH8SjUzJas9I1C46LAnSXHoc+MEVXsQdOJymmzPgJuBUDjcmFzJ5D2bS2C9g7gxlO76b6nJSulDin3H2G6LLuFMCk/OmJXBiIPbrfn5VoBlHAxG/9ggqnbACvH9ddmxKEPGBvHCmuAObPltxpWSciKrfJy1hKQ2DpYyTHWAFZFxYnD29auZ/9MF0tQ4wmmcGpQaCReg8x7MxOikk9Gq1AfV4AdELQEcrE8rdWgGpxijXWZOTP6hL07fudgfJRI2VfOXPm99RonQqqv10fAGlB8Bn9rFkjH3JdgY24KCgM8HSFtd/fsT10mP0ISh132RhZ9x8lhPiCGFNeVBHbMWy8lTyOVhk75JKkkjoaAOwZuMIVGLdQ8PivKyIVykR0ytqxFrhmdWkqAk9173m1Mshz/mooaHshiDXtTcPdzm42lWH+1asQ8kWGHsHmm5bTmeJuV/oj4IXfDnlF6x1b1+1MdCIT9q8s+iYwaW15sck8xBEKITzMxwFFPq2cYQYt/P3IeKiHm89tD9VdM6x0JsDwtnBFWzN6Otn/uTPYM5feyxvEM1sT1Sl+nZiYpq8aO8qZmZrqSzEp1lOnQZxI/S ObeMgfhU TEkznbPGvsddZMbrfpTdox2h/+tlt/YBcK3ZJB4i80VWqFtQ9LsnjQ8+AgR+mVfORgHtXMA4GlOxRDPNmZeXck3Q0nKVVnOHd9P+x6ib4DVz47YFQNMgNjHWZeFn1ZpXKqjL70n1RZ9D/hDPQk0b1Jr5U17D6ICGKwTaHlQWJCGvua9eIIA7M+1bw54nZLHzGxoAk2HL61HXpUYF8liiT5Nj8bRU45S/+xZ06VQDONsC9z4aTW3J+4UwDk1ZNC97Y6bIZ55EUw6AZ5MlpzAs+Nd9QUdZ5wXTYx804GgBQ9hd6uObs5V+Vb85Ew/Kg3wWumvVkTXaDI/QEh10jBl5b8WEXvNK3xJAiuVv/5tcNSVL7OAlVFCP3cvBGXNTStKDrFARB5OD+t2Iw9SiHbCcsnQ1jCMwfFNCcoXhLaP+X8wQaFk7r/OrGtPtac3Y1tqZBpvHYq7gEFf0pXTTQ3W98r4XAL0MzAuRmKjSy3Pa1BHkLMoz4JrCh18+xpV6twka3yLpEnwuOa8vClbCKf/2BVdNjt2/cWnL2tj8ePU98+b6gMiq7TmfTGBirLF5+AL/mIJeUVCFsMPT052tGrQeK5t2HBuIX9Cz27zTEkaxGnfnyVMVNzFSH9ufVHOHF4P/VwzNo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song No feature change, simply move the routine to a standalone function to be re-used later. The error path handling is copied from the "out_page" label, to make the code change minimized for easier reviewing. Signed-off-by: Kairui Song --- mm/memory.c | 32 ++++---------------------------- mm/swap.h | 8 ++++++++ mm/swap_state.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 28 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a0a50d3754f0..0165c8cad489 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3803,7 +3803,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swp_entry_t entry; pte_t pte; vm_fault_t ret = 0; - void *shadow = NULL; if (!pte_unmap_same(vmf)) goto out; @@ -3867,33 +3866,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { - /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); - page = &folio->page; - if (folio) { - __folio_set_locked(folio); - __folio_set_swapbacked(folio); - - if (mem_cgroup_swapin_charge_folio(folio, - vma->vm_mm, GFP_KERNEL, - entry)) { - ret = VM_FAULT_OOM; - goto out_page; - } - mem_cgroup_swapin_uncharge_swap(entry); - - shadow = get_shadow_from_swap_cache(entry); - if (shadow) - workingset_refault(folio, shadow); - - folio_add_lru(folio); - - /* To provide entry to swap_read_folio() */ - folio->swap = entry; - swap_read_folio(folio, true, NULL); - folio->private = NULL; - } + /* skip swapcache and readahead */ + folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); + if (folio) + page = &folio->page; } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); diff --git a/mm/swap.h b/mm/swap.h index 758c46ca671e..83eab7b67e77 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -56,6 +56,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); +struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -86,6 +88,12 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, return NULL; } +struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf) +{ + return NULL; +} + static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, struct vm_fault *vmf) { diff --git a/mm/swap_state.c b/mm/swap_state.c index e671266ad772..24cb93ed5081 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -861,6 +861,53 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, return folio; } +/** + * swapin_direct - swap in folios skipping swap cache and readahead + * @entry: swap entry of this memory + * @gfp_mask: memory allocation flags + * @vmf: fault information + * + * Returns the struct folio for entry and addr after the swap entry is read + * in. + */ +struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct folio *folio; + void *shadow = NULL; + + /* skip swapcache */ + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, + vma, vmf->address, false); + if (folio) { + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + + if (mem_cgroup_swapin_charge_folio(folio, + vma->vm_mm, GFP_KERNEL, + entry)) { + folio_unlock(folio); + folio_put(folio); + return NULL; + } + mem_cgroup_swapin_uncharge_swap(entry); + + shadow = get_shadow_from_swap_cache(entry); + if (shadow) + workingset_refault(folio, shadow); + + folio_add_lru(folio); + + /* To provide entry to swap_read_folio() */ + folio->swap = entry; + swap_read_folio(folio, true, NULL); + folio->private = NULL; + } + + return folio; +} + /** * swapin_readahead - swap in pages in hope we need them soon * @entry: swap entry of this memory From patchwork Tue Jan 2 17:53:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509228 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BE52C47073 for ; Tue, 2 Jan 2024 17:54:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6D716B013C; Tue, 2 Jan 2024 12:54:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D1C846B013D; Tue, 2 Jan 2024 12:54:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBDA26B013E; Tue, 2 Jan 2024 12:54:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A80B86B013C for ; Tue, 2 Jan 2024 12:54:06 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 72F871207AC for ; Tue, 2 Jan 2024 17:54:06 +0000 (UTC) X-FDA: 81635119692.07.1ECC3A8 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf10.hostedemail.com (Postfix) with ESMTP id 8FDBCC0010 for ; Tue, 2 Jan 2024 17:54:04 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IucIqIma; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218044; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5EuDdHpwOIkkgSSbClAkFkJceQknRHYQiFDZ/42t/cQ=; b=ByROHejaM7ZTPFzk5FBD/M/Ce/KrYq9WUwLW5lmW1yPQOTQcgsldYUw250NW+ROyzV0WwB ru78uVr7Q2qG4wGg24L5bfnjg8mFXus0lTX/1DmO41FFKsEd4cl1ZOcIXB6sL7vosvhVuu mGoPAh9fnpul1C3dESxiQBo0CQ+EBdA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IucIqIma; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218044; a=rsa-sha256; cv=none; b=L7nITgbbW9tcfMZQ6S4apqpjqYxMP6iYRevUOCZ88AebPJb0XV7E7kVGm1n71P9rFwOx7W Qswjxhjo+lY26EhL5W2BztkgE61cIGKunIrhRMMPZNEyuvXXzv6LK1+DpKNGv3/g0Fj6G2 t1WsK4x+4rIL8t9fUgrAxBMDCSFyHh8= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1d43df785c2so21398265ad.1 for ; Tue, 02 Jan 2024 09:54:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218043; x=1704822843; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=5EuDdHpwOIkkgSSbClAkFkJceQknRHYQiFDZ/42t/cQ=; b=IucIqImaCG/cI4+vPatsNiBboZpxA8Cseaf9FjGvMPdP3BGEaroPP8oZ4jJAftXpT2 A/A/nKvXCkUSaqqH1QVc2X/fc5KUAVUZSxUlEelAEGD4UQxQvFCJx0wBNx1jhBhmSTIk uiC5djy2O2gk/TArYLjLlNNFyQYcm88aq931Do9/86rym8HMSnH1I3icHqQS+InzMuAK 1MGBP1QFe0zHwzXmOjA7eTqeTLXf/MNjC6ZYKVMYlPxzSA8OHpnXCQzAY7KkIl+3fvK2 vnyDF56ieGtA0v0U8AapP4T3aCxi0YkbdEhNprFT9olLQTtpOeo1bmBU0+82i25co/nL fTNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218043; x=1704822843; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=5EuDdHpwOIkkgSSbClAkFkJceQknRHYQiFDZ/42t/cQ=; b=ldNQvYnI4ehMNYX+z1dMBrxDhHjL0O00XjFf8WjayH6ITP63ncoxJI/m4n2eUAxted PazlSgOdwG991G0FFmQkD88UKDxUkiUVi4Tfey2n70RfmOo+EwWilOEGyM3a5QROOWNr 4GcxEvg2rFjkoNEjlSF1Z3rsyvpj2uxK3tDtp8H96hLHt69S6m6wCbo8uN8o4Az22WcP reMtg71UAj7KAGh9ZMwmi7sMfvHvkyUd8QkrlhKuBidEtrljYIHKq3zXIn45yaixD167 QEsO2fC3wWpmniu4lDbBV943QMUcsPVsSHOEEqyH1i3W6Pkp83voEjoaOlaFPvCNRD94 gfvQ== X-Gm-Message-State: AOJu0Yxe4cwkD7tRKsKx5k2woPdTB3gLh9hXR4EofzGvjkOhieWw3xRU 5W3TJKIWK2Ty8fdJMf3fR4CHx+W4kU905aNN X-Google-Smtp-Source: AGHT+IGfDUQj/99XwdeiZZ73lh5kEPQ3dWYF+KkQDfvXFyCLAaB8H9KSj/gNI32pWe4Z84HwpG1X3w== X-Received: by 2002:a17:903:184:b0:1d0:c986:8ac9 with SMTP id z4-20020a170903018400b001d0c9868ac9mr9755498plg.22.1704218042880; Tue, 02 Jan 2024 09:54:02 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.53.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:02 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 3/9] mm/swap: avoid doing extra unlock error checks for direct swapin Date: Wed, 3 Jan 2024 01:53:32 +0800 Message-ID: <20240102175338.62012-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FDBCC0010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4gksndtd67wcdkhpp4ukg9tkwm6xj9hm X-HE-Tag: 1704218044-209268 X-HE-Meta: U2FsdGVkX18uJ7uCpTVwiaVlvbKRzzpNp5OgwH3tTnyAotL5/pAzyH3BEZxVq3ViZlmdDLwLFjP7XTzOiFcx3GsP9xD4AlprxOqujx3L4IhnHmaEzN32Ot2OMYjwvdHA7/SBP1VtKKBTjvqSfv7bGvKNyJAFAaAQjqq3GJxNGHUE22HPkUgFYDy/ayoV9HhkkzoOU1u/+B0hOmUpgaPYmhWLN7gdlv9SCSb9EudYfkBK5RCln1tVUoceMSuiH+ihPlTrME15DmJIhGvt6asJIp/Qc+rZyWK71obvveNPsL9QJzX5TatW8To3H5CG13pw5X7u2re5C7RgRSAIkUkW4xfR9ZR2+KSyz7TYHeJ4L0tfE+zXEVaJnWmWAhqp+HSEVExpBoAKh4MOJaA+Y0GLaHNdX/CNWs/7axkA6qmDU4EE1uOT5MDERF2BS/33NXQ2eBuYvOak/AzZwVW2txHY96lpiBeK4IupO81b4SepC3TTX/wC1CWxFXZPwzDrZy+bDp0dzZpBUjIf7ot2jyPZ/9/H2hzR5eBiCvmm4jui/bTC7/IO7tUIa1l2xj4C8Bf9TgP5DtOaboyJLU7ar3NzIOsV6WRiYIAkcULpRBiFBwIS70THyayYo/vpb87K43Cgssy7KOjWMKkSLUdVXmJN2z5iwS5260/efgaJRo7NQXZ+u/mhc5qUFv/IqTHEKKMRdddAHhyrir8/L3edaCciXGwtbz7aHxb4dE7odFj8Hl/eyIbwrZzVlN0sTsTsPhTAXSmHhw7y7EhXkbUfyQ9LtUISztMR/cvuX1/MD7lMmEyWdP6Ufr922Cjyk6StuCGRf4y3Non/Oa/+QnaWcBSqTD2jWx8quWAwKviqduK65XSobrGgzqXM6Qj1eZDSdniK0U7qmTWc7/5GK6+Re4QKjdlGC9wgD0yPOyf+TDPL4PjX2/eBzrZaqXHVmht2P30JJWuPLLlXQ0TCbSJhh5M W5/Ql2gC Lq6SMX9D8wovT4ToGow0o/DUPP439FR4ydjWNnuTqVXTJOHAvok0wK6nUDfS5QvgjB3nSt+dUf4xLRZAx2RRFOIbgObeG2domO/6U9NIoJbA3KIHrC0cDcnkcmVhRJdG+qkR4aZby2IltsdvhU0DyiBUHZXjCGioUtgpyzmEPjwSjTpxGKNQfqf+xHsd27Oz/6HfCsOTt1ClE3Fx9qgNScEg9460DkcYnMsl8M74997G5PQYc+k6s0NqksCq+0LsIjwKS86L75CNcMaCdUIP4Xh7VdOjla6FlEX12TNWkgzGBXRAQFBENjqi+n1uCPU4WFNnYzKmjIsUvoliFHHlaUxC6ftVv5sPTbgmM6ESkX6RP78MWmv6a+yKhqYD8tX7knwwRECx+jV9A18q1x7HO9xcLFJvaJYyFX8smaDxx6gABldtsddz3IBiGb4Mhf0n9SnC7xJuwwBzzESF7V2g27wnGedXURJdBV0JIMoY2rbHLl0+pN1VlLgAcoEeskmzlksE7SIJXEq0ob2WV4kNeRE3Dva5F+vOv4jzJa+20p43vOSoO0sX0mUtgnINs0oAGpIThymnjjv2+DxeP6TXxp4O1yMKscDVGTZV/NM8vk4LbK5q7QjJHbMVlb0i7Mp5m0BEV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000240, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When swapping in a page, mem_cgroup_swapin_charge_folio is called for new allocated folio, nothing else is referencing the folio so no need to set the lock bit early. This avoided doing extra unlock checks on the error path. Signed-off-by: Kairui Song --- mm/swap_state.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 24cb93ed5081..6130de8d5226 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -881,16 +881,15 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); if (folio) { - __folio_set_locked(folio); - __folio_set_swapbacked(folio); - - if (mem_cgroup_swapin_charge_folio(folio, - vma->vm_mm, GFP_KERNEL, - entry)) { - folio_unlock(folio); + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, + GFP_KERNEL, entry)) { folio_put(folio); return NULL; } + + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + mem_cgroup_swapin_uncharge_swap(entry); shadow = get_shadow_from_swap_cache(entry); From patchwork Tue Jan 2 17:53:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4DB4C47073 for ; Tue, 2 Jan 2024 17:54:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B8DF6B0147; Tue, 2 Jan 2024 12:54:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 468D36B0149; Tue, 2 Jan 2024 12:54:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E4D76B014A; Tue, 2 Jan 2024 12:54:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1BED66B0147 for ; Tue, 2 Jan 2024 12:54:10 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E0002A167E for ; Tue, 2 Jan 2024 17:54:09 +0000 (UTC) X-FDA: 81635119818.04.E80265B Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf02.hostedemail.com (Postfix) with ESMTP id 0568E8001A for ; Tue, 2 Jan 2024 17:54:07 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hqcqjNaU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218048; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HKSeSe6aIz/giM2hLWohMCNrft/gPTDUxVx5SpP3Eg8=; b=dBmBpDT18TOigpxqqf+LHYlEqkCfKWqU5mIwErBuhN1ptinB62anMBpQDT+lM5KB9Gu01G 3/0U6HFsLF+fv8oeCPEP1UVvapFHpVEo/OWVdK/m6JBk3dpsuvPfykVmxVbUEEDiU31U+K OPVbPbN/dUBq3j1esEvRyEogXbiSORc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hqcqjNaU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218048; a=rsa-sha256; cv=none; b=UR9+DEyxzvrNILdrRlsZL87G0vWyLuvmbJNxfWZ/mrX/x1dc8WxB216ZlsAZbFHEjkJfwq AmSri0pLAXBP3++ty3jllcShNIkecEVuBC6DgpIjr44uYjZoyZ2Wdtxq8kz98hqfXft0qn aobeUjI4NC8irAImp7Gt+y8yyCzfLBE= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d3e84fded7so46826675ad.1 for ; Tue, 02 Jan 2024 09:54:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218046; x=1704822846; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=HKSeSe6aIz/giM2hLWohMCNrft/gPTDUxVx5SpP3Eg8=; b=hqcqjNaUTTrjGYv7ym3xuEooVS18Cirv6hXInfviuC5pSGfje8/BRmXP9DHIxL2f81 blONYiJYRGBtjHpTYRfH+HJbopkPo0UU7zaZg2zfUNjBAE8O+wpVpya7kNarz7CnqWjC aiBXKeICYmBlauybu9S7QjxH7tYzrCoe8VEyCT893sLc1QBoP+/3tRZvtpZCJM2mjLyH pGKN+QP7gyxxRUqZ3+v4HKsmUNZYMcYaqyHOeCd7IJMHfilX/RCc+UZjbTFtUOjdSILM FBLGp7t2UYtPBWB49hj4TWbvyU++FhwBJHmwZshZE8g83FHVvm3jLaiAZvcKwSKUlHkA PchA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218046; x=1704822846; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=HKSeSe6aIz/giM2hLWohMCNrft/gPTDUxVx5SpP3Eg8=; b=eZ1laakK9L2nn9/HD1iS74S9abMBCzZ3wOFosP+pXp7lcmxDHBd1vFxoJAKAQnhHJD +wE/KT2/tOwRfY97BvBLfR7l5Ze4MCayYT1ixRhEuH9DLgSHvu8VK5ff/FUK81DZOpB1 ++TK3V3sn4nvT43n/wuEwaXJrMN3lshJNeZZSTqHwKdNcpHgwhsoPSQJyrn/XdSt16Oq Xw4TEAdGLJK4zS8OXZigtOJfG1NmYiqlnMF4KwViLFz8OQSgs3Z33UrI2HczHFVOjw8e EYl8B8EYPYhY4gantnbfvryYVbJd7uoYYs4JWkL/xKpiHrvxnNZuYsnG7iEatrSmdyXS YYtg== X-Gm-Message-State: AOJu0YxNo3E9oKi1JNAa3WM8LSaCWts1CqKSOjppY0n8tJ+nck1NaK/q vTQ13FiRETSdZLsCeGRjqM+kvcpXO13MQ6dP X-Google-Smtp-Source: AGHT+IGjD60IJ5O934G0uk9U2RzbZjBO7ugPyT2rW4oxx9bLV6dvHfHyD2MlY2xhFWamcsU7Iv7RfQ== X-Received: by 2002:a17:903:1251:b0:1d3:529a:83b6 with SMTP id u17-20020a170903125100b001d3529a83b6mr8433403plh.17.1704218046394; Tue, 02 Jan 2024 09:54:06 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:05 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 4/9] mm/swap: always account swapped in page into current memcg Date: Wed, 3 Jan 2024 01:53:33 +0800 Message-ID: <20240102175338.62012-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 0568E8001A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: gwxbtkj71yw487iepxc1otrswiag3y5k X-HE-Tag: 1704218047-546649 X-HE-Meta: U2FsdGVkX18wjCq6hi4f1tRI6ONCnUy75vItiZsNwqgdbzyTw0I41v9nyf/xtrNt4MlrPOp9qNh1fxRpQyrAuF7FCrm7RhKuB+EL1wZN6pA48U9EpjZ4ZRInrnwYXxYX6JwLfOMmj3Lt/1p+QRBylj+sFTIjyqYJrqQp2mJS4h22H/QB8h1Wcv87QOQeG3CmSVv1G4sauizRCZU4nH9TTXNDUMVpJ6emjGbL32i+ZVUUzipajDlmjyjrQPrJWhMKdh0LTXWf10jmoC4Q5yqAGpNSz6OFZdSlGkRCNT2SXmJZeAlSHWts8LT2HeA44vDFQ8h2xprVDawse1wAvuTHvmBs6/tNeZHGHIbcr3a0xTlKPvvi2742/7r5Ref6/xD3M3DK6Nn4dO1zvVagL/bhqzjYZ2IaomaNYjFzPWOy62LFNLgznHrODUVdEZykSPjQWGWA78trUG7iBNokZVoEj7NB10ZINhgdJf8OAMdDNt8+2aO6hWrZ2KhDTkdXq8e6gJ6m2ZdIGzB6F6Q8JxaHY2AP1By+WaIh5FZhCioFrvsEeqJ79D8msxQAePboQ9DLUzz0SWtNVHPtzzRdZRBFRPXdukRoeOX22135q7m6U9rHsrvVBRRABHdyRgqEz5i+3ECgkcYSA9HuWdFMf88Apj5CZI0+Z2Q6a4egm+7Gsru57sgslW16KeVzYn8wVl1NsIOw66DZ39fNT31+NZcoh8jHXS1Y+pX9U3u/Jrl9uaIZYXjwfjOrRAVMVSfEjawqKVziF4ciReX3UmTQeHgWGlp76tT04BHDl5ALCQNxUiKQA0nUIUznAxwmoNdOi27W7Noym+JUT1ocKiT3CxEGGt6PpVQ9YHO0nAO2GX0hQ8R+0Jdhh7KF6zM/GdHlzZSrPkb5STvPy/Y4rEPMZA2hSxvUG0/amIjDR4Cb+YPjuuOfA5XptTZEwME/Ws4oX0E3FJDFEkWE2ygcsXPnPPW rmsojlu/ hnX3MSTMQkX6+8DbkP49POjVv1JyTM34MEoTvNKnlcn0YpM4QWQe2KeIsByza4J5IB8EPj+c/yxHxEi/GIwNZ9mDFRNaj2FgXXuBzF0ShBIMr1mi1O0vSk1ilcXLkGUB5E+UtFKed9FA32K4v6dPIZR9Qtye0qKOPG9/vWHDWLmoPPPguxTAZIIgwP1VTNNVkIUDGHyQinIUEDD/R21qqFAuW47WQqRiTkQxpIM9rrDOboakHw5AbBzyWf7ufy02wSxYZarZG2IBV0RBhpV927MXdIW+XIKuFHLXkfXCNF4NGCnzi3g2AY2rdyADJQNF7L1UDDOP99KmerT3QKSkVLJAenOcLhEVn+64plodfrmFhC3v5TOESobqy67WtuAqPJsmo21++O1E+tuq4aszrNjKkF8KwhN0HNlCPvK+3UqtSopu0OunLfDAkmTmhu5AxcT189L1WHSHkDx7bo2nTcSA0ei7Dx1QfSEsDar5slxyvcWLta9YYoOKfvQJvUlG7n4iXQ4MYoXQ32kKB03iKxsODTth0zc/clorEtYflCKFHULQJnMiecuACaP39pDvApqwVNcT4Fz/QF8/B5Ln791SD8lvlPzXgBz3MTqwk0Dsfc8PcbUFZGnKlI5DTU7JjmMc2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, mem_cgroup_swapin_charge_folio is always called with mm argument as NULL, except in swapin_direct. swapin_direct is used when swapin should skip readahead and swapcache (SWP_SYNCHRONOUS_IO). Other caller paths of mem_cgroup_swapin_charge_folio are for swapin that should not skip readahead and cache. This could cause swapin charging to behave differently depending on swap device. This currently didn't happen because the only call path of swapin_direct is the direct anon page fault path, where mm equals to current->mm, but will no longer be true if swapin_direct is shared and have other callers (eg, swapoff). So make swapin_direct also passes NULL for mm, no feature change. Signed-off-by: Kairui Song --- mm/swap_state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 6130de8d5226..d39c5369da21 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -881,7 +881,7 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); if (folio) { - if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, + if (mem_cgroup_swapin_charge_folio(folio, NULL, GFP_KERNEL, entry)) { folio_put(folio); return NULL; From patchwork Tue Jan 2 17:53:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 665B8C47073 for ; Tue, 2 Jan 2024 17:54:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F41CE6B0149; Tue, 2 Jan 2024 12:54:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EFA306B014A; Tue, 2 Jan 2024 12:54:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1C2D6B014B; Tue, 2 Jan 2024 12:54:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BBFDB6B0149 for ; Tue, 2 Jan 2024 12:54:13 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 911AC1A07CE for ; Tue, 2 Jan 2024 17:54:13 +0000 (UTC) X-FDA: 81635119986.30.43221DA Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf05.hostedemail.com (Postfix) with ESMTP id B4459100005 for ; Tue, 2 Jan 2024 17:54:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gKwvfrOv; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218051; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nlZ0fg6NatnLjyD4rpGx2P4cih5u8+SQurmVS5ry9aQ=; b=WWP+Lre2PiCZiSyMkJfNav4ZES+vvzQhGHNs7FqZ47AXiyXu1SFIAte+J60slB0CTioEab iqDOKZ2Q1/m7H+PSXVwmu5RwDGhOjbyFlbdILDpPNZA27brsJS3+OhB5eTFpoCa0nDY3Om jFfvZhs20T+gj/JKLoZ3kJlB/laFN5o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218051; a=rsa-sha256; cv=none; b=S05Qmk4FKrbuTsz/LDSqd1RgsXAKYD5v8aa4Y4LxTfyJo+gsG7qFKkCOIH8jdMI83/wp1i gV7VOq/W8a96R6d8driHNte80bHG98IhWcvzmLKicwBfjunCB3FQGS6oKu/UyLdFlnLdkn 9CiKPD/MrKb/3Mj5C7gj3+9r8NIsiVc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gKwvfrOv; spf=pass (imf05.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d3aa0321b5so76952855ad.2 for ; Tue, 02 Jan 2024 09:54:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218050; x=1704822850; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=nlZ0fg6NatnLjyD4rpGx2P4cih5u8+SQurmVS5ry9aQ=; b=gKwvfrOvIB/5zisa1TuPIqLNz0Pms5BbtXv+1zjbyv8mWJsbE5LVvB3D/1trXVIs/p Zb2V4rWfOilX302Wa++XZiTp+kYsnnx8IqAx48sImfn4qNAqr99z+Ehmp3fdufPTFtQ4 6plsuu0wOpzCWlKmVR7RWC+AvO0sULdJ6f9CUt+ZlvIZrO8bXHos39TNYz6EKjAUsFlu +DUWjY6iF0SJJKY7uoFVsxM3UNXwNWjx6MSy52HJeFfozqy5sLXFy3mR6vch1R8BOA+g J6cCLx+Fmj2e15Kew5H6PetWC6S39BMHKT5fxkxKa5LkfO8+helpG3xmuuJV26uMGNqC pvlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218050; x=1704822850; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=nlZ0fg6NatnLjyD4rpGx2P4cih5u8+SQurmVS5ry9aQ=; b=ms2YUfl99GcjNm1vpnPSMQalDWPFKPX1hbYbh//6qeEFJQcHiWegGB2zZI/7kZn1qh Pd/xgTfEQxJeg5SRABLGEMqQuxfpToLF0YDo16psxko2PRGagcT+sUqz/9aqpOHICLO+ JYjIcNQvKFkvn5MLLkahjdnnVTTNigD/QO5k6mEJilZlcxOlyxxTjXkrP94hJfXdAaeE tNFTF8H+G1nURAFxZ7e0ftTlbIFGlk8iZobXipkZhYZvGRH1i92g716dcD5ONWhxOvgO qNabjGSK+rQxBEwiPKM0in/G5BtlSdqCs8TBvpJIhehPzk9DwUlQM7HK41dMhtos2j85 8Hmg== X-Gm-Message-State: AOJu0Yxi3tCkRzDHU2/weW+v85Hpt88LTCg3H08hp12uuBkAaxqD/Oyi SgorRyme524ySgsx7XyTY8LCgJx6I9ckO+x8 X-Google-Smtp-Source: AGHT+IHg3EaXPsfJ3BsQyYM+9C4NVV8c46gR3f+IfIC1eTcpnGbk9EIEm5Ji/30WqG1cQYmyBx9asQ== X-Received: by 2002:a17:903:41d1:b0:1d4:3b72:b294 with SMTP id u17-20020a17090341d100b001d43b72b294mr20703980ple.88.1704218049886; Tue, 02 Jan 2024 09:54:09 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:09 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 5/9] mm/swap: introduce swapin_entry for unified readahead policy Date: Wed, 3 Jan 2024 01:53:34 +0800 Message-ID: <20240102175338.62012-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: B4459100005 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: u1up63owzr1snup5ugaotiik6qu3fbnz X-HE-Tag: 1704218051-779397 X-HE-Meta: U2FsdGVkX1878dycg5sIRuLEwITMVhLNRFtZAKAFWe3S5CnDpvBbXPXOzOKeEp8shYlbBP1L6X/KsQv5/rPMXCCrp07HEQq+iBfIduyWC7vekYfn0XZVkjrV21WOt3dZH4+XIBc7Ec8p/jN/YLOKZluaIJmAxKZJnscDGRfHUA7LuFlqhWmTGbiE0jLlNY4J3Ovbv5sQvnoCkO3c9Y8P2xDnGeqWMEtoNUY5eEMYnng8Z4jxUvX66YPWZaG+MAAKRSZG6pmWMfyy7K7QZVf6EMXrbbFe8tHS9EyBNLneSdRFlcWpRK+CWVPG9B51Uw6HKCnHOap+8+KaP8BqWrk04wP5koPn7Hm2j6XGDPffcR8B3WdyRGQKJy+UtGE676Lkcj9uRlxH0y6SMQqgCkejj0bKWde5dZlJKnGKC+WMCUE8Bg/OoFL29DW3t6yFhKhNrZNZlvsXYfbM1Gj7tQk10sOwExQE8GEl0gXhA1zvKF359xf7FtY3UgOUQErKKQNndNDGWw1213fKjla8iovxIU55K/MPIvnzvZY2X29KwHaEjdMR6BrlHtdGK3WoHKfUIfXV86mZkYjcYw5ime9I/hDVrliLvNCpuvg/aaHsqlVF46+D5B9Q6lHZ7FbjZXqWz/9jlsgM+0zDh72uF+RaLT1YH2g+iu4oi7Ic9ams6W3DEHFtqyXYMXYqEoArGsFQwL6I/oQrWK3Iz+Mao9sAlA0FGkgSqVjpovecOXvMq8PU0aOY/+/lG3mVWNGHi77JyCOLbOf6OwiVroTIKz3vzN83Lhtdxw+FI8cZZ2Vd0Q7+JnjVjfpGO0OVZYnpYflC7Td76/Y5xHTcmIAvc1wwjaDPiNCa4WwOFuOWrJm/9BcyPDpOFqj86rgokWbdB3DaV9uz74t54Jz+P6tFEuSF5qTtmy+Efo+PfRi6w5TmBtIGHOybVrXCGsMxQugt2HdUAIjmiM2vfGyCigpcAnk Ju2csSU6 7NQIGuQdxumHEUN4XNux06nepey68aNa1m6mi+of3xT+j5zruefeo5/TZYQFgW4Eai29JDK8Pdt0ve7NKSY7Zb4Lbw5W5WCjpBmFr+gmferExc5gOA3W9hw2hl2TvtoY62FTp1C4dm3pqrOWXajhLtyhihYzLKLwegCBJPrgokthn/+g9vZgRqH11GZLtN3IvECEdBRf6Ugu6II1McixxMGpUtGC3DXAJgdGuWzp2TSzAK5XTo5c/37bP+daI7MeY+K6DacKfZnL4MPChy7E5KTrr3lwyfpyPfz96JnzmpUjsx9ZDLwRjcKgZGB/1fTkB1rn144+q2Q2AUzHdzrQ5mKDiC0PkuGENAl8nAIQhNNSp+Nen3LkeuO0rf9HPilkaD0NK/WwR2TXgPeJjBD2CzR2wnbN+W5b0gK+cwoX/ur2gbuo5lHigFq9CGZt+PjYSiUbMXhbeRKypoyXTZyzYAKcpFo3Y72vclBz46vFyVeWaLGa/eqi7PENSkNWa6l8H6YPePA3L3ptaD4V9Cwzyz2vOhUwqHm1MfnBfrEszyR5g4LRPtctTqNgB8wM4pnKOCHoG5CFee4EF7gYrkHql9byFFPjE0DCr7ZFbViL4mFlbsVaqyCJ7QEDVuyRMDNUmljQveL3cvzWuRv+8Oxrf70HaXHNz1RUdHksgqXOcobTavqw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Introduce swapin_entry which merges swapin_readahead and swapin_direct making it the main entry for swapin pages, and use a unified swapin policy. This commit makes swapoff make use of this new helper and now swapping off a 10G ZRAM (lzo-rle) is faster since readahead is skipped. Before: time swapoff /dev/zram0 real 0m12.337s user 0m0.001s sys 0m12.329s After: time swapoff /dev/zram0 real 0m9.728s user 0m0.001s sys 0m9.719s Signed-off-by: Kairui Song --- mm/memory.c | 21 +++++++-------------- mm/swap.h | 16 ++++------------ mm/swap_state.c | 49 +++++++++++++++++++++++++++++++++---------------- mm/swapfile.c | 7 ++----- 4 files changed, 46 insertions(+), 47 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0165c8cad489..b56254a875f8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3801,6 +3801,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; swp_entry_t entry; + bool swapcached; pte_t pte; vm_fault_t ret = 0; @@ -3864,21 +3865,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swapcache = folio; if (!folio) { - if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && - __swap_count(entry) == 1) { - /* skip swapcache and readahead */ - folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); - if (folio) - page = &folio->page; + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + vmf, &swapcached); + if (folio) { + page = folio_file_page(folio, swp_offset(entry)); + if (swapcached) + swapcache = folio; } else { - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - vmf); - if (page) - folio = page_folio(page); - swapcache = folio; - } - - if (!folio) { /* * Back out if somebody else faulted in this pte * while we released the pte lock. diff --git a/mm/swap.h b/mm/swap.h index 83eab7b67e77..502a2801f817 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -54,10 +54,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); -struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); -struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); +struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf, bool *swapcached); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -88,14 +86,8 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, return NULL; } -struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf) -{ - return NULL; -} - -static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, - struct vm_fault *vmf) +static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, + struct vm_fault *vmf, bool *swapcached) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index d39c5369da21..66ff187aa5d3 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -316,6 +316,11 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) release_pages(pages, nr); } +static inline bool swap_use_no_readahead(struct swap_info_struct *si, swp_entry_t entry) +{ + return data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1; +} + static inline bool swap_use_vma_readahead(void) { return READ_ONCE(enable_vma_readahead) && !atomic_read(&nr_rotate_swap); @@ -870,8 +875,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * Returns the struct folio for entry and addr after the swap entry is read * in. */ -struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) +static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct folio *folio; @@ -908,33 +913,45 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, } /** - * swapin_readahead - swap in pages in hope we need them soon + * swapin_entry - swap in a page from swap entry * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information + * @swapcached: pointer to a bool used as indicator if the + * page is swapped in through swapcache. * * Returns the struct page for entry and addr, after queueing swapin. * - * It's a main entry function for swap readahead. By the configuration, + * It's a main entry function for swap in. By the configuration, * it will read ahead blocks by cluster-based(ie, physical disk based) - * or vma-based(ie, virtual address based on faulty address) readahead. + * or vma-based(ie, virtual address based on faulty address) readahead, + * or skip the readahead (ie, ramdisk based swap device). */ -struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) +struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf, bool *swapcached) { struct mempolicy *mpol; - pgoff_t ilx; struct folio *folio; + pgoff_t ilx; + bool cached; - mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); - folio = swap_use_vma_readahead() ? - swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf) : - swap_cluster_readahead(entry, gfp_mask, mpol, ilx); - mpol_cond_put(mpol); + if (swap_use_no_readahead(swp_swap_info(entry), entry)) { + folio = swapin_direct(entry, gfp_mask, vmf); + cached = false; + } else { + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); + if (swap_use_vma_readahead()) + folio = swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf); + else + folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + mpol_cond_put(mpol); + cached = true; + } - if (!folio) - return NULL; - return folio_file_page(folio, swp_offset(entry)); + if (swapcached) + *swapcached = cached; + + return folio; } #ifdef CONFIG_SYSFS diff --git a/mm/swapfile.c b/mm/swapfile.c index f7271504aa0a..ce4e6c10dce7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1866,7 +1866,6 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio = swap_cache_get_folio(entry, vma, addr); if (!folio) { - struct page *page; struct vm_fault vmf = { .vma = vma, .address = addr, @@ -1874,10 +1873,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, .pmd = pmd, }; - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - &vmf); - if (page) - folio = page_folio(page); + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + &vmf, NULL); } if (!folio) { /* From patchwork Tue Jan 2 17:53:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29DECC46CD2 for ; Tue, 2 Jan 2024 17:54:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B489F6B0151; Tue, 2 Jan 2024 12:54:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD0A16B0152; Tue, 2 Jan 2024 12:54:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 999176B0153; Tue, 2 Jan 2024 12:54:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 82CDB6B0151 for ; Tue, 2 Jan 2024 12:54:17 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 65B75A085F for ; Tue, 2 Jan 2024 17:54:17 +0000 (UTC) X-FDA: 81635120154.01.18F7D0D Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 96A0AC0008 for ; Tue, 2 Jan 2024 17:54:15 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eLDjO8BG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218055; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WQ0rvcDFd1GF49xQGBwNPnCf4JL45lf63n9SBHxFbwk=; b=M6zIkgCMxkWWhd4LpQ5ScTsgktJ8VuaPe8qEbJ8TRiRKNmeWDUoLi3lqbcOPwOxg1YC49/ nKTieK6tXXzZyRLtP8WXh+LZ/qht1wV4735xojFOA/Y6TO2iAWOCX6aYS0VE6VgHVVyIb0 2gKuIe9VclS6Al+aO2BAE73Rfc6MiX4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eLDjO8BG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218055; a=rsa-sha256; cv=none; b=8ZLN4iX6sBze+wJ9HaA/6zIjI9fdYatTQ5mbVrH79f6rngJnWZ2ri+eKSDoPxhvNN9HHQw NG74DZoaSEtldfR4i/KuE+InTcwCIQ+lNMfnh75OCwC9mQtPi2l6d/ubYFdgO6rPRLzod2 ekRQbNfEGSTWPkydoA+UpS3547aN1TU= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1d3f3ee00a2so33824755ad.3 for ; Tue, 02 Jan 2024 09:54:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218053; x=1704822853; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=WQ0rvcDFd1GF49xQGBwNPnCf4JL45lf63n9SBHxFbwk=; b=eLDjO8BGK2wP9aETufJqbqm9V2S4GnUurHCmtdI5G41RJS6XLtnm8lMSiTL4w6Q4dk BPETMPnYvJ3Vyq+6hoXwfXsOarT4V2uB9oO0eMXnJWbiQoAA8xYVCXcCeHp9hPpp1rd1 dGG7nSk55jJeIFyhohFzrLU1wKvCDmGdNlMfqzWz12pDXd3e+r5sgiXuS61NjLLhaDep UxmtvIjQa2k9JJAXOlB35u2Muyjf7ugpirohSJ+JZqcEyRaxENUWji42uGbyVJJB/dal vQHOcxrVVxOKfAIs5vuGRsmnj4i1mRsWW2DUmAghxhMPoUvzF1kFeT2zUkD5gsRNzBMv aveQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218053; x=1704822853; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=WQ0rvcDFd1GF49xQGBwNPnCf4JL45lf63n9SBHxFbwk=; b=NKQNBt+uGtLZr3CgjqP4sbcQdKp4VrfCimbg7oJFf+mkCdVE4Pz9OVcXiXw2bHKE18 8BzA/24Ubobd0v8gbSoC3wP0IevIZaOptrOZW1fRK9Z20jQErFGwKPdRchlkD5ssCO0L 8aqE53orw7+InN2TjwGkXgqPE4FUKiCxTQwf69ps+uILJ3Nt+f2LbgvX1Qa5m/ESK6cV foMmLZL2TBKCAiPs/9s3q8L4oo/ReEOcm+xa+p6xSsx+1jKntQPf7lNvs0NlnhXHvMPz KPQumt/453ZH/FEH5CY7XDHygJCmNt2vNGV44H4a/uoYKuRSrjd73tH3tGMoZU5ZBA+G nBvA== X-Gm-Message-State: AOJu0Yy46aySSYbUKiq+Qu9byX2vtfwXL3Jruk1Frg06wfffwlBeWg1/ JnMAGCpNPc0tnpaJssh8tCzcYdGDnRcFkA2M X-Google-Smtp-Source: AGHT+IEqRxzrscJ5aDzXjoCzOKW/mGxYYasxFtxeT189T8sXpnySEuTBNSzOKUPECfaBTd/IXyLMhw== X-Received: by 2002:a17:903:2352:b0:1d4:c1b3:c6c with SMTP id c18-20020a170903235200b001d4c1b30c6cmr1260292plh.51.1704218053424; Tue, 02 Jan 2024 09:54:13 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:12 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 6/9] mm/swap: handle swapcache lookup in swapin_entry Date: Wed, 3 Jan 2024 01:53:35 +0800 Message-ID: <20240102175338.62012-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 96A0AC0008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: rntuf6y9r3kbn4tktm7sthdjuehj7m54 X-HE-Tag: 1704218055-625523 X-HE-Meta: U2FsdGVkX19AksxKBOcHdtQMV1UwoO023joaHgUQHD8mSMWZGXWaIr+2V5Nrq7whkb/pOcgdl51YXwF0pqJUb7Ex+ETK+EYipm7DVytzf2U5IWmYrWN+U1i6qQuZYpRqHaBEls8DAYolYML+7XHRpDPuMUyW5HyIZQqXtMe/STZ6YyTGA2rK9voxc1zjaqWXl7c2gBWYHbNH4QGBRZ5zz55fPpzUnFvcRoogFKAiYXS1SXVUk3ZqlvMgHnqtIkE50Sfy4c9SFB5Gbp3RXQtH7DvKanEGpUGdC37jH6Lwij0RzhKEk6ce8CbybwtEm06/7B25idSk89O7nwe/3Ea+2C+pbI/F2Na3FpeAjrKeLDfyXli9pmZOa2Sup2rC3/YH2SzZOFkr9QJe8H4ejsZYuZouQAPt6PDSBAshI8u0LTB+xKRAR5oUyxNU+pNCWlMPNSgKx0UA18UqfWA9xjv7tb0ZPl6vkkGfkr12P+ZZ4RgvmvHfQG0FyyyCADFoTimkMO8X9CrkSWkTT2GbjtTWtGfDa2NY/t6Gg6ONIjvHxLS90USm6uzKHHGUQe445sb7gSO9f5U31cCVS3OK9Z9K/hWyhxfzdi+AtvPLd3IO684YPZeThP2TDZKA637zk9OhRUMLHQjJsAMJ/9SbPNYNGz4rfbag8glWZuoITc1rYsUDS19vxnUsle39vp67YVYzo4+egxr5jYDhi2dDX9KqWp6zIwstLhnzEw2FE2SHRXurxecanDprFKGYcSv1/Se7yYjJYKRwGnafiNJHy1Rn01cYRO3ozivNMsY+VB3ekjkQIrIs+8G9wZYonz3fUpEgPWuzt9XpPKXAXUfJ7dtpVZ4SLwhjlZ3AiOTlACjuOMKx52N811VeC2t9FoOo0g/JDTB/3y1TNCtMHZ6Y7+2iG+fnJD30d8gUlnH9MkYDr/PY7/gkQqppUJnY3QB0JI0dobPkB6TlCJ3tHvn5ZEr W+C+9GoG KsNViIClETq9wfXdazVwPeONOECztQKZq7ubR9S6oSBoxkzhbPAFaJiPxBTSMEgy5GFIxFXQ8ECBeyLGqQdofm67FLXLpw0nolCRyDFO+bveRAIJWwD5MAzcyw20sbv6DGHsQ4kwOK0SL9Dvvz/t5d8tU47q9GGLgf6JKUZv2lLAEsfNS6tQEo6vS4xRdlCXkgltVVncUi9wnVoq/K9yXI+qG4fHcq3pdgYoMzOLBNv7Q+zQTCh3MRy0G4/ZAwVf+tVllAnhUDMHw4wIeRNVBXQmu5CuNzhNi2zyKh93XnIFxXZ1HTsqPhjoY7gjtthnevGoD/c2tCE8yYGI98eK79Q2YgnzGZwZyAabGwv9aEwpNeUweabxHneg+xC4p1OUR7bVqqYqLi/sZ5JCvjB1WC0aSbQ9nGAQZ+7NZ8SjV/CrTWa/f14P/2igxB3ijHdhTjh4YRHvaqpBA3ZSKszkMPE0w2VUJBY1WJda4ShdS+rrdofgbCnaIa4tF0jFU/GQEAnyTwFDUSHWSSDFX2vjbZwAYuMIkA++/U5575Q9DsS6WcsqzzLIVX82ZigAmPv+x35jv0XA1kXKrLKT0WUDVesZ1Q+/PsW318Mw/1/+7SswSO2IqyT/dnl0pv5tjWam51OAu33nWo7MGfs8Z5q1n9FUb1b/4WYarjbUfWOfCWQB62D4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Since all callers of swapin_entry need to check the swap cache first, we can merge this common routine into swapin_entry, so it can be shared and optimized later. Also introduce a enum to better represent possible swap cache usage, and add some comments about it, make the usage of swap cache easier to understand. Signed-off-by: Kairui Song --- mm/memory.c | 45 ++++++++++++++++++++------------------------- mm/swap.h | 20 ++++++++++++++++++-- mm/swap_state.c | 22 ++++++++++++++-------- mm/swapfile.c | 21 +++++++++------------ 4 files changed, 61 insertions(+), 47 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index b56254a875f8..ab6e76c95632 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3795,13 +3795,13 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct folio *swapcache, *folio = NULL; + struct folio *swapcache = NULL, *folio; + enum swap_cache_result cache_result; struct page *page; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; swp_entry_t entry; - bool swapcached; pte_t pte; vm_fault_t ret = 0; @@ -3859,31 +3859,26 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!si)) goto out; - folio = swap_cache_get_folio(entry, vma, vmf->address); - if (folio) + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + vmf, &cache_result); + if (folio) { page = folio_file_page(folio, swp_offset(entry)); - swapcache = folio; - - if (!folio) { - folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - vmf, &swapcached); - if (folio) { - page = folio_file_page(folio, swp_offset(entry)); - if (swapcached) - swapcache = folio; - } else { - /* - * Back out if somebody else faulted in this pte - * while we released the pte lock. - */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); - if (likely(vmf->pte && - pte_same(ptep_get(vmf->pte), vmf->orig_pte))) - ret = VM_FAULT_OOM; - goto unlock; - } + if (cache_result != SWAP_CACHE_BYPASS) + swapcache = folio; + } else { + /* + * Back out if somebody else faulted in this pte + * while we released the pte lock. + */ + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (likely(vmf->pte && + pte_same(ptep_get(vmf->pte), vmf->orig_pte))) + ret = VM_FAULT_OOM; + goto unlock; + } + if (cache_result != SWAP_CACHE_HIT) { /* Had to read the page from swap area: Major fault */ ret = VM_FAULT_MAJOR; count_vm_event(PGMAJFAULT); diff --git a/mm/swap.h b/mm/swap.h index 502a2801f817..1f4cdb324bf0 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -4,6 +4,22 @@ struct mempolicy; +/* + * Caller of swapin_entry may need to know the cache lookup result: + * + * SWAP_CACHE_HIT: cache hit, cached folio is retured. + * SWAP_CACHE_MISS: cache miss, folio is allocated, read from swap device + * and adde to swap cache, but still may return a cached + * folio if raced (check __read_swap_cache_async). + * SWAP_CACHE_BYPASS: cache miss, folio is new allocated and read + * from swap device bypassing the cache. + */ +enum swap_cache_result { + SWAP_CACHE_HIT, + SWAP_CACHE_MISS, + SWAP_CACHE_BYPASS, +}; + #ifdef CONFIG_SWAP #include /* for bio_end_io_t */ @@ -55,7 +71,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf, bool *swapcached); + struct vm_fault *vmf, enum swap_cache_result *result); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -87,7 +103,7 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, } static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, - struct vm_fault *vmf, bool *swapcached) + struct vm_fault *vmf, enum swap_cache_result *result) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index 66ff187aa5d3..f6f1e6f5d782 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -917,8 +917,7 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information - * @swapcached: pointer to a bool used as indicator if the - * page is swapped in through swapcache. + * @result: a return value to indicate swap cache usage. * * Returns the struct page for entry and addr, after queueing swapin. * @@ -928,16 +927,22 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, * or skip the readahead (ie, ramdisk based swap device). */ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf, bool *swapcached) + struct vm_fault *vmf, enum swap_cache_result *result) { + enum swap_cache_result cache_result; struct mempolicy *mpol; struct folio *folio; pgoff_t ilx; - bool cached; + + folio = swap_cache_get_folio(entry, vmf->vma, vmf->address); + if (folio) { + cache_result = SWAP_CACHE_HIT; + goto done; + } if (swap_use_no_readahead(swp_swap_info(entry), entry)) { folio = swapin_direct(entry, gfp_mask, vmf); - cached = false; + cache_result = SWAP_CACHE_BYPASS; } else { mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_vma_readahead()) @@ -945,11 +950,12 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, else folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); mpol_cond_put(mpol); - cached = true; + cache_result = SWAP_CACHE_MISS; } - if (swapcached) - *swapcached = cached; +done: + if (result) + *result = cache_result; return folio; } diff --git a/mm/swapfile.c b/mm/swapfile.c index ce4e6c10dce7..5aa44de11edc 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1845,6 +1845,13 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, int ret; pte_t ptent; + struct vm_fault vmf = { + .vma = vma, + .address = addr, + .real_address = addr, + .pmd = pmd, + }; + if (!pte++) { pte = pte_offset_map(pmd, addr); if (!pte) @@ -1864,18 +1871,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte); pte = NULL; - folio = swap_cache_get_folio(entry, vma, addr); - if (!folio) { - struct vm_fault vmf = { - .vma = vma, - .address = addr, - .real_address = addr, - .pmd = pmd, - }; - - folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - &vmf, NULL); - } + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + &vmf, NULL); if (!folio) { /* * The entry could have been freed, and will not From patchwork Tue Jan 2 17:53:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54361C47074 for ; Tue, 2 Jan 2024 17:54:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D942C6B0152; Tue, 2 Jan 2024 12:54:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D427D6B0153; Tue, 2 Jan 2024 12:54:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBC146B0155; Tue, 2 Jan 2024 12:54:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A7C376B0152 for ; Tue, 2 Jan 2024 12:54:20 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 85776A18C6 for ; Tue, 2 Jan 2024 17:54:20 +0000 (UTC) X-FDA: 81635120280.01.766EC53 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf04.hostedemail.com (Postfix) with ESMTP id 9CB0540021 for ; Tue, 2 Jan 2024 17:54:18 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OuRIbcsN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218058; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wKrEvGkZ8GQyBX+qIUqWP+XNbS2tofN+iPdO+E071Zk=; b=LVlOvDi3DqG2U3/fPDEbuCIuwApcanyU42jrbGWePWKix1BD1hRZIfBq/lZIvNVe4xt60c TT8pFuxuc+plwzQGkSmMsVmng+2ZDV57/14/yHuNpi2HIcsCpBvWXgfHRqp7eS6e41cQkv dYm0bA9CRvT2E/CuYW48RIINGlv6w7I= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OuRIbcsN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218058; a=rsa-sha256; cv=none; b=6RgOMl13l77FM2714EpYHLKyDvyhOERwZZG9IEcWhunEPTvHL1Dxtnvp6fAFw2iX24gRG1 k+TjQRmhcM45zUHpmk5d2EQXBLer7u6t5l8jR7WBVBOEYeLTfecpwcUjiyTlxjTHWfCM8z Otv98nqqpdSWBj2SBbhQfdTTbLvzrPw= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1d2e6e14865so44973735ad.0 for ; Tue, 02 Jan 2024 09:54:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218057; x=1704822857; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=wKrEvGkZ8GQyBX+qIUqWP+XNbS2tofN+iPdO+E071Zk=; b=OuRIbcsN0+uyt6Zi+T28b6jyAzTU9qrBLo8OchfCNpsNE95WWWOOfG/crm0y2LONBP 6uSWHnuwFIQzNMTBVv0Ml+BYRM8+3NwoTAqdhl0RT4Zk4I8X6+zenL25tR2M4XeKgy3s ecQ7pyuiw0zcnJ8D21bVijMT9PpzPfESjRGLSbhIpKG9Vkwa5UKU7E/417WjPXWYHjwE +xPSQs/Sq7G/CiO6HGpRZDPX/kwLZ8/y8GRvNDAT/JtIxoKgmNV2Ccg8xmXAqdP4yVqn hIMWkAonV02RgXtEeEqf9imY9tNHOMsUYUk42u9z7BOe30J9WEQj3cBEp/73FFTJBjJl gJxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218057; x=1704822857; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=wKrEvGkZ8GQyBX+qIUqWP+XNbS2tofN+iPdO+E071Zk=; b=d5peSDLxEKJyVa82bQv4FXuagH0V1q0D1A5IIL9Nt0ZmPf+B7vKS/lyBTxV7EUD6f3 UpubyObVCh52cklMvvEHWW9I93zLjXJLlyMo0Mfg+Wciz3wq5+roIAfvFtixAxIc+EeD i12HzKQSHqkzWyRjdPEG/Fdhxw76YHWtS/2CQJnDZR3oFC96kdGh5TKiCBH2dyitmmee Y9dCvKdRtBa1v9QvxIhbMpVMcvXacXNXibjljAtkbe5mt3qkSDcG3eoQRj0VFoGCgT4R LXwGXpc1Qi2FVZIoJzr0S/10BdrlkRyodzneY/DpqiRdd4emQDRI41Waw+ySOdRSnWBz n2fg== X-Gm-Message-State: AOJu0Yz7UZXG7tZeIOggDcsjPOEoIU89BirjkfiytOwtVvoRfV5N5J93 TDQ2nUwbmdbewOqUzxksmZwUqmbwM5F0VBFe X-Google-Smtp-Source: AGHT+IEdREnUFQzLQLXziwcbdTCFJIEnWtbesYR/Yi2j7ybSJZObtayDa/HB7ljxXyH3M9oPbUbcVA== X-Received: by 2002:a17:902:e88e:b0:1d4:cdcf:97e8 with SMTP id w14-20020a170902e88e00b001d4cdcf97e8mr238233plg.126.1704218056928; Tue, 02 Jan 2024 09:54:16 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:16 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 7/9] mm/swap: avoid a duplicated swap cache lookup for SWP_SYNCHRONOUS_IO Date: Wed, 3 Jan 2024 01:53:36 +0800 Message-ID: <20240102175338.62012-8-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9CB0540021 X-Stat-Signature: 4g3mqfmzqkcxbbu99s4gtr9i47d3zcgy X-Rspam-User: X-HE-Tag: 1704218058-804457 X-HE-Meta: U2FsdGVkX1+jUlIBWRVzpOuNHcYi6Mgngb/nKAgKKXggd+vaPLvQtTMP546yu2ver5U0ekPS9v/dh+lToCOmj3hWw/tlZYmzlc2PO2vOz0UCR2g53mSsxMdsMOmv8xByHW016AlYwF78BOQnmn7dM3x90Tjo1smcPm+5tN4IoEtUN2EPgSJISmmLhEKfH/qBpnQc5d8i/oJs9vjgBEm3U/wTLCUcwzj8SpxxOJO+a0C/0v9oA7jDD279Pw70YiiHRfZYCkX2P6pKNUJsKDXCM7av5+mideFT2PLQ3MILr5b7XEyO8/c9ecSUTe/3QsIaCBS3D0Ac8EkW6FfEBju0Xts/CQEZ+NqIf6FUHHOe6cg/eKSuXO3h9PbzN4uuHDM9jAMDLBQ36J32QMQq1Pnp7Q0MG4ffS3G+fw+xhzc+9EaUYLiRjM0EA/Gy17D0kOSzpU2yy68JYo2beC5N1Mqf9jK5dkKz/vglG2AQwatmFFzyZaNxhFYWS18LgPmXo8n7gXKF2AbwcMoKQt8eo2R4jwl8FnYET3k9mos/xqJvqtB7cJIv6ExTqDeSN8KRs5ZMLoeRcSDP/S+mfTSqU3769Bdsby29ol+GpEJ5XX07gKBtrlxJE7G3CiE/eIoxTmYmLM+/8a5TrKhvT9gk1L7bhTZELTwqs4ePFX23j9oJGwVD2xCmX39m9DflOUon6F4T+CFpqJsvjFYd9LnNWBYHDmvhnulkrrzXI+qJoPJUeWv4ucF2+PBAU/+FvxudUsBpV9UmdItPr47YT9aHUwgqlP9OW1PG9V59uXLPwweqPNaKI8hazSxGicAOMWlnlTr33ml1jgCAKPU7YvcFa10s9q6CReVOhDnbXyJ1hnQ9GCPeQi9k7BMHizD/wX0Whd5NspPCXVGGa3wpGuxiQK7RbS3Jzon658y3k2AzgOdkZYpietoD6SvILDb25xjGKZeYRZTshnM4MZoLiTnpxNe 7jm+pDC7 YB643/qCCxBhKVm5JzSdt0+nNv4Qq6/JjEEsLUFcbXhc8Grp7d8E47mZD8yJjiZw+/FuHhoonSoG1O0U5dqE2M1kxtRNdhyI0Dtjhmn3sf608U5GDLMpAZciIgOj9BGCfCksHTQ6tkJKgy1FN6HaocNHieAd7r22yKSQ+oJJ6QQmI0aOrXSsRpmHd0Zd3wuYXAD0VbUg1qIxnh1mT1lUud4WjkM8y3ynXlCc//iic3pJ3vg8WHvP+J1kori4F/ndT3Vw0wxhj6dLLakZqXe89ZE7tQtyLNnv0VRJbi4seLrgDEtgyIQ1xgsGf5tRWTsDcp0VfHKiJ2oEXrPiRvgCSMbjxmKEHOu8hf/5KkEGUfEm5RcxEn87F0m1SjtkpayL8yY/j/dnHOZjqx/xoeA6ydPojMlhAaqzoxsZHN6SYnLPIhfZnzYTZluLKflUZeyNRlV9Z0iyrKAfav7jo27ixJZMhkF4CT5HYL9M6sZiVjcPr6krHcHkEiwUFZTofD8Xul9h8bHM0Hy4E/ecp418qmJTl8HfGMVBaCUquoSB9xU/FrFb8nrSxQiXnkz/TY77TchiwCmODHh58GUb/mXzac9gyDdMB6PNd6QuxdYk5ZdvPYtMKHFlxq+uzKnPZlayJZbU52qGpF1PqBbCH84/BZHOgqgSVS5Vutybt6OekkvgrKOs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When a xa_value is returned by the cache lookup, keep it to be used later for workingset refault check instead of doing the looking up again in swapin_no_readahead. This does have a side effect of making swapoff also triggers workingset check, but should be fine since swapoff does affect the workload in many ways already. After this commit, swappin is about 4% faster for ZRAM, micro benchmark result which use madvise to swap out 10G zero-filled data to ZRAM then read them in: Before: 11143285 us After: 10692644 us (+4.1%) Signed-off-by: Kairui Song --- mm/shmem.c | 2 +- mm/swap.h | 3 ++- mm/swap_state.c | 24 +++++++++++++----------- 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 928aa2304932..9da9f7a0e620 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1872,7 +1872,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } /* Look it up and read it in.. */ - folio = swap_cache_get_folio(swap, NULL, 0); + folio = swap_cache_get_folio(swap, NULL, 0, NULL); if (!folio) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { diff --git a/mm/swap.h b/mm/swap.h index 1f4cdb324bf0..9180411afcfe 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -58,7 +58,8 @@ void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr); + struct vm_area_struct *vma, unsigned long addr, + void **shadowp); struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index); diff --git a/mm/swap_state.c b/mm/swap_state.c index f6f1e6f5d782..21badd4f0fc7 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -335,12 +335,18 @@ static inline bool swap_use_vma_readahead(void) * Caller must lock the swap device or hold a reference to keep it valid. */ struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr) + struct vm_area_struct *vma, unsigned long addr, void **shadowp) { struct folio *folio; - folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); - if (!IS_ERR(folio)) { + folio = filemap_get_entry(swap_address_space(entry), swp_offset(entry)); + if (xa_is_value(folio)) { + if (shadowp) + *shadowp = folio; + return NULL; + } + + if (folio) { bool vma_ra = swap_use_vma_readahead(); bool readahead; @@ -370,8 +376,6 @@ struct folio *swap_cache_get_folio(swp_entry_t entry, if (!vma || !vma_ra) atomic_inc(&swapin_readahead_hits); } - } else { - folio = NULL; } return folio; @@ -876,11 +880,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * in. */ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) + struct vm_fault *vmf, void *shadow) { struct vm_area_struct *vma = vmf->vma; struct folio *folio; - void *shadow = NULL; /* skip swapcache */ folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, @@ -897,7 +900,6 @@ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, mem_cgroup_swapin_uncharge_swap(entry); - shadow = get_shadow_from_swap_cache(entry); if (shadow) workingset_refault(folio, shadow); @@ -931,17 +933,18 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, { enum swap_cache_result cache_result; struct mempolicy *mpol; + void *shadow = NULL; struct folio *folio; pgoff_t ilx; - folio = swap_cache_get_folio(entry, vmf->vma, vmf->address); + folio = swap_cache_get_folio(entry, vmf->vma, vmf->address, &shadow); if (folio) { cache_result = SWAP_CACHE_HIT; goto done; } if (swap_use_no_readahead(swp_swap_info(entry), entry)) { - folio = swapin_direct(entry, gfp_mask, vmf); + folio = swapin_direct(entry, gfp_mask, vmf, shadow); cache_result = SWAP_CACHE_BYPASS; } else { mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); @@ -952,7 +955,6 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, mpol_cond_put(mpol); cache_result = SWAP_CACHE_MISS; } - done: if (result) *result = cache_result; From patchwork Tue Jan 2 17:53:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4637FC47073 for ; Tue, 2 Jan 2024 17:54:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB96B6B0155; Tue, 2 Jan 2024 12:54:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D69B26B0158; Tue, 2 Jan 2024 12:54:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0B106B015B; Tue, 2 Jan 2024 12:54:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AD9196B0155 for ; Tue, 2 Jan 2024 12:54:24 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 94A87A0864 for ; Tue, 2 Jan 2024 17:54:24 +0000 (UTC) X-FDA: 81635120448.01.96A147B Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf03.hostedemail.com (Postfix) with ESMTP id B575020013 for ; Tue, 2 Jan 2024 17:54:22 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gyxTJ2DM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218062; a=rsa-sha256; cv=none; b=0AVKUpKnwB+sBHgkO7mipqTpSgjM2Fl41VVxG8fiEzJFVcHNY/q/uWzbaAYbJfOLuTiGWQ beChp8+oxqA1TfTR+YGz9C/T9ENS17Gx97Iv+R5CckusczAT2qks/CtYL4MTSmn1LNxWOe 8dUN7MhQTBOibHqCjrjhDKydvjc26wY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=gyxTJ2DM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218062; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Plx2aGPJbBehNqIW4p1L+Cff2qlR20r6A012vbacRHU=; b=Kaib8Qf/YQPwqh/BT9IZSDRIZm40csx5NxygoqMm0LHRhG5KqPz81uOK7L9/EceXVi4wn/ 4VLFWyHMpzqFTRJ6hr3TYP1Yq8wvDjE8Ozn6wXe8fhZIhGC02tOj9xEL36fjl9dpiuVSHh fmQHWYaFREaY1E3Cp7V2FUR1I8yCXQg= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1d3f2985425so34728175ad.3 for ; Tue, 02 Jan 2024 09:54:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218060; x=1704822860; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Plx2aGPJbBehNqIW4p1L+Cff2qlR20r6A012vbacRHU=; b=gyxTJ2DMcPqfkpRg4OIRUKzhsS0/RpWDDGQRPcodFE95Y31ycGknQL2sl5mHkhqn9t aN1Mg/Pi138qohhUTxwh3z7cyfPJCnC8roBmAvIazNwtT7bTbw3yD+ZGLlsCOsDBL7Q2 mQl45v3v3NAkx8bzLbUCn+ldLrBFU2LCiPmgP2pMlRTGK+yJn4Ke3cxuNp3leBh+SMM5 uE+hmuT+PeSOwYrNI/WuOnXNO3j6OUVYYXw5DhatNCsqJRAnDzfhX9X+CooWhtGXUUVv V3Sv3dl3bZ2u+krokKBfe8PUxrUrE4mThPTHlI8ERY4G4QXltDXsKEcdUr17vbBFymVM BZmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218060; x=1704822860; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Plx2aGPJbBehNqIW4p1L+Cff2qlR20r6A012vbacRHU=; b=ef5UkpvTroQGvpP1WCCsCatFvZ1U+Noe7jMyejh3XL4mvsM7PFVqxPAIfhq99/EJ9c e/7862mddwipECQFJZOWWYjugGmtbCILiCMDh4xXVZhnzmEIf1mPjjpmRRAGJZzksDZV XCqUgZ5zMRCZTQgQbPD6aU2bny0P171n+G+YCjjH4ha8DKLXQopulZGLOdyieUjEXjgn TXVwK96XA3/QeXLvPDTL0WzS/L/EgfR3SblyTvHmcKVtmbcm8BgA2m4bkn2spaDyqD1Y fwzE8Fxj3PbVoFJBgJTeR7m7Aug0L+1u1zgrmjxhbjAsWy2Va/tZdKgo0ocMZIX0QRZK hz6Q== X-Gm-Message-State: AOJu0YyuJdHzdR6mC+qXnSRuEsfE8HtLXU3mpvhCpsBPcms5DYR//mqb cdjNhnyLrvbesCG4LOUU31Nn+w1HxrXMgR/r X-Google-Smtp-Source: AGHT+IGr1fEmKCpFGmKewU8/hVrmqMb8AM0dTCE+QEyY4/Rq+9azV/ZjBTyHv60Nu5X7VVBSfSChoA== X-Received: by 2002:a17:903:110d:b0:1d4:cc31:71e with SMTP id n13-20020a170903110d00b001d4cc31071emr483019plh.67.1704218060519; Tue, 02 Jan 2024 09:54:20 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:19 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 8/9] mm/swap: introduce a helper for swapin without vmfault Date: Wed, 3 Jan 2024 01:53:37 +0800 Message-ID: <20240102175338.62012-9-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B575020013 X-Stat-Signature: ghk7gczj3p959kap65xq6djp48hqnoka X-HE-Tag: 1704218062-983891 X-HE-Meta: U2FsdGVkX18H2pwm9xof9iWsCyZm2ReYA7dyBwXzUxitXSqTIjgClI2oISmH9y3p6uoEq1RxRkbisanCU37D6G051WN8KauufG8zhGytEq+ocIAZKub7lgHPkwBJLL1dOc6zNVMgWF3+peX4eWCtkfNfBF52KX5NJTqBLGiRvp6Rq+pDlaj6lgTmNuCatR9GCRY9bUTq4XvmjYLtiNj6SiP/33OKJIZSBYxICLvty9P4EsR0F1Vo7SJelYrXOZpQ9z6agCPO7e5zOGN6y/fpsoPEh4nA9FbN4zVLVDUq8ewQwJU5oc1ulZMly3MgcF1rwgI6VeiQSD/1KfcvlIKUKutYBf8/MHFYARxH0B0oGuS3CpyxDTMyGdR6oLpd9STugOBDp6F0KkhwJTMu8/eXbiouhkOpdaqCNnFXlsMCtoZsvPGsSbVaHtIuaclRZKAHlkVC9ONSQZ31+YRtXx5VE6Re66lDRv71rxQ98D6BEgrLLlOqCxVgC/TqajhJP91Ba5N2fWtVxcLlPeODVd5zPT3aRLUwhY3MkReYj/XSH8691aS4E8536bkqSlY6j7p89sfugmY7q+9QZ4KahoRgzumGDOli03ESaMCvM2+h4gZpX3gim4Bd2F8QR515bcAsWzrFwOMoB3Q1uFaz3pBjClDQYfyn1eh3Tow8YiRxcevNJ5sjweNXc/ARIWeb6pDtUHknAmm96TpBORsAbJwt6x3cmJhCPaoWFPpwq98zigl0D9wA8OJyRd4jgPAPQ7Uvwlffab61uGmkgAyR6b+N8gqYTD2VPAsCfrUACUH284fKAFaS/RWm/8tj4J5nQpee0JYPtDEXSJnndswB3n7I9oXbA6FDHV87nM6MH1YMgCdCzkr3q5LZBNW+uUHxJUncqoEul7t63dp7qvG/Rlv/iDNpYj+yG0nCywAo6Rxl9iqqO/Zx0l5hU9WdBmVgQEbFZIkDHDWeD4Qg1LAF8of ewqUh2js g5LiuD5kCWZKpLrWHz48diNvXcoD5jqG7bljVVI49+t04LZT60Z1N32f/667hP447srgJD3UjnM/wUQpiWitZ7SzG30OBG74XAI6BLzTI7QfbVM/bblGqxD1p5SUz/tN9y9GlooprIFE+Ff0uITFD4DGmNxyQXjViERh/iFFyPfhBJewaQ8jzAjdYPa0oueXErVri2pB3DfJ9LkctKOrjBvMSIDDfxl8qd5rsOPKCgf7PtXnnvjTllN5c2bLMmSuQhz7MdDFn+Ezus0RKXHV9LI2MfoChGTn3NNjQsbchJuzA9BFX9+9eRoigf4cfCp0T08wrtmYQr/rcI3NPVJS+5BTUeco6I2PNuoE7fc2TfsX3hMm9DE+wPtCtRYetvi30L2AcRlZbtfYW9x1ZODrsdSm9hp535HK2IwODxCiF68xQziYBbYgSuqG/b14huBTMp2GJDpf2n6e3EFS9Qg+v4skYf9pIsdUU0oGRvOQkiYdGIuYclwlHfs2mMjaPdPrPY2i9OGqEvGNoXOEQTV4YpZ7KXFGnT6TAaBImCXF0QK5LcHZZy3HFNjY40D1YqFVudSF6vlfD6GdaA1BJ21AtdFwpGGzVqmMgPhJu/aBFug1TPu3867UtrzCGVJUtM0HMgjFP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song There are two places where swapin is not caused by direct anon page fault: - shmem swapin, invoked indirectly through shmem mapping - swapoff They used to construct a pseudo vmfault struct for swapin function. Shmem has dropped the pseudo vmfault recently in commit ddc1a5cbc05d ("mempolicy: alloc_pages_mpol() for NUMA policy without vma"). Swapoff path is still using one. Introduce a helper for them both, this help save stack usage for swapoff path, and help apply a unified swapin cache and readahead policy check. Due to missing vmfault info, the caller have to pass in mempolicy explicitly, make it different from swapin_entry and name it swapin_entry_mpol. This commit convert swapoff to use this helper, follow-up commits will convert shmem to use it too. Signed-off-by: Kairui Song --- mm/swap.h | 9 +++++++++ mm/swap_state.c | 40 ++++++++++++++++++++++++++++++++-------- mm/swapfile.c | 15 ++++++--------- 3 files changed, 47 insertions(+), 17 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 9180411afcfe..8f790a67b948 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -73,6 +73,9 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf, enum swap_cache_result *result); +struct folio *swapin_entry_mpol(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, + enum swap_cache_result *result); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -109,6 +112,12 @@ static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, return NULL; } +static inline struct page *swapin_entry_mpol(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, enum swap_cache_result *result) +{ + return NULL; +} + static inline int swap_writepage(struct page *p, struct writeback_control *wbc) { return 0; diff --git a/mm/swap_state.c b/mm/swap_state.c index 21badd4f0fc7..3edf4b63158d 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -880,14 +880,13 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * in. */ static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf, void *shadow) + struct mempolicy *mpol, pgoff_t ilx, + void *shadow) { - struct vm_area_struct *vma = vmf->vma; struct folio *folio; - /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); + folio = (struct folio *)alloc_pages_mpol(gfp_mask, 0, + mpol, ilx, numa_node_id()); if (folio) { if (mem_cgroup_swapin_charge_folio(folio, NULL, GFP_KERNEL, entry)) { @@ -943,18 +942,18 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, goto done; } + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_no_readahead(swp_swap_info(entry), entry)) { - folio = swapin_direct(entry, gfp_mask, vmf, shadow); + folio = swapin_direct(entry, gfp_mask, mpol, ilx, shadow); cache_result = SWAP_CACHE_BYPASS; } else { - mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); if (swap_use_vma_readahead()) folio = swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf); else folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); - mpol_cond_put(mpol); cache_result = SWAP_CACHE_MISS; } + mpol_cond_put(mpol); done: if (result) *result = cache_result; @@ -962,6 +961,31 @@ struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, return folio; } +struct folio *swapin_entry_mpol(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, + enum swap_cache_result *result) +{ + enum swap_cache_result cache_result; + void *shadow = NULL; + struct folio *folio; + + folio = swap_cache_get_folio(entry, NULL, 0, &shadow); + if (folio) { + cache_result = SWAP_CACHE_HIT; + } else if (swap_use_no_readahead(swp_swap_info(entry), entry)) { + folio = swapin_direct(entry, gfp_mask, mpol, ilx, shadow); + cache_result = SWAP_CACHE_BYPASS; + } else { + folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + cache_result = SWAP_CACHE_MISS; + } + + if (result) + *result = cache_result; + + return folio; +} + #ifdef CONFIG_SYSFS static ssize_t vma_ra_enabled_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) diff --git a/mm/swapfile.c b/mm/swapfile.c index 5aa44de11edc..2f77bf143af8 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1840,18 +1840,13 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, do { struct folio *folio; unsigned long offset; + struct mempolicy *mpol; unsigned char swp_count; swp_entry_t entry; + pgoff_t ilx; int ret; pte_t ptent; - struct vm_fault vmf = { - .vma = vma, - .address = addr, - .real_address = addr, - .pmd = pmd, - }; - if (!pte++) { pte = pte_offset_map(pmd, addr); if (!pte) @@ -1871,8 +1866,10 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte); pte = NULL; - folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, - &vmf, NULL); + mpol = get_vma_policy(vma, addr, 0, &ilx); + folio = swapin_entry_mpol(entry, GFP_HIGHUSER_MOVABLE, + mpol, ilx, NULL); + mpol_cond_put(mpol); if (!folio) { /* * The entry could have been freed, and will not From patchwork Tue Jan 2 17:53:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13509234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6466DC47073 for ; Tue, 2 Jan 2024 17:54:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF2776B015B; Tue, 2 Jan 2024 12:54:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7AAA6B015D; Tue, 2 Jan 2024 12:54:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1BD16B015E; Tue, 2 Jan 2024 12:54:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BD8AD6B015B for ; Tue, 2 Jan 2024 12:54:27 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9FA011A0787 for ; Tue, 2 Jan 2024 17:54:27 +0000 (UTC) X-FDA: 81635120574.04.207DB46 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf20.hostedemail.com (Postfix) with ESMTP id BA8661C0018 for ; Tue, 2 Jan 2024 17:54:25 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="M/msYg45"; spf=pass (imf20.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704218065; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sYlN8vSKv83J8k3KTy0zWmAC8wYuURkZt3/TH4YiDKw=; b=8WGxxtABo+/GkdiVuoiDW2nWNYX7OZaQy+fKhaAhI+E8enbkS4jwlABV2RlFhShmbZw6un UGefNTFZ7KSev/ooOtPoVMIc/AW747zzEL6zio9noslsXMcoHazRhW/gd7hfw/609I8Hwj mrbPH1F1tU9kt5RnB64gS2WBrt8HzBk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="M/msYg45"; spf=pass (imf20.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704218065; a=rsa-sha256; cv=none; b=jgYy2Ll2bzjVBmC0dnmbzcl6G+ZxWaDSEiR49JvN65NWjeJ8SUDWmVgJKEerTQpqyyHSQ5 rfv6WQcX5B3mctvodDHILGw0eQOn9ZEBRPdPRV+hgiEPEmuwYho4C2gCmyDfcjlARWHIIs KDrRSQ0eQKBG13kT6PuRz95QvRm8TU8= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d3e416f303so23753885ad.0 for ; Tue, 02 Jan 2024 09:54:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704218064; x=1704822864; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=sYlN8vSKv83J8k3KTy0zWmAC8wYuURkZt3/TH4YiDKw=; b=M/msYg45qoI7HVsH4Sd6XG0Xt/P9RC9bCoxRWk7zpf9BcaTRVkriV/KrrPD06J/eP8 pYW3Kq0xKRWIA5onwpHjwFRezci9wWy6dq1tJWpt9HzOoiJhlcWWCcz5uwEao6wrYOtv qZY6pYTFN6JFFSqDRdoSIL1dGnurXR79bHRgMFOtOQBP2wcetyA+6mO4JSwuWSLZpq92 46lD9xCDzCsMUEyoeafJKQUsoM3E36aPOy1KajrfPHZOiFNC8jMsSqPwzQaXAYSAvMi4 YjpsVashXOR+FWBx9IjNdmhelhugUMBKPbtgOitov9XyU4OcxlcJPxm2oxGGynq0vcvT 4Rxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704218064; x=1704822864; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=sYlN8vSKv83J8k3KTy0zWmAC8wYuURkZt3/TH4YiDKw=; b=G6PCYcptERva58aVllXhBd//LTs/qyD5Ighrm4XxOx9EWp2h8CW5U07fI1DjjkLNRa 9hOGbfiV5tGIHihw1fSyUNRAX8npqAZjQunpPk1zeRKdOUd/HO4FpdriAkV5MzwVHzVN X7hlxnkADIOG9IxA7zsvYiGZu9VnEBuaD+lTuddB/x6fP1lF8J2RcWAH0QAx3YkvLQ2P iqSdmECHZpLTKldn2w9IHUtvW4wXgj+OD32a/QfURSSEFDTm+2rLDAcPZRqK3NLqWfMB qrA3tDZ27FJZ71Vx6Xs+cLUxi8hVnDlKSaT5+bGSXaOBlt6p34qhFz4dvTRN+FYiLFFx zdsA== X-Gm-Message-State: AOJu0Yy7eycA4RKPnTqUWB9jmaCrxZNtWSG5QLFaF9BaezfBprzBm0UT moPKUDuWZNe1M/RBKA43rvO1C2ZNuxijlRI4 X-Google-Smtp-Source: AGHT+IEl/2qWBPW9Z5FVh74mUbrKLR970e9KepR/VbDcVwyuR9p3xpCkYWqKFuuUY+RAsB8Q97us6w== X-Received: by 2002:a17:902:ea06:b0:1d4:2066:68c with SMTP id s6-20020a170902ea0600b001d42066068cmr8090026plg.130.1704218064019; Tue, 02 Jan 2024 09:54:24 -0800 (PST) Received: from KASONG-MB2.tencent.com ([115.171.41.9]) by smtp.gmail.com with ESMTPSA id be10-20020a170902aa0a00b001d3c3d486bfsm22151969plb.163.2024.01.02.09.54.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Jan 2024 09:54:23 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 9/9] mm/swap, shmem: use new swapin helper to skip readahead conditionally Date: Wed, 3 Jan 2024 01:53:38 +0800 Message-ID: <20240102175338.62012-10-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240102175338.62012-1-ryncsn@gmail.com> References: <20240102175338.62012-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: BA8661C0018 X-Rspam-User: X-Stat-Signature: n5yyyj347zzd5km4x6nwytz91optsemj X-Rspamd-Server: rspam01 X-HE-Tag: 1704218065-202784 X-HE-Meta: U2FsdGVkX18Nqg3COH6VqxFU2W3u1ezVvVbFMQwHzIgDI78eWrYG2HB0HNP4B6zfuTiLTBhtICrQfLU7iq9PENyErCJxfWAmQCVD2s1WGLRQx+kIWuNiUBSHK8rZolyjraNnN3Ii9JcDuFCzcZQwW3YBesdSQODzCZpN4NS/ZlBFJ7sEmON86RNGpZ1H8hkpIBQgGmlR9ZPImIbwD0Xwdm+CZSSddkuC6nuXZY+r/VtfbNNhA7ZVfCtFgbXnJT1hXM+5BESfIPo86wypcMh6QNrmIYruQNNb9r303Fxyrfej4f21fRJxxIK8viWIcA6Jbp3jgLh7Ai9i2cEdKW4K165ifRJSKBNpoGkqDqw89UlMihaYpT6g34L/tum3h6J5Bt8qXQckUXmVKNDvt+i82+C97t8n4Wqmf43bi3UvXUWYdkuXP0prSv7nWabqIQFI/LWCWTdyS6BFGbCOgpLzKJyfX0wHCVMny1IElZlbMHQMHJ1z+AJ9CGCQW/KsFC8AipERl6yUkta6mKSSnRCk7i4X1t/O6zF/ZyzdGT86MShlRDMzpB4G5XtnnHfgWZ5VrkawsJMXQQhEpcK+lhkYPUpOs92lvt8LKFhg7NJFL1+H5YgPcvTlavR+vfuWlJDf7aUTYG2dB9EkX29ofXeRonAbsQj6GqA6YAIrKIDGygPg0WoG/SPwjCkrrhWbbfRoKkiRLlrL7Ec2D2ou96zT1mhmiGVQwufwZ23Cr3h4GowCg945GFfVDbPmdXkg9/RQQkANIk/Ob761xoESB4BId/o9jH/yfqymnQ7upwXsU7CCtF6Mp1V8ft4A//R9pMNuGL1MkIxtb5znKllV1aKApFoii/8L1Rve8dNR3qLkjUmjLshzc3I0CmuwwYW7I/uG3xdQERmWvVk2BlQqA31COweKzTEQZGt7Ppjy8M40xA1hYqU5BYTQJkpadhNl8NaWZvasDwvzeijzIi2Hn36 +ptEfbkU RER+XzWn1+ynkpO4W2m0DwUX26selCSt3lvg3Xu5wv63tbs+yY6DBDojQ9PfAdSUoQiSy2y6cPdZoolBst7pA17IgsoY+DXJnsja45xcAKesT2sgaj8bSnzgjmcsL5MQJaxngtmBR/yxR0F+JbQn+R+Oezi391TC7YL2NgR5P3CZ752XI6XX75NtK0BRmiLX0h6AlbCCCvkTCKUCBKjKgoa0TyEZE51pUOSA3i+DZnJoVbiswnvn6JnpVwBc97nMcQ2l3Zn3Pk8vV0CPWDXskACrzr7hWJ3v+k5kU1y5WV95I91KgZhr5RyvvvrqbSLNZw3X2swrwtSVJWJn/qIcewzSDNvb8l8G4iGBcTOVDPuak0C5FVCFOrzgQrJbfdlC1Rt8WIQOsQmUFCfPLuzHzEZ2n7uKdJXHJr8B548nI9n+iFiXsAhncRbejb70FwcVJgs9Rhy7WY/bxeTh4s6ClByn6KmXx46iTPnB8dFNef5YSg1+W9nbaq/iG3Pp6aDHzjc6dD7rxjfVSy1+/DIwkh2lBLSvMyBOihdMoZee3m8C3q+C7oigo7rO3UYVD+imspbdYNCiHyR3Y5F40bBdBwvJJL0NeFVbyuAB/toZ91cJeB8XoxTNxXajgXHinUP5qPRbF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently, shmem uses cluster readahead for all swap backends. Cluster readahead is not a good solution for ramdisk based device (ZRAM) at all. After switching to the new helper, most benchmarks showed a good result: - Single file sequence read: perf stat --repeat 20 dd if=/tmpfs/test of=/dev/null bs=1M count=8192 (/tmpfs/test is a zero filled file, using brd as swap, 4G memcg limit) Before: 22.248 +- 0.549 After: 22.021 +- 0.684 (-1.1%) - Random read stress test: fio -name=tmpfs --numjobs=16 --directory=/tmpfs \ --size=256m --ioengine=mmap --rw=randread --random_distribution=random \ --time_based --ramp_time=1m --runtime=5m --group_reporting (using brd as swap, 2G memcg limit) Before: 1818MiB/s After: 1888MiB/s (+3.85%) - Zipf biased random read stress test: fio -name=tmpfs --numjobs=16 --directory=/tmpfs \ --size=256m --ioengine=mmap --rw=randread --random_distribution=zipf:1.2 \ --time_based --ramp_time=1m --runtime=5m --group_reporting (using brd as swap, 2G memcg limit) Before: 31.1GiB/s After: 32.3GiB/s (+3.86%) So cluster readahead doesn't help much even for single sequence read, and for random stress test, the performance is better without it. Considering both memory and swap device will get more fragmented slowly, and commonly used ZRAM consumes much more CPU than plain ramdisk, false readahead could occur more frequently and waste more CPU. Direct SWAP is cheaper, so use the new helper and skip read ahead for SWP_SYNCHRONOUS_IO device. Signed-off-by: Kairui Song --- mm/shmem.c | 67 +++++++++++++++++++++++++------------------------ mm/swap.h | 9 ------- mm/swap_state.c | 11 ++++++-- 3 files changed, 43 insertions(+), 44 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 9da9f7a0e620..3c0729fe934d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1564,20 +1564,6 @@ static inline struct mempolicy *shmem_get_sbmpol(struct shmem_sb_info *sbinfo) static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info, pgoff_t index, unsigned int order, pgoff_t *ilx); -static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) -{ - struct mempolicy *mpol; - pgoff_t ilx; - struct folio *folio; - - mpol = shmem_get_pgoff_policy(info, index, 0, &ilx); - folio = swap_cluster_readahead(swap, gfp, mpol, ilx); - mpol_cond_put(mpol); - - return folio; -} - /* * Make sure huge_gfp is always more limited than limit_gfp. * Some of the flags set permissions, while others set limitations. @@ -1851,9 +1837,12 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, { struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); + enum swap_cache_result cache_result; struct swap_info_struct *si; struct folio *folio = NULL; + struct mempolicy *mpol; swp_entry_t swap; + pgoff_t ilx; int error; VM_BUG_ON(!*foliop || !xa_is_value(*foliop)); @@ -1871,36 +1860,40 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, return -EINVAL; } - /* Look it up and read it in.. */ - folio = swap_cache_get_folio(swap, NULL, 0, NULL); + mpol = shmem_get_pgoff_policy(info, index, 0, &ilx); + folio = swapin_entry_mpol(swap, gfp, mpol, ilx, &cache_result); + mpol_cond_put(mpol); + if (!folio) { - /* Or update major stats only when swapin succeeds?? */ + error = -ENOMEM; + goto failed; + } + if (cache_result != SWAP_CACHE_HIT) { if (fault_type) { *fault_type |= VM_FAULT_MAJOR; count_vm_event(PGMAJFAULT); count_memcg_event_mm(fault_mm, PGMAJFAULT); } - /* Here we actually start the io */ - folio = shmem_swapin_cluster(swap, gfp, info, index); - if (!folio) { - error = -ENOMEM; - goto failed; - } } /* We have to do this with folio locked to prevent races */ folio_lock(folio); - if (!folio_test_swapcache(folio) || - folio->swap.val != swap.val || - !shmem_confirm_swap(mapping, index, swap)) { + if (cache_result != SWAP_CACHE_BYPASS) { + /* With cache bypass, folio is new allocated, sync, and not in cache */ + if (!folio_test_swapcache(folio) || folio->swap.val != swap.val) { + error = -EEXIST; + goto unlock; + } + if (!folio_test_uptodate(folio)) { + error = -EIO; + goto failed; + } + folio_wait_writeback(folio); + } + if (!shmem_confirm_swap(mapping, index, swap)) { error = -EEXIST; goto unlock; } - if (!folio_test_uptodate(folio)) { - error = -EIO; - goto failed; - } - folio_wait_writeback(folio); /* * Some architectures may have to restore extra metadata to the @@ -1908,12 +1901,19 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, */ arch_swap_restore(swap, folio); - if (shmem_should_replace_folio(folio, gfp)) { + /* With cache bypass, folio is new allocated and always respect gfp flags */ + if (cache_result != SWAP_CACHE_BYPASS && shmem_should_replace_folio(folio, gfp)) { error = shmem_replace_folio(&folio, gfp, info, index); if (error) goto failed; } + /* + * The expected value checking below should be enough to ensure + * only one up-to-date swapin success. swap_free() is called after + * this, so the entry can't be reused. As long as the mapping still + * has the old entry value, it's never swapped in or modified. + */ error = shmem_add_to_page_cache(folio, mapping, index, swp_to_radix_entry(swap), gfp); if (error) @@ -1924,7 +1924,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, if (sgp == SGP_WRITE) folio_mark_accessed(folio); - delete_from_swap_cache(folio); + if (cache_result != SWAP_CACHE_BYPASS) + delete_from_swap_cache(folio); folio_mark_dirty(folio); swap_free(swap); put_swap_device(si); diff --git a/mm/swap.h b/mm/swap.h index 8f790a67b948..20f4048c971c 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -57,9 +57,6 @@ void __delete_from_swap_cache(struct folio *folio, void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); -struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr, - void **shadowp); struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index); @@ -123,12 +120,6 @@ static inline int swap_writepage(struct page *p, struct writeback_control *wbc) return 0; } -static inline struct folio *swap_cache_get_folio(swp_entry_t entry, - struct vm_area_struct *vma, unsigned long addr) -{ - return NULL; -} - static inline struct folio *filemap_get_incore_folio(struct address_space *mapping, pgoff_t index) diff --git a/mm/swap_state.c b/mm/swap_state.c index 3edf4b63158d..10eec68475dd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -318,7 +318,14 @@ void free_pages_and_swap_cache(struct encoded_page **pages, int nr) static inline bool swap_use_no_readahead(struct swap_info_struct *si, swp_entry_t entry) { - return data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1; + int count; + + if (!data_race(si->flags & SWP_SYNCHRONOUS_IO)) + return false; + + count = __swap_count(entry); + + return (count == 1 || count == SWAP_MAP_SHMEM); } static inline bool swap_use_vma_readahead(void) @@ -334,7 +341,7 @@ static inline bool swap_use_vma_readahead(void) * * Caller must lock the swap device or hold a reference to keep it valid. */ -struct folio *swap_cache_get_folio(swp_entry_t entry, +static struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr, void **shadowp) { struct folio *folio;