From patchwork Fri Dec 22 10:22:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13503164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B3B0C41535 for ; Fri, 22 Dec 2023 10:24:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E107D6B007E; Fri, 22 Dec 2023 05:24:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC0346B0080; Fri, 22 Dec 2023 05:24:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C87DA6B0081; Fri, 22 Dec 2023 05:24:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B6D716B007E for ; Fri, 22 Dec 2023 05:24:09 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8DA89A0813 for ; Fri, 22 Dec 2023 10:24:09 +0000 (UTC) X-FDA: 81594069018.18.EB66D66 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) by imf28.hostedemail.com (Postfix) with ESMTP id CAECAC0013 for ; Fri, 22 Dec 2023 10:24:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=KxVOvMYI; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703240647; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=gqSwQxD2fbJPgUWI2ZYQp2p14IC1JkNGTdv/EQhR7s4=; b=mq+wkSBDGysackOfGVNQo8yxrOGSmHnM+LzGGxsAfUNJODR/rZgYS8GAWCNaaCjGmLRvN0 PHPxW+4yhZxXf99TpXajsMy6fEjAoJd2Xn4BD7xtgqn15oqTGZbv3BEDRC7VLNEA62Lqz6 7q0giuJQ4uArTSn/p92RisX3pAzcH0A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703240647; a=rsa-sha256; cv=none; b=rljMrCw/sES+YBXJtbAmxX9pCYT86sk7nNuo0R9D4Y8eAv+RujVfeBXwv4qya6ZMgWHSP1 1dxOjL5Ym2kC5HkjUTFA/7K92d95GD876k2HPMGt4h73giKRN3XOn754IOV4ZARk80tVz7 GkyGb3O2zZgtzlEeyQkzThAqKrnCcJY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=KxVOvMYI; spf=pass (imf28.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.167.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3bb8f3d9f98so141601b6e.3 for ; Fri, 22 Dec 2023 02:24:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240646; x=1703845446; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=gqSwQxD2fbJPgUWI2ZYQp2p14IC1JkNGTdv/EQhR7s4=; b=KxVOvMYITrP8ZY0XYRmCR1NScePAzrplD25fveaO1KhfL5OPa/DndEzhRT0WYx6YB4 BK3KGDJOSdI0+AaoXTIB2OrZNwzjNmWTihpknqK0Pez720EocmflXC97vEwX1lEdK4eV ZMKLjgzQgx1y38nBT1+W1Y8/DxT3A93l+P+TZMiNy4IEq28pSPZTR8UcTQWYsxFj20HY XKE8SELhqXV+H8SdleypAZFjv498lmcueGmvnMDUrNKX4o+y2rUm2nR+8Sm0RQK0ya8H 98xISRI50yOPOhkDyNcoC2EOb9DDXrOm7dhxZXOYG778Z4LF9hDg2dylUgVlCkBpdiW7 uPIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240646; x=1703845446; h=content-transfer-encoding:mime-version:reply-to:message-id:date :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=gqSwQxD2fbJPgUWI2ZYQp2p14IC1JkNGTdv/EQhR7s4=; b=JP5vd0jhhMTzxX5+XXPztZ/05X9DT0HFkpg9gIX0XAbru/ULkoljLj+hBKQUNuyga1 ORWGRHgZKFU3QMHyycPFCpRqdXbKmf+e2Rki07D9i9Ocx3V7f5idDIfYHGvOmDWWV3oK 5p8exl+kvOwwmYdDPv2AlUcbOAzXnw1iKgEwmRGsYqZ1UqYFso7VnXDew3rCF/McTLk8 MM7UblYvZkKSeQ7XD8SWIjfNk1nVzf2LCelGiswkgRxhKVIsc7K5sKDZFCC4a0Myty4l KNVZ6rNUfTvfvZTTEV+o8w0iJ8FniJJvto9lBQ54r2L5+hYVC+42QYrDbDoiZYWEsdYe bCng== X-Gm-Message-State: AOJu0YzQ6a5a2ueXWsf4sSHguXTSu0a2A4c2sfCqNT1KFIvU/H9+kCUj o+Y3nrbn+dcNksisRylBYd79OORSf/cY8gta X-Google-Smtp-Source: AGHT+IEJa5V83giUnWT10v5sm3MCod/3MLsT57MZYMEmnJ7bVD0I31MCuTB3XHt55809YhFqtspSXg== X-Received: by 2002:a05:6808:2e4b:b0:3bb:721d:8ac9 with SMTP id gp11-20020a0568082e4b00b003bb721d8ac9mr1551420oib.18.1703240645875; Fri, 22 Dec 2023 02:24:05 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:05 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 0/3] mm, lru_gen: batch update pages when aging Date: Fri, 22 Dec 2023 18:22:52 +0800 Message-ID: <20231222102255.56993-1-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 Reply-To: Kairui Song MIME-Version: 1.0 X-Stat-Signature: q3ssetubmd66fpw595yxb4ktn3ygcdqy X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: CAECAC0013 X-Rspam-User: X-HE-Tag: 1703240647-600835 X-HE-Meta: U2FsdGVkX19LnETTJvE8uKSGWgXEjixVZhLrwIN1Ts12NveMgieXhNpBdnCDt+Ld3Gfrii6QIUjpkQn7K9Xrg8EZpR8DgSk0yAgGYiiXNcqmrASR3FvDntGEMkvomc1bWbzhBWNAvHs4z6biIOYfPYShY9bjJ/gqQS4MoFNA57r12IU9OzR66yxFJ5bY2KbgEP1FoAOQT6KiCs9R6uuEfMp5pSh6KYm7fB1Jh6wgOaEa+8oUQUH5TsGutrvmMy9rnkA32wgxLBTwyUZgWuo0n5DgX5SNXj66Kr3lwYuLPIoG2WMFzhhJwoEj8HcDUVJW45KR5bkED+4J4gTdpDp1MHWH9W0EGw0iLNdk4krJ2Wivp0RTlfS5moVx3DonTiHXwdzWNfHY70vgVkIMuVfA/1qJad9innnYCUnjYqsn6pjdpv55LC41l4LaexJ8dqx6bzFlk9TD6fVCS27ewaG2i+ipAH0Ktp1MxuLqWmCtvzIVd818ftbX3z1n/JfroMxWMigMsXMbI2KrazVaoCZJ8YRArd42rzGe1+myX+Q+kB8q6K+07rdHHLXl6xdNLEoliz3Ce7xBbg1y3FmM2+fZr1RUqJ5WYG6lfSfTx5PIP+IC3YTmwjWoEGBFMmJNjy9dDNzndaC30ICujJkxUZ+Eag3gR79zC5b0E9KgZ8FNbqjoor2gOVPnEKBcbtuSpmCvi+KxwyS8KhCZ4y42JR+KhinI5At0SA7fJO46HqmBUmesGpfkQPh4jG//x3NTkEhOJu53zZ2vRyMZvYiIoapJn1mg1oyd7A9glP67ZsTv2cqSqSRyZu0Qie556SImaECD72HWzUkzve0sMx2aIynS6/BB8WcCNaGPxjep7gwN8uEuHLU+Kzzejnr+rBwPRuXDsWX80bf/67gCTlS/FpCNmGswBWxv/qnfAuBCeVbr28lRFiapSUtOXP2JmSunrEPwnfOcO3JpMgQ7adFgHJH 0zyHStCs P5XMasE7YLAJHtfrhPMGbA3u6tEOqZmQ/d0Y7aAnAChRcZmE+i/MSyx/JP01mkulwEHL67BZM0ZJ2mm2nUCcZk+uTL50dpwU1VB1Rp2XR+x1S6QtpvivbMsgYS7Zb6so0cnSL2Im96MtjP6nk171G5W7Ti56dubAfn+H5vRI5P4iRQY6wAAonPWAwMY9/pYD6mKAZ9/AisTK/pxoScfx2xZsqZIIY6BbbWTCESajPrU9CrFpKkul7v8vnbesxUNkDCnXyFh005qxaTEzM9PaQxhf/mQR9VPGznrUYKTgy0PCdT/gwSwyq8J7G9dglBtrPsqCD/XPfX5dyUT99R3vKxCEClzQP/vxmYJCH0iJrlCJNjBMee1/P3iCD7gnSwC/TEaeLDpzUfSXZVE4ougtrVsERXXDD7iSDnbG5MdgUIx0GsCZcsyxkg6QjAb/YK8SNcKOJWtiKy/eVXuyZiYYtWcuqbXP/h8o2ZTDB990Qw491q89NC5VHwfoT7XKR1J8bW0eTPbi0SJyfOdmWAoaxS/WHe27BNZ1WkiTpVJq/e5ACEiCk5d/bOprXHj+mSJTkBWLJsRli1CQ/EWCaSMb8AO74HEpEq4yA7yE49y9Tv5vjkOgJrct7Dpn90b4axEa0CMPwn35Vu9Yt+Q0kkzKl9JdK0F2r+lTcz2mfWGaoCHQfh+I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.413548, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Currently when MGLRU ages, it moves the pages one by one and updates mm counter page by page, which is correct but the overhead can be optimized by batching these operations. Batch moving also has a good effect on LRU ordering. Currently when MGLRU ages, it walks the LRU backward, and the protected pages are moved to the tail of newer gen one by one, which reverses the order of pages in LRU. Moving them in batches can help keep their order, only in a small scope though due to the scan limit of MAX_LRU_BATCH pages. I noticed a higher performance gain if there are a lot of pages getting protected, but hard to reproduce, so instead I tested using a simpler benchmark, memtier, also for a more generic result. The main overhead here is not aging but the result is also looking good: Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-3: 44890.50 Ops/sec (+1.8%) Kairui Song (3): mm, lru_gen: batch update counters on againg mm, lru_gen: move pages in bulk when aging mm, lru_gen: try to prefetch next page when canning LRU mm/vmscan.c | 140 ++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 124 insertions(+), 16 deletions(-)