From patchwork Wed Mar 19 19:30:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2735CC35FFB for ; Wed, 19 Mar 2025 19:31:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 404D0280005; Wed, 19 Mar 2025 15:31:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 38CAF280001; Wed, 19 Mar 2025 15:31:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 192B7280005; Wed, 19 Mar 2025 15:31:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E74F3280001 for ; Wed, 19 Mar 2025 15:31:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DBE121C81DA for ; Wed, 19 Mar 2025 19:31:31 +0000 (UTC) X-FDA: 83239294782.16.ECDDED2 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2040.outbound.protection.outlook.com [40.107.223.40]) by imf23.hostedemail.com (Postfix) with ESMTP id D8A7114001A for ; Wed, 19 Mar 2025 19:31:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=rcFspfru; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf23.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.40 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412689; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=IVMKreYDz0DrkhOmW3SSkO0yClBTLsz+JeiJd6tjfc3b9kDM20h4rJtIhV6d8j2YDcFP5Z YrzJ6yXMTxQNHe3bUAruBg/TQVwJ24xyvbw2cBzevyvhX4HD/edxzhMtnuBmXD5FTljt/z w91xshkaVH1U2Ch0ixINRKdGpb5V2sU= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412689; a=rsa-sha256; cv=pass; b=NopzLBtOsdl0e85lDtCk7SmpVY4YdlSZskJ/+Uh1l+NiWqJFLXhQ0heyW7DTJvmGACwoPq HHbwJxTN8pAo2WNqDVrXpVoaJeeC24y+y7znupwApKp5CZ6CH4X4PV7U7DCc2hoF0XU/dL lS5nYkRTy2gW6EfWfZIjQ8Lnwu++WTo= ARC-Authentication-Results: i=2; imf23.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=rcFspfru; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf23.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.40 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kibCpY+AWIHnpIBsHswAK7OW9k8W0vfpD4oZeaGnpRyaTaoPs1NooAsWgc/bp1oep/N5HQAWXVxmKNId6bVE8MYl53R1DZZMKHZSJ1KBQJijIQTOs90osHu4Ki+9Rj9gHjQPKta91IeDE7T7nQZNRAjdKgLByMg18MVfGTbpKUvezQ7T7cGkGR1Z20w9+cQberO9lGagvw+sp+1f53LN3bd/yaoXCq4XrvpqRlfHvTJMBpCNaNDcoH78ZwNKMc7p+rG3wy95MtZcbA8fW9MRWXWoOJp97RtfgYY6ULUYs3Rb3gysWK3HvHv6chGhCVfH2VqK1pb9ETyAbVpa13nmLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=BQxN1+zoYrzxYDKXDthRpMaUaTcmQV4i80r86u5rVouBhOGvaasLvkBGiwlM/LwHMuxCVyC6pIvykXJex3cLoO3+/BDa6P+t48PSu5TfUp/akkmkum0gByFvYszDcCwARkQ7UQmhZe7IEj/UEocRq7jIDRRYsQsmFSmWiyiflnPtgTybTR2eIRPQeuwduIA35tgd4JvYoY5gESSNeeSaiDphW5/EcifTxsUPqGsX1J2cD357Jgujuy04YlMtqWdVGjh+eJ/0lQbh/xmvuKMoz/HE45kPo4Y3ZIZMt+y2ZiRBTYKa2NJ3DhiyUaFN28iGtV/Viu3qrJk7W3S3xCZjwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=rcFspfruOE/sAd/tYVASi3Ib8RxA4n+CSPANb6WncngYZwng7fYy3Xg0hB62tpvH1wDEMiZrnOZhmLpB0fzsVVp1Zm19X0WGugptC4J8Miqe4xZhQnL495gjjpphGLEfTxm6199RsWG4herI3gEh4ptSl2gc84tItucGYC/xpYE= Received: from SA9PR10CA0015.namprd10.prod.outlook.com (2603:10b6:806:a7::20) by IA0PR12MB7675.namprd12.prod.outlook.com (2603:10b6:208:433::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:23 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::85) by SA9PR10CA0015.outlook.office365.com (2603:10b6:806:a7::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:22 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:12 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 02/13] mm: Maintain mm_struct list in the system Date: Wed, 19 Mar 2025 19:30:17 +0000 Message-ID: <20250319193028.29514-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|IA0PR12MB7675:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e91f9a3-0125-4018-4357-08dd671ca186 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: dexsYIeqHGJqJtQI2EXSAibliOi3YzKMApOlwl2TqowRFfLgxr1MOTyuSD7PvOFhWadrbBOa5bdaxUiHpruKEBwTJt0W0epoh1bzTuQrIoMERDwMLoWdPgYRjrjO8E8CMLSR+WadaTCmJX6XkMXE13JehXlw5vXEdy39m6i5Mk78wwlnExHh/8671u9iqowCP0chs7KdC2dgQo9gpwtxlBnHnfGtkqy+AUPzky1W0zK/WVPJ6/WBijiJbV5/MVGTMf2xPJGz5nny2x9HDTkW1CvCm/dpvQhuiKUgRtZ5pXu9gjwdK1WJy8FBv2nRh5xET80sws4Zht1wdTkl69PEi0rYzsH70/415UHc6rZKXk5+yEFs2H0cbJXTXRZ79r8ldgKRQtVtj0vIPh6u8Xtkx2ulBbb1okXiWiIuf2S+YNf7INExZvbSrVEInhsuvt0SmjMlah2Y+kYTRjKZ6BPfwrXaDMDJwWmtmMftKTAX3AF24p95kE7PV3qLKlr7bib3QXUShKRb0KM1Em2U93ekvn+5GdQHo4FZjy4l9IzX85HHoE97IoLMIEoE/pEPHp5owgn9Fffpo2SBSdcJPej40OAZUp6TTW0+p2YZ5m7Xr4fMdw0wQY08Hsh2GwjCFOvvqRFh/7VdGvY3enIdVEiwn744sPTQ99LyZAfnuAWeCg3UZso/bIJpttI6tlKceoXJwUoJitDIML62JaFcx4rAuYS9VONsmtqK2c5GhLwg1vVLly9oqsUhOW3V3eWpNl/snk6+B1JcRJwtvdlatmQE6xZCtETCkgtdosMx45M/3jagSSgabe9kCVWywMdR1Tj3vPfhmPVlt3iKzCoCbTEaZnamQ4x/TlsGeFeuCA6LIZyOzZWEkJuBjXv4cCeDIleBB85vK+lqqCG9w/eR5LtK5sWiDlNOMhTzgP9KDTtm2jcPQZ17WLqM1zVqJ2OZYEwyWI17xeVHrh/3QjCBDO7r6+Xmim5FpTEuzafLDKX1EMqxTZyIL/fls5+8/gRuL8zM2/TCdSJzr8Px5zx78OWJjjXeEzzdTZIxYWSHMSI7o9//y6nKxlu5xLugGWywB2OwtgDpZD81RysEfb8B1pJHYB8JSbl+UNgmvq+XDfC9HuX+VLXp4f954X2gzzoKrKZ3LcO+02ZHpT97WR2usYIDfq3ssGmv3POznNnIh2mzkGL1wFCXMvbpcs9hBT87di6FGNiMyqaPvtIpLv6rbhrSAbSnZY8flN3PEUZVtJVNyM+yWX8YEia30F45p9oL8zzdW2BkpdWbNOb2XC6EAO/Nci+oTEE8DgDnTfuMD30Tokg268RRAYxtzGdpSo/GsmFKCh1rj6QxSMc/CPqY9Zv7Ig4epp4VIhkOJ1VXMzCwCk5V41htmvybGZ+8iA0nVdvNCdUJRxDoFvtVou67+k2C55Uzy1yk296UKUUdowAO3BQH3UWkvN+vE2t+2h1xN3jFRsLJ0e75aVHBvIuqhD8NLmr8Xa/AtcujGCjIiJXwT/A= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:22.2373 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e91f9a3-0125-4018-4357-08dd671ca186 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7675 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D8A7114001A X-Stat-Signature: datza4xdywug6sudsotqtridcw39czft X-HE-Tag: 1742412688-186618 X-HE-Meta: U2FsdGVkX1+jLxlmuq/QdJVbc+b5ys1kI+2PPd5ed9WZ9NL9kde2A4xgpNtTX4dJyeoYfMYfCWaPu7dhBKwo4j0Bx7sZGaD96TobLGp7lh3X2c6OJhvT182SIk/hiz6FLNAHkSIVnZYMpD6U2D6etndZUKxaqr4fsv5WIa9y+SysY2BgWwAZSnAE11a9roji7mNv/KG5ijzvokWDukNV9h5WPuYB8buLPT+Kpx0G9fDs5uamdzvotOM8wAjK5wNPXTR0fVlvDU6tBshHORzGuJZesAYLZPmCFGdKJ2kt82zyRAxr/q53dcDn8J9QumzirPIAqLfZfu7HDmHgMISK74ln59qUyHFLdVn2eq+0TfOKiL6qEQWgS/0O4e6TJMxnIkRS+E7zEt6sH2mqRlqf6fnDWpuHDL94mZFD/6COasUjZAghVk5BHSdQIVr6PhlItMeV2UcHe99wmEQDXu3jnIIwxjjHWldz/5gEVUsMGzxVCfIq/RA+3TelpuMz7BlyQpEUU6ke6CODHc+/oiT0dyth+TliPX9kqqs+kFjTL5VREtBEyqBHQezZT6g4jjYA7jltjHFm2cPI1yaFzE/gM8Fl5zoqj7R41O4qwhgwsJbaXq7TDYi5M5TvN3HKQuoy90OYeXff/vmNoAX1JzUw1NUyGEpAAwSnVOHdIKVtar/Wc1CTfS/ZIA4vzub8ochCtjli5WPDMvCj19ax9e/g0PlTo28CFHB0woiZhI/ZunEEob9+/rgucWl8nlGDReebkxJV/T3rsrdBMshoag/2+bbFvplNAsGFJcHRynGZeWxfP9yrDtGGs6f8YSv3tYxC+Ax4edwZW1F60Ytp9x5Xagvbl7Ew/KT/yyKo4Sr95rIAnbr8dfR65e3Bxlp/Wb580iFNFiEm5lmWjIqm04QwGLMG9KpwDVoRt6y/7JKSOz4qcZywCpylU3vdEiXhHdJs58cBjOHoQCsj2/b9hfd n35gN6/A HOXa2H2ygM1h4iDrn/tOLnAjeDrUFJYGoIcCEK7aJjIbwoKB4i47EgpkM7uwUm95r03oEhEIKBGkkl+hDVSP60xje28VQNS2F1tVmuBM4gAMROaOKtEzvcPc3Z5VDpou6MjuMC4JivMjEl9676qGfVsQ/yfgziMgMoNsMXO/UoaTapFOINBBsInmePieHJbUr8T6muRCSkebbYqOlEEhoVfRCfwdjRgpm/r6d2BtNXbf06l0ceUTeDbXl0/mQRd+keUDW5JBl4G44eu8wovcK+ii4AWy3FEJufqonUGrzbZGfigPotvfJaGZDorvkZOIWzEzttNWqvSM81Ea6oD/TrnQp1U/tJz789C+9zhTbBMNJOuxYK45H5Y3zddG3mfkY3eCEm9FTMtV9gzsybHdpuT9AooWYEIYB43gJvpKMujmHRqq3ttP63sEjrtvpnsxmbcIozbr+iINAMQ7OpXa+61oUmSwRdFNXOkoJBZifKqEOdxurxlvlNP0xRp7+/ZLRtM9ab6aF3gw02pvJYpXHUjsNmugFmjk9+hATiUULIjCy2nAo8JIwnFyz9prSxp41E+i+rRoHOfkK1Tgm2RZxx12YCVF0BPl6kMTb/2xb3kPrgpMccBwyePQLjUujDPP9AXloAxQtUZZnnvNjLY9xYbC4Q4cp8tU2UkB34ff2XRPmUbp+Z1ePs8zEILGI1Me32/dL+JzjbhQD5qZ8YigYcEIRY1AJpAWumu2BqoVG4eMuqO1a5n85Ki6RAPjUTMJ/VM+dV6p0YFIBRn2/1MMifNr0753Os1DqiWaTR0O1K/73LhPW+CsMvgzNaSLrQpHJvIgUbYjxhTLORng= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a hook in the fork and exec path to link mm_struct. Reuse the mm_slot infrastructure to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- fs/exec.c | 4 ++ include/linux/kmmscand.h | 30 ++++++++++++++ kernel/fork.c | 4 ++ mm/kmmscand.c | 86 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 include/linux/kmmscand.h diff --git a/fs/exec.c b/fs/exec.c index 506cd411f4ac..e76285e4bc73 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -68,6 +68,7 @@ #include #include #include +#include #include #include @@ -266,6 +267,8 @@ static int __bprm_mm_init(struct linux_binprm *bprm) if (err) goto err_ksm; + kmmscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -288,6 +291,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) return 0; err: ksm_exit(mm); + kmmscand_exit(mm); err_ksm: mmap_write_unlock(mm); err_free: diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h new file mode 100644 index 000000000000..b120c65ee8c6 --- /dev/null +++ b/include/linux/kmmscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KMMSCAND_H_ +#define _LINUX_KMMSCAND_H_ + +#ifdef CONFIG_KMMSCAND +extern void __kmmscand_enter(struct mm_struct *mm); +extern void __kmmscand_exit(struct mm_struct *mm); + +static inline void kmmscand_execve(struct mm_struct *mm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_exit(struct mm_struct *mm) +{ + __kmmscand_exit(mm); +} +#else /* !CONFIG_KMMSCAND */ +static inline void __kmmscand_enter(struct mm_struct *mm) {} +static inline void __kmmscand_exit(struct mm_struct *mm) {} +static inline void kmmscand_execve(struct mm_struct *mm) {} +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) {} +static inline void kmmscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KMMSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 735405a9c5f3..f61c55cf33c2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -656,6 +657,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, mm->exec_vm = oldmm->exec_vm; mm->stack_vm = oldmm->stack_vm; + kmmscand_fork(mm, oldmm); + /* Use __mt_dup() to efficiently build an identical maple tree. */ retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) @@ -1353,6 +1356,7 @@ static inline void __mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ + kmmscand_exit(mm); exit_mmap(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 6c55250b5cfb..36d0fea31dea 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -7,13 +7,14 @@ #include #include #include +#include #include #include #include #include #include "internal.h" - +#include "mm_slot.h" static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); @@ -30,10 +31,21 @@ static bool need_wakeup; static unsigned long kmmscand_sleep_expire; +static DEFINE_SPINLOCK(kmmscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); +#define KMMSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HASH_BITS); + +static struct kmem_cache *kmmscand_slot_cache __read_mostly; + +struct kmmscand_mm_slot { + struct mm_slot slot; +}; + struct kmmscand_scan { struct list_head mm_head; + struct kmmscand_mm_slot *mm_slot; }; struct kmmscand_scan kmmscand_scan = { @@ -70,6 +82,11 @@ static void kmmscand_wait_work(void) return; } +static inline int kmmscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) == 0; +} + static unsigned long kmmscand_scan_mm_slot(void) { /* placeholder for scanning */ @@ -117,6 +134,65 @@ static int kmmscand(void *none) return 0; } +static inline void kmmscand_destroy(void) +{ + kmem_cache_destroy(kmmscand_slot_cache); +} + +void __kmmscand_enter(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *kmmscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kmmscand_exit() must not run from under us */ + VM_BUG_ON_MM(kmmscand_test_exit(mm), mm); + + kmmscand_slot = mm_slot_alloc(kmmscand_slot_cache); + + if (!kmmscand_slot) + return; + + slot = &kmmscand_slot->slot; + + spin_lock(&kmmscand_mm_lock); + mm_slot_insert(kmmscand_slots_hash, mm, slot); + + wakeup = list_empty(&kmmscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kmmscand_scan.mm_head); + spin_unlock(&kmmscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kmmscand_wait); +} + +void __kmmscand_exit(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free = 0; + + spin_lock(&kmmscand_mm_lock); + slot = mm_slot_lookup(kmmscand_slots_hash, mm); + mm_slot = mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + if (mm_slot && kmmscand_scan.mm_slot != mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free = 1; + } + + spin_unlock(&kmmscand_mm_lock); + + if (free) { + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kmmscand(void) { int err = 0; @@ -162,6 +238,13 @@ static int __init kmmscand_init(void) { int err; + kmmscand_slot_cache = KMEM_CACHE(kmmscand_mm_slot, 0); + + if (!kmmscand_slot_cache) { + pr_err("kmmscand: kmem_cache error"); + return -ENOMEM; + } + err = start_kmmscand(); if (err) goto err_kmmscand; @@ -170,6 +253,7 @@ static int __init kmmscand_init(void) err_kmmscand: stop_kmmscand(); + kmmscand_destroy(); return err; }