From patchwork Wed Jun 3 14:11:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11585735 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F0D7618 for ; Wed, 3 Jun 2020 14:15:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1E3C420738 for ; Wed, 3 Jun 2020 14:15:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="SqJqL3dS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E3C420738 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 41F2A8E0007; Wed, 3 Jun 2020 10:15:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3CFD78E0006; Wed, 3 Jun 2020 10:15:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BE228E0007; Wed, 3 Jun 2020 10:15:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id 1521A8E0006 for ; Wed, 3 Jun 2020 10:15:29 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C8834180ACF62 for ; Wed, 3 Jun 2020 14:15:28 +0000 (UTC) X-FDA: 76888098336.29.blade10_697a35ba34531 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 4759B18082828 for ; Wed, 3 Jun 2020 14:15:25 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=416e8e2f5=sjpark@amazon.com,,RULES_HIT:30003:30012:30054:30064:30070,0,RBL:52.95.49.90:@amazon.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: blade10_697a35ba34531 X-Filterd-Recvd-Size: 10538 Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Jun 2020 14:15:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1591193724; x=1622729724; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=faje2xiXtBmZmE30JnFGMsjEdqz81v/UC5sZtQepIaM=; b=SqJqL3dS7qZ+2FbKKBIhg9/xmo6Ior8qaStnOHAuxu5mgzv2Itus3yLX EiukerZMuNFU6pS/4m4Kv+zWnQDHcU7MEOGpKhAnF+flQbwDFp+x8smEx mEuAWAcm5VlQ8JipOjHBCRVY3aQfTl7Sz+qq+mAEjDs5F+TpjYWxPpNmy E=; IronPort-SDR: 6ctg4vvr34L9vkWItNnSgSIDRGwYm4wKTCYNXC5cMzWDUXISMusWumRj3NY3IxwqNwOseav9XI x/T10QSfLJuA== X-IronPort-AV: E=Sophos;i="5.73,467,1583193600"; d="scan'208";a="34144210" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 03 Jun 2020 14:15:09 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com (Postfix) with ESMTPS id A775BA2685; Wed, 3 Jun 2020 14:14:58 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 3 Jun 2020 14:14:57 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.90) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 3 Jun 2020 14:14:41 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC v2 7/9] mm/damon: Implement callbacks for physical memory monitoring Date: Wed, 3 Jun 2020 16:11:33 +0200 Message-ID: <20200603141135.10575-8-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200603141135.10575-1-sjpark@amazon.com> References: <20200603141135.10575-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D45UWB002.ant.amazon.com (10.43.161.78) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Rspamd-Queue-Id: 4759B18082828 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit implements the four callbacks (->init_target_regions, ->update_target_regions, ->prepare_access_check, and ->check_accesses) for the basic access monitoring of the physical memory address space. By setting the callback pointers to point those, users can easily monitor the accesses to the physical memory. Internally, it uses the PTE Accessed bit, as similar to that of the virtual memory support. Also, it supports only page frames that supported by idle page tracking. Acutally, most of the code is stollen from idle page tracking. Users who want to use other access check primitives and monitor the frames that not supported with this implementation could implement their own callbacks on their own. Signed-off-by: SeongJae Park --- include/linux/damon.h | 5 ++ mm/damon.c | 184 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 189 insertions(+) diff --git a/include/linux/damon.h b/include/linux/damon.h index 1a788bfd1b4e..f96503a532ea 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -216,6 +216,11 @@ void kdamond_update_vm_regions(struct damon_ctx *ctx); void kdamond_prepare_vm_access_checks(struct damon_ctx *ctx); unsigned int kdamond_check_vm_accesses(struct damon_ctx *ctx); +void kdamond_init_phys_regions(struct damon_ctx *ctx); +void kdamond_update_phys_regions(struct damon_ctx *ctx); +void kdamond_prepare_phys_access_checks(struct damon_ctx *ctx); +unsigned int kdamond_check_phys_accesses(struct damon_ctx *ctx); + int damon_set_pids(struct damon_ctx *ctx, int *pids, ssize_t nr_pids); int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, unsigned long aggr_int, unsigned long regions_update_int, diff --git a/mm/damon.c b/mm/damon.c index f5cbc97a3bbc..6a5c6d540580 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -19,7 +19,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -480,6 +482,11 @@ void kdamond_init_vm_regions(struct damon_ctx *ctx) } } +/* Do nothing. Users should set the initial regions by themselves */ +void kdamond_init_phys_regions(struct damon_ctx *ctx) +{ +} + static void damon_mkold(struct mm_struct *mm, unsigned long addr) { pte_t *pte = NULL; @@ -611,6 +618,178 @@ unsigned int kdamond_check_vm_accesses(struct damon_ctx *ctx) return max_nr_accesses; } +/* access check functions for physical address based regions */ + +/* This code is stollen from page_idle.c */ +static struct page *damon_phys_get_page(unsigned long pfn) +{ + struct page *page; + pg_data_t *pgdat; + + if (!pfn_valid(pfn)) + return NULL; + + page = pfn_to_page(pfn); + if (!page || !PageLRU(page) || + !get_page_unless_zero(page)) + return NULL; + + pgdat = page_pgdat(page); + spin_lock_irq(&pgdat->lru_lock); + if (unlikely(!PageLRU(page))) { + put_page(page); + page = NULL; + } + spin_unlock_irq(&pgdat->lru_lock); + return page; +} + +static bool damon_page_mkold(struct page *page, struct vm_area_struct *vma, + unsigned long addr, void *arg) +{ + damon_mkold(vma->vm_mm, addr); + return true; +} + +static void damon_phys_mkold(unsigned long paddr) +{ + struct page *page = damon_phys_get_page(PHYS_PFN(paddr)); + struct rmap_walk_control rwc = { + .rmap_one = damon_page_mkold, + .anon_lock = page_lock_anon_vma_read, + }; + bool need_lock; + + if (!page) + return; + + if (!page_mapped(page) || !page_rmapping(page)) + return; + + need_lock = !PageAnon(page) || PageKsm(page); + if (need_lock && !trylock_page(page)) + return; + + rmap_walk(page, &rwc); + + if (need_lock) + unlock_page(page); + put_page(page); +} + +static void damon_prepare_phys_access_check(struct damon_ctx *ctx, + struct damon_region *r) +{ + r->sampling_addr = damon_rand(r->ar.start, r->ar.end); + + damon_phys_mkold(r->sampling_addr); +} + +void kdamond_prepare_phys_access_checks(struct damon_ctx *ctx) +{ + struct damon_task *t; + struct damon_region *r; + + damon_for_each_task(t, ctx) { + damon_for_each_region(r, t) + damon_prepare_phys_access_check(ctx, r); + } +} + +struct damon_phys_access_chk_result { + unsigned long page_sz; + bool accessed; +}; + +static bool damon_page_accessed(struct page *page, struct vm_area_struct *vma, + unsigned long addr, void *arg) +{ + struct damon_phys_access_chk_result *result = arg; + + result->accessed = damon_young(vma->vm_mm, addr, &result->page_sz); + + /* If accessed, stop walking */ + return !result->accessed; +} + +static bool damon_phys_young(unsigned long paddr, unsigned long *page_sz) +{ + struct page *page = damon_phys_get_page(PHYS_PFN(paddr)); + struct damon_phys_access_chk_result result = { + .page_sz = PAGE_SIZE, + .accessed = false, + }; + struct rmap_walk_control rwc = { + .arg = &result, + .rmap_one = damon_page_accessed, + .anon_lock = page_lock_anon_vma_read, + }; + bool need_lock; + + if (!page) + return false; + + if (!page_mapped(page) || !page_rmapping(page)) + return false; + + need_lock = !PageAnon(page) || PageKsm(page); + if (need_lock && !trylock_page(page)) + return false; + + rmap_walk(page, &rwc); + + if (need_lock) + unlock_page(page); + put_page(page); + + *page_sz = result.page_sz; + return result.accessed; +} + +/* + * Check whether the region was accessed after the last preparation + * + * mm 'mm_struct' for the given virtual address space + * r the region of physical address space that needs to be checked + */ +static void damon_check_phys_access(struct damon_ctx *ctx, + struct damon_region *r) +{ + static unsigned long last_addr; + static unsigned long last_page_sz = PAGE_SIZE; + static bool last_accessed; + + /* If the region is in the last checked page, reuse the result */ + if (ALIGN_DOWN(last_addr, last_page_sz) == + ALIGN_DOWN(r->sampling_addr, last_page_sz)) { + if (last_accessed) + r->nr_accesses++; + return; + } + + last_accessed = damon_phys_young(r->sampling_addr, &last_page_sz); + if (last_accessed) + r->nr_accesses++; + + last_addr = r->sampling_addr; +} + +unsigned int kdamond_check_phys_accesses(struct damon_ctx *ctx) +{ + struct damon_task *t; + struct damon_region *r; + unsigned int max_nr_accesses = 0; + + damon_for_each_task(t, ctx) { + damon_for_each_region(r, t) { + damon_check_phys_access(ctx, r); + max_nr_accesses = max(r->nr_accesses, max_nr_accesses); + } + } + + return max_nr_accesses; +} + /* * damon_check_reset_time_interval() - Check if a time interval is elapsed. * @baseline: the time to check whether the interval has elapsed since @@ -1058,6 +1237,11 @@ void kdamond_update_vm_regions(struct damon_ctx *ctx) } } +/* Do nothing. If necessary, users should update regions in other callbacks */ +void kdamond_update_phys_regions(struct damon_ctx *ctx) +{ +} + /* * Check whether current monitoring should be stopped *