From patchwork Tue Jun 16 14:08:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11607717 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1460892A for ; Tue, 16 Jun 2020 14:12:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C9A5020707 for ; Tue, 16 Jun 2020 14:12:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="A3CBr2E8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C9A5020707 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F01BA8D000D; Tue, 16 Jun 2020 10:12:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ED87A8D0006; Tue, 16 Jun 2020 10:12:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC76A8D000D; Tue, 16 Jun 2020 10:12:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id C62298D0006 for ; Tue, 16 Jun 2020 10:12:11 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 821DC801E779 for ; Tue, 16 Jun 2020 14:12:11 +0000 (UTC) X-FDA: 76935264462.17.wave00_25142b026dff Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 482071805B2C9 for ; Tue, 16 Jun 2020 14:11:12 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=429833f5f=sjpark@amazon.com,,RULES_HIT:30003:30012:30051:30054:30064:30070,0,RBL:52.95.49.90:@amazon.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: wave00_25142b026dff X-Filterd-Recvd-Size: 11390 Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Tue, 16 Jun 2020 14:11:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1592316672; x=1623852672; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=uuuAHwsA0agsAV11LV6hPkkJjfjwCTMrcn4kO5fu174=; b=A3CBr2E8tEG0OuVHUmScsTmNW2hF2neZW/26QwtYlc1d2u33DmW6ehdV teUazh/B8/iG0Zd36Wmgp6BfSqbPeMixtycJB0lRX7WWJUlZ50dNG6f/U CjiRfiCa1+aOjEuYcG4q+b6aWMfu5P7rcJ7SJZdc314f8JM/NlzvjwpLm I=; IronPort-SDR: q3pM3I96QRx5llJ88veRcQR89iKQAGSfZfyjSimy16PvbPBoRRyEF0QknRKScKrK7jcFjFz1i3 EVvU6o/aVUsw== X-IronPort-AV: E=Sophos;i="5.73,518,1583193600"; d="scan'208";a="36497058" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-e7be2041.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 16 Jun 2020 14:11:09 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-e7be2041.us-west-2.amazon.com (Postfix) with ESMTPS id 07937A2277; Tue, 16 Jun 2020 14:11:05 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 16 Jun 2020 14:11:05 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.162.109) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 16 Jun 2020 14:10:44 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC v4 5/8] mm/damon: Implement callbacks for physical memory monitoring Date: Tue, 16 Jun 2020 16:08:10 +0200 Message-ID: <20200616140813.17863-6-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200616140813.17863-1-sjpark@amazon.com> References: <20200616140813.17863-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.109] X-ClientProxiedBy: EX13D01UWB004.ant.amazon.com (10.43.161.157) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Rspamd-Queue-Id: 482071805B2C9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit implements the four callbacks (->init_target_regions, ->update_target_regions, ->prepare_access_check, and ->check_accesses) for the basic access monitoring of the physical memory address space. By setting the callback pointers to point those, users can easily monitor the accesses to the physical memory. Internally, it uses the PTE Accessed bit, as similar to that of the virtual memory support. Also, it supports only user memory pages, as idle page tracking also does, for the same reason. If the monitoring target physical memory address range contains non-user memory pages, access check of the pages will do nothing but simply treat the pages as not accessed. Users who want to use other access check primitives and/or monitor the non-user memory regions could implement and use their own callbacks. Signed-off-by: SeongJae Park --- include/linux/damon.h | 5 ++ mm/damon.c | 201 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 206 insertions(+) diff --git a/include/linux/damon.h b/include/linux/damon.h index 076852bab7aa..6c0e9bb35a1f 100644 --- a/include/linux/damon.h +++ b/include/linux/damon.h @@ -227,6 +227,11 @@ void kdamond_update_vm_regions(struct damon_ctx *ctx); void kdamond_prepare_vm_access_checks(struct damon_ctx *ctx); unsigned int kdamond_check_vm_accesses(struct damon_ctx *ctx); +void kdamond_init_phys_regions(struct damon_ctx *ctx); +void kdamond_update_phys_regions(struct damon_ctx *ctx); +void kdamond_prepare_phys_access_checks(struct damon_ctx *ctx); +unsigned int kdamond_check_phys_accesses(struct damon_ctx *ctx); + int damon_set_pids(struct damon_ctx *ctx, int *pids, ssize_t nr_pids); int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, unsigned long aggr_int, unsigned long regions_update_int, diff --git a/mm/damon.c b/mm/damon.c index ab115db1f20c..fdf3425befb2 100644 --- a/mm/damon.c +++ b/mm/damon.c @@ -27,10 +27,13 @@ #include #include #include +#include #include #include #include +#include #include +#include #include #include #include @@ -534,6 +537,18 @@ void kdamond_init_vm_regions(struct damon_ctx *ctx) } } +/* + * The initial regions construction function for the physical address space. + * + * This default version does nothing in actual. Users should set the initial + * regions by themselves before passing their damon_ctx to 'start_damon()', or + * implement their version of this and set '->init_target_regions' of their + * damon_ctx to point it. + */ +void kdamond_init_phys_regions(struct damon_ctx *ctx) +{ +} + /* * Functions for the dynamic monitoring target regions update */ @@ -617,6 +632,19 @@ void kdamond_update_vm_regions(struct damon_ctx *ctx) } } +/* + * The dynamic monitoring target regions update function for the physical + * address space. + * + * This default version does nothing in actual. Users should update the + * regions in other callbacks such as '->aggregate_cb', or implement their + * version of this and set the '->init_target_regions' of their damon_ctx to + * point it. + */ +void kdamond_update_phys_regions(struct damon_ctx *ctx) +{ +} + /* * Functions for the access checking of the regions */ @@ -752,6 +780,179 @@ unsigned int kdamond_check_vm_accesses(struct damon_ctx *ctx) return max_nr_accesses; } +/* access check functions for physical address based regions */ + +/* + * Get a page by pfn if it is in the LRU list. Otherwise, returns NULL. + * + * The body of this function is stollen from the 'page_idle_get_page()'. We + * steal rather than reuse it because the code is quite simple . + */ +static struct page *damon_phys_get_page(unsigned long pfn) +{ + struct page *page = pfn_to_online_page(pfn); + pg_data_t *pgdat; + + if (!page || !PageLRU(page) || + !get_page_unless_zero(page)) + return NULL; + + pgdat = page_pgdat(page); + spin_lock_irq(&pgdat->lru_lock); + if (unlikely(!PageLRU(page))) { + put_page(page); + page = NULL; + } + spin_unlock_irq(&pgdat->lru_lock); + return page; +} + +static bool damon_page_mkold(struct page *page, struct vm_area_struct *vma, + unsigned long addr, void *arg) +{ + damon_mkold(vma->vm_mm, addr); + return true; +} + +static void damon_phys_mkold(unsigned long paddr) +{ + struct page *page = damon_phys_get_page(PHYS_PFN(paddr)); + struct rmap_walk_control rwc = { + .rmap_one = damon_page_mkold, + .anon_lock = page_lock_anon_vma_read, + }; + bool need_lock; + + if (!page) + return; + + if (!page_mapped(page) || !page_rmapping(page)) + return; + + need_lock = !PageAnon(page) || PageKsm(page); + if (need_lock && !trylock_page(page)) + return; + + rmap_walk(page, &rwc); + + if (need_lock) + unlock_page(page); + put_page(page); +} + +static void damon_prepare_phys_access_check(struct damon_ctx *ctx, + struct damon_region *r) +{ + r->sampling_addr = damon_rand(r->ar.start, r->ar.end); + + damon_phys_mkold(r->sampling_addr); +} + +void kdamond_prepare_phys_access_checks(struct damon_ctx *ctx) +{ + struct damon_task *t; + struct damon_region *r; + + damon_for_each_task(t, ctx) { + damon_for_each_region(r, t) + damon_prepare_phys_access_check(ctx, r); + } +} + +struct damon_phys_access_chk_result { + unsigned long page_sz; + bool accessed; +}; + +static bool damon_page_accessed(struct page *page, struct vm_area_struct *vma, + unsigned long addr, void *arg) +{ + struct damon_phys_access_chk_result *result = arg; + + result->accessed = damon_young(vma->vm_mm, addr, &result->page_sz); + + /* If accessed, stop walking */ + return !result->accessed; +} + +static bool damon_phys_young(unsigned long paddr, unsigned long *page_sz) +{ + struct page *page = damon_phys_get_page(PHYS_PFN(paddr)); + struct damon_phys_access_chk_result result = { + .page_sz = PAGE_SIZE, + .accessed = false, + }; + struct rmap_walk_control rwc = { + .arg = &result, + .rmap_one = damon_page_accessed, + .anon_lock = page_lock_anon_vma_read, + }; + bool need_lock; + + if (!page) + return false; + + if (!page_mapped(page) || !page_rmapping(page)) + return false; + + need_lock = !PageAnon(page) || PageKsm(page); + if (need_lock && !trylock_page(page)) + return false; + + rmap_walk(page, &rwc); + + if (need_lock) + unlock_page(page); + put_page(page); + + *page_sz = result.page_sz; + return result.accessed; +} + +/* + * Check whether the region was accessed after the last preparation + * + * mm 'mm_struct' for the given virtual address space + * r the region of physical address space that needs to be checked + */ +static void damon_check_phys_access(struct damon_ctx *ctx, + struct damon_region *r) +{ + static unsigned long last_addr; + static unsigned long last_page_sz = PAGE_SIZE; + static bool last_accessed; + + /* If the region is in the last checked page, reuse the result */ + if (ALIGN_DOWN(last_addr, last_page_sz) == + ALIGN_DOWN(r->sampling_addr, last_page_sz)) { + if (last_accessed) + r->nr_accesses++; + return; + } + + last_accessed = damon_phys_young(r->sampling_addr, &last_page_sz); + if (last_accessed) + r->nr_accesses++; + + last_addr = r->sampling_addr; +} + +unsigned int kdamond_check_phys_accesses(struct damon_ctx *ctx) +{ + struct damon_task *t; + struct damon_region *r; + unsigned int max_nr_accesses = 0; + + damon_for_each_task(t, ctx) { + damon_for_each_region(r, t) { + damon_check_phys_access(ctx, r); + max_nr_accesses = max(r->nr_accesses, max_nr_accesses); + } + } + + return max_nr_accesses; +} + /* * Functions for DAMON core logics and features */