From patchwork Tue May 21 20:07:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954451 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E1A414C0 for ; Tue, 21 May 2019 20:19:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BB3C28AFC for ; Tue, 21 May 2019 20:19:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E34828AFF; Tue, 21 May 2019 20:19:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2F2628AFC for ; Tue, 21 May 2019 20:19:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD7196B0006; Tue, 21 May 2019 16:19:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B875D6B0007; Tue, 21 May 2019 16:19:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A508E6B0008; Tue, 21 May 2019 16:19:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 6C8426B0006 for ; Tue, 21 May 2019 16:19:19 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id b3so48927pgt.12 for ; Tue, 21 May 2019 13:19:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=Gl8pp3KYsdZV699fGKd7n8I36sqXFKD3Pr1ctINiCIU=; b=td/3B4AouEVoLpMyjT8PDgoosgpW6A9zLVvz/gUth0te+0Whl9xbhG9MDCiW5OzoEQ yn2Y4RkpIR2nn89NWpIdHX9Nn1sZM3lgwEMwcZY892rYFrP81Zg2zWXGEYv49hRRXVmD GnHwOwf5ic+JLGzOiADpTcEQ2tz/l/lXoiEYHo8XfgBB0mFq9sD6W68HQ8k/JAZCYGG9 FOZm8dsnJrLhRqbfCiygoODGpYJnw3fqRnpsU6fQzmgSFF1MqX/ZXwmhU3oqRpB4UTGL X5GbxHPmswuWwdy55JuNlNGTlXV3rok91xKZWTat7KXGMs0GT5vU0sRFcH1pTLVhkleN 3kRw== X-Gm-Message-State: APjAAAW1UR7xqm+ifUByVbVYjgyVQvtcACRlpftxNAMLX2PhS/PaYC9Z gQuQ6oZ+JUr6QVkc5JdUZ4jA6SBj/ZH6kuCj4e1BIGOHLrib/pifJY9xRpKAGrbOiPecCx6OVwh FiyCWxjepErGnJWvaTeXYMX7wOdqdtIN3G5wRNZITHxs8I8swsDMcHIc+9S/GuB9FRw== X-Received: by 2002:a17:902:e18d:: with SMTP id cd13mr17846296plb.301.1558469958965; Tue, 21 May 2019 13:19:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqy3MBxDC2YkJ1XSDqaDLAr2fb+mv92epFDWEWpN3ttlbvG0esw6mC3jJO8YNEUkpvN1Ujts X-Received: by 2002:a17:902:e18d:: with SMTP id cd13mr17846244plb.301.1558469958211; Tue, 21 May 2019 13:19:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558469958; cv=none; d=google.com; s=arc-20160816; b=UtmF8hynVfqXURdTUXG1atxDLvQIBokmYUlJG4NjgFqPIOhPZlOnSI+JkBi91gfyI/ 9zWkQkiUWiD2LV52pq2Al3DTC/SvlGFGCIZm0UjlEYb6QdwSx5e75djpxVIPCw/G8qNR xjAy6IUpbGJWJxxv0+o5LKkU2bpqBBykCXZ2PuupK+86QOfQSNfpVeypKNCArWt9ioWi nAQb8jRoPum+KY/MbKc/hCUMWXfN7V3PR6DHd3GH9S1sjOhM4iCAouu8572I5eH+ZNAw 4LZqHSHzv0rjhAOQB2Uf9JWB+HD0E22Icoob7VlONxwm96ZTKwwyzwihOfOMHlx8l2rc JUow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=Gl8pp3KYsdZV699fGKd7n8I36sqXFKD3Pr1ctINiCIU=; b=lqkCeNPGjf9Ax5aQkcYv8EFf+NtgP4d1YxeAfIsNJrNaT4ObMEoknay4Max1uPXKux W8p+4ll9Tp5yrbVFfZu7fthcy28KLM1Y6e+7r4k1i/IqDSS+9LUO8/Q257D5L7bTIusG VAHm1iUEzVQE+5RAcS9qUX0s+0V6NvEIo/xgFO7vqQCba9WBQuREEIdoDonpp7nh86qB EAkgjkxKhRb2kWfvURnCJAqwV/xZ42cAfKKVvyHHi+3KcprlbMFA4cS9+4LOmew32bpO /7j09JB0ZMFOuc/jfAU5iy+pN9bc8MmAgdQWDM44BRBSxBMfx+JY29jR1pL5OnEpPx7a yvug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=mdTCJovY; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id l21si22780245pgc.190.2019.05.21.13.19.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:19:18 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=mdTCJovY; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKIq3m032264 for ; Tue, 21 May 2019 13:19:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=Gl8pp3KYsdZV699fGKd7n8I36sqXFKD3Pr1ctINiCIU=; b=mdTCJovYDY8s62vH1SNfYC0cOM+pHXoPII8bODX/C8DtB2YTVnWgJlyd5giamFTvuNJ7 cq3TjfATua+VjkXWpQkBPQXT2AmynX5+8btcc9igt9sPjC9BcoHnnd+K8q857rub/yVY eYQ3tCJB5DCkeFWW+1CeYCImw/DSi8IAReU= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2smhav1qa6-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 21 May 2019 13:19:17 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 21 May 2019 13:18:47 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 0071E1245FFA1; Tue, 21 May 2019 13:07:49 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 1/7] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Date: Tue, 21 May 2019 13:07:29 -0700 Message-ID: <20190521200735.2603003-2-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Initialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params(). Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting. For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vladimir Davydov Acked-by: Johannes Weiner --- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 2915d912e89a..f6eff59e018e 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1268,7 +1268,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); + memcg_link_cache(kmem_cache, NULL); slab_state = PARTIAL; /* diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..6a562ca72bca 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -289,7 +289,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); +extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, void (*deact_fn)(struct kmem_cache *)); @@ -344,7 +344,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache(struct kmem_cache *s) +static inline void memcg_link_cache(struct kmem_cache *s, + struct mem_cgroup *memcg) { } diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..6e00bdf8618d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) } static int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct memcg_cache_array *arr; if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); return 0; @@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs) return ret; } -void memcg_link_cache(struct kmem_cache *s) +void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) { if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } #else static inline int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { return 0; } @@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, memcg, root_cache); + err = init_memcg_params(s, root_cache); if (err) goto out_free_cache; @@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, memcg); out: if (err) return ERR_PTR(err); @@ -997,7 +997,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); s->refcount = 1; return s; } diff --git a/mm/slub.c b/mm/slub.c index cd04dbd2b5d0..c5646cb02055 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4215,7 +4215,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); return s; } From patchwork Tue May 21 20:07:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954453 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 712DB14C0 for ; Tue, 21 May 2019 20:19:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E87F28AFC for ; Tue, 21 May 2019 20:19:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 52C8B28AFF; Tue, 21 May 2019 20:19:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8837F28AFC for ; Tue, 21 May 2019 20:19:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B3306B0007; Tue, 21 May 2019 16:19:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 964BF6B0008; Tue, 21 May 2019 16:19:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82B786B000A; Tue, 21 May 2019 16:19:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id 5F8896B0007 for ; Tue, 21 May 2019 16:19:31 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id o1so18457496ybk.17 for ; Tue, 21 May 2019 13:19:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=Cazplb3chHwkBW5FsrtHWIy87NqExlQCyzIp9ZVAnzo=; b=fuFZfusg6PCboil79BOOqmDSgr41E/T0MPBk+I0fxoAUJOOR/9qmo9UXPftaSn47bT Cfhdf3oaOoRnthQRV04xf8wOR1DAwjeA5L+yFqw2fKcK1Vkk5Kkg4sJmLyPLv9BU34Tp R5QAkbR5vn1xmxcg7yU3smfPEScRrJgrH6e3VvaidGodu8YxTyvBZ6YBnK36xzT0alpe js+3cAh+dYVnJfN/Q+kIrVWooSAmEoB1+cFComtU0zq/zlAXQ4lfWMQa49PjNLFafpd+ 0L/rNgZ1SQk13qla42gYYMcfNUG0v6yIdFp/t+I0PGQwsqwuxrhdsjN15VNCKYpdCrCc d4CQ== X-Gm-Message-State: APjAAAXOwyD41Iywl0em4os4sEgRErVgddP4Qnh4mCnTsSLKPYYMLkZD CE8pOLCdgJF8FUYMTmnWPmsgGCBpv2I3YooIKAwQ/dtu436RhxzCwI+XgMGUKy9U06ZxvgudalK yxpIh5q6AOTUBz5G5tosD2AxXKR82SLsBTL13kw+ZralLKS+LVbXOD0bvrHql+H6mrg== X-Received: by 2002:a81:5e84:: with SMTP id s126mr18735079ywb.32.1558469971083; Tue, 21 May 2019 13:19:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqyUKDj4efStZIArMtlCGWiuC1efUIRTR9ebenYcTUPUdI3nA2bC8ZpTnauUe1ia4UEY4iev X-Received: by 2002:a81:5e84:: with SMTP id s126mr18735046ywb.32.1558469970198; Tue, 21 May 2019 13:19:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558469970; cv=none; d=google.com; s=arc-20160816; b=fJtLgZJRcMv38Fjh7jei0kv5Q7AIgCGgR7bhwcO3x0kE9Dtl5VPaPCZasOyb6/9BbF GxZNk4Zq5p26hduINgienkFeYV+dIiQmhky+fzlISGRjAQBspo5TXaHmsUUfB7IdXWWD t+QJ++dQIkSx5/aB5oTkoLU+0pSxbVP7xZfdL5Ca7Pfh5AaxBSl7aKnfQy94jUxVaiAo 3G3tWzIFgdTIpTfhWH0a95bKxXYGC0RDbyhynVHW1As7hFgst+ETYJj6Hz2YnxsBYkk8 zS5Tkb2/GjFYDGZf52AklRQ008wNTBk+tGc+oVMHJibc1bUlaVRMvqE3xvVROuMmVo0q C4bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=Cazplb3chHwkBW5FsrtHWIy87NqExlQCyzIp9ZVAnzo=; b=ZSrkD5wSXFkCEi2Nx9u78FwGlCpst8d7B3z71o0uU0p3ZqttuUVqga/PoOBYJAwl3U i7tokgVQnJr3a2SWoDe5+izOw47wk7mqy93wo9dK+VCWcnkKNRd5hYv6FoKfBomMUVBQ fnE2My2rGao8+CmmhAbVMyiwdlxfqhpa1hd08f/WtWlElEdeDQi2uM1Yj7YjTo8lJUi9 EFMqvEZuIx3CBfivmtOA0GCMZRZeDpqn34qJ5IxkAiacE47/t0KMPb3/IKyQYB5uFRnw Mv5UxCE4rTwCI5BjsdMtnCcURzymLqZJvceblxwiq7MInoU+rrEOIWxj6lEMjJmiqEhQ 96cg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Yf9rvjcb; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id c127si6282198ybf.122.2019.05.21.13.19.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:19:30 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Yf9rvjcb; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKIMRc014421 for ; Tue, 21 May 2019 13:19:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=Cazplb3chHwkBW5FsrtHWIy87NqExlQCyzIp9ZVAnzo=; b=Yf9rvjcbnbjlBXdy+b72aQzmGWIn67qtqGCs3lUHJGm38/z/QE7yRXkpyYIzhylJu1nC OEBYnTGR/cMO+meQaJptak2glEfZFvCTAtfxY/8IPY67EPK9Cu+FFiSAnAndcLnGmSG6 zhvDPCudfolKSbQiumTqhSMj5V7p44RAQgw= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2smr4jr109-17 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 21 May 2019 13:19:30 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::127) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 21 May 2019 13:18:47 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 0430B1245FFA3; Tue, 21 May 2019 13:07:50 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 2/7] mm: generalize postponed non-root kmem_cache deactivation Date: Tue, 21 May 2019 13:07:30 -0700 Message-ID: <20190521200735.2603003-3-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches reparenting, but requires some generalization. Let's decouple all infrastructure (rcu callback, work callback) from the SLUB-specific code, so it can be used with SLAB as well. Also, let's rename some functions to make the code look simpler. All SLAB/SLUB-specific functions start with "__". Remove "deact_" prefix from the corresponding struct fields. Here is the graph of a new calling scheme: kmemcg_cache_deactivate() __kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_schedule_work_after_rcu() rcu kmemcg_after_rcu_workfn() work kmemcg_cache_deactivate_after_rcu() __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specific instead of: __kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_deactivate_rcufn SLUB-only, rcu kmemcg_deactivate_workfn SLUB-only, work kmemcg_cache_deact_after_rcu() SLUB-only Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/slab.h | 6 ++--- mm/slab.c | 4 +++ mm/slab.h | 3 ++- mm/slab_common.c | 62 ++++++++++++++++++++------------------------ mm/slub.c | 8 +----- 5 files changed, 38 insertions(+), 45 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..47923c173f30 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -642,10 +642,10 @@ struct memcg_cache_params { struct list_head children_node; struct list_head kmem_caches_node; - void (*deact_fn)(struct kmem_cache *); + void (*work_fn)(struct kmem_cache *); union { - struct rcu_head deact_rcu_head; - struct work_struct deact_work; + struct rcu_head rcu_head; + struct work_struct work; }; }; }; diff --git a/mm/slab.c b/mm/slab.c index f6eff59e018e..83000e46b870 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2281,6 +2281,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) { __kmem_cache_shrink(cachep); } + +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ +} #endif int __kmem_cache_shutdown(struct kmem_cache *cachep) diff --git a/mm/slab.h b/mm/slab.h index 6a562ca72bca..4a261c97c138 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -291,7 +292,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)); + void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 6e00bdf8618d..4e5b4292a763 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -691,17 +691,18 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, put_online_cpus(); } -static void kmemcg_deactivate_workfn(struct work_struct *work) +static void kmemcg_after_rcu_workfn(struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work); get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - s->memcg_params.deact_fn(s); + s->memcg_params.work_fn(s); + s->memcg_params.work_fn = NULL; mutex_unlock(&slab_mutex); @@ -712,37 +713,28 @@ static void kmemcg_deactivate_workfn(struct work_struct *work) css_put(&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn(struct rcu_head *head) +/* + * We need to grab blocking locks. Bounce to ->work. The + * work item shares the space with the RCU head and can't be + * initialized eariler. +*/ +static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head); - /* - * We need to grab blocking locks. Bounce to ->deact_work. The - * work item shares the space with the RCU head and can't be - * initialized eariler. - */ - INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work); + INIT_WORK(&s->memcg_params.work, kmemcg_after_rcu_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } -/** - * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a - * sched RCU grace period - * @s: target kmem_cache - * @deact_fn: deactivation function to call - * - * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex - * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @deact_fn is finished. This is to be used from - * __kmemcg_cache_deactivate(). - */ -void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { - if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.deact_fn)) - return; + __kmemcg_cache_deactivate_after_rcu(s); +} + +static void kmemcg_cache_deactivate(struct kmem_cache *s) +{ + __kmemcg_cache_deactivate(s); if (s->memcg_params.root_cache->memcg_params.dying) return; @@ -750,8 +742,9 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; - call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + WARN_ON_ONCE(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) @@ -773,7 +766,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) if (!c) continue; - __kmemcg_cache_deactivate(c); + kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } mutex_unlock(&slab_mutex); @@ -866,11 +859,12 @@ static void flush_memcg_workqueue(struct kmem_cache *s) mutex_unlock(&slab_mutex); /* - * SLUB deactivates the kmem_caches through call_rcu. Make + * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ - if (IS_ENABLED(CONFIG_SLUB)) - rcu_barrier(); +#ifndef CONFIG_SLOB + rcu_barrier(); +#endif /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB diff --git a/mm/slub.c b/mm/slub.c index c5646cb02055..8abd2d2a4ae4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4024,7 +4024,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) } #ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. @@ -4050,12 +4050,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s) */ slub_set_cpu_partial(s, 0); s->min_partial = 0; - - /* - * s->cpu_partial is checked locklessly (see put_cpu_partial), so - * we have to make sure the change is visible before shrinking. - */ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); } #endif /* CONFIG_MEMCG */ From patchwork Tue May 21 20:07:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954449 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B4E91390 for ; Tue, 21 May 2019 20:19:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 06ACD28AFC for ; Tue, 21 May 2019 20:19:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EB59228AFF; Tue, 21 May 2019 20:19:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 670A128AFC for ; Tue, 21 May 2019 20:19:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AD066B0003; Tue, 21 May 2019 16:18:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 45D736B0006; Tue, 21 May 2019 16:18:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 324766B0007; Tue, 21 May 2019 16:18:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f70.google.com (mail-yw1-f70.google.com [209.85.161.70]) by kanga.kvack.org (Postfix) with ESMTP id 0D5016B0003 for ; Tue, 21 May 2019 16:18:59 -0400 (EDT) Received: by mail-yw1-f70.google.com with SMTP id b81so18943670ywc.8 for ; Tue, 21 May 2019 13:18:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=JPu1ZqgGG6VS+4jhCMwR1QB6RLw1Ggqmre9BJXnrVoY=; b=ktylAWoG7mR3+p0bynefdAvd099fMHTQaw3zQ1Y6uv9AVj8ER3ZdpRD1/w1t55qIcj MF5sIRIziiWC9uV41xRES3jB/rJ7d8PmDNfi+WYttMkHFX+DKRJhrag51Uqzi+qRq4mo MfCTVjTOje9oM56DdVn2dmggkzQjF3DAktNgtqP4W5sxeUbs3kbNuNfX9IUqrAm2Z9nk XFeiWBgf0pW9kEllTDY2bTq2eT0vi9dnj7mfVGeiwNO8acCmNTEXG2KrF9PpNEkzxIPM Tg4Oo6XiWr7zwnIKuJauFTS2rwUQoaDH6yPJ9VAGXAB06VXwrVkGnXDX5hRdsUdYNd+W os9Q== X-Gm-Message-State: APjAAAVlSXWMuiH4IFQSCwJjr1DUiPN3q8fJHV4LcXtjBNxVf+4TdZny ipJIRBV34Nfr558bzBlld1bjE0ry7HsFHP2schR2PuUQvdhHt/qiW6J33+ZHbU0107raKSJUisl EUBGWZMT1aLrCFPlMjFCsYvIB2qKqMWWU79B4X78eAcdKq98JSyW92diffwMzPB7G9g== X-Received: by 2002:a81:a189:: with SMTP id y131mr2729443ywg.245.1558469938691; Tue, 21 May 2019 13:18:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqxH9+u6I47t6+syyrJS3wqQzi8KWX4r8Og6mhjEXdv+3y+5/2+B5LjVgk7bJ9JSBR/Ax7Ke X-Received: by 2002:a81:a189:: with SMTP id y131mr2729411ywg.245.1558469937759; Tue, 21 May 2019 13:18:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558469937; cv=none; d=google.com; s=arc-20160816; b=dluIhtOjSMxjd7lk4BAUWtDlxMKBX2z2YYYPvLih4ipu4Z6qtROBgklLPzhIXW8cOs 8fEVO1y7wueQjmMVE3EepAxwmTAKVJrgEqY0uOgFGnBbUsYSYi6Zpf8ag8/PcHu0QCvV 3PlfqCSPAGkruZEBRuZOZbXQIdV9QZCZh+iU288PvdbMEBXbpE3VCP/pkk2BC8Wd16Jz l09fMFwWD8RRgExKo/4nS7UwNBpDpTStfhKwQrG7Opleo+7tFPTVnLj168re+TKTpQ43 gNzav5B7X5bTV+jc6IvwbZ/VNcSLckKnHmPU/w4okqEb30EuD0JIxorzwrLHL+r2xX1C GoRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=JPu1ZqgGG6VS+4jhCMwR1QB6RLw1Ggqmre9BJXnrVoY=; b=lC2MHIoV4ZZ10epVpkxSoLvdi5dcJt6I4jWsCDfiAdkvdVZN8Aqr7WwJu1FpXi9+E/ pTlQWQZSoX/AFyMPkOwSwNTc7ogPUZQyD3z5H+0GYudHyBHe0tAR7j9J6nkAj8XBZjTY s0fZBPwMH50ta9DALZGq2ZKLLoESkAd1wGsx165trpNtWmBJ4JQHBCFavi0tQ1oU3m7q cHCgnfohA+dfsUpU9PVlNAz5QmW4Jq3FWlOi7l3nn2NkDf+E5T8Rw5xuGX1GU79GxWW1 h/rEAQ3qP09U5v3vF4gyKnrOzkt5MmwDhpUJiCKLz7wKj4lMUpMZss9RQ2a1P5aRCfXy om7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LOQ0FTTD; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id y133si6729207ywa.233.2019.05.21.13.18.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:18:57 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LOQ0FTTD; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKHVeV004126 for ; Tue, 21 May 2019 13:18:57 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=JPu1ZqgGG6VS+4jhCMwR1QB6RLw1Ggqmre9BJXnrVoY=; b=LOQ0FTTD51OOaEqSKGW5yZmJhgz69LWxCXc8HqSY/aGFBwFM4G7X7GgVrLsugfDnwhWb t78peT3qAtIElp9GxmcqXcvcgop/y8m608MgmZjqrvjAuPnM6c39gIXYblpjei22sAAA WyaSVOrzdxSEP2qrH4QpHH9glnz0P//S0po= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2smb4mjrym-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 21 May 2019 13:18:57 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 21 May 2019 13:18:47 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 08C751245FFA5; Tue, 21 May 2019 13:07:50 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 3/7] mm: introduce __memcg_kmem_uncharge_memcg() Date: Tue, 21 May 2019 13:07:31 -0700 Message-ID: <20190521200735.2603003-4-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work. This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_uncharge_memcg() if memcg_kmem_enabled() check is passed. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 36bdfe8e5965..deb209510902 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1298,6 +1298,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge(struct page *page, int order); int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; @@ -1339,6 +1341,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, return __memcg_kmem_charge_memcg(page, gfp, order, memcg); return 0; } + +static inline void memcg_kmem_uncharge_memcg(struct page *page, int order, + struct mem_cgroup *memcg) +{ + if (memcg_kmem_enabled()) + __memcg_kmem_uncharge_memcg(memcg, 1 << order); +} + /* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48a8f1c35176..b2c39f187cbb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2750,6 +2750,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) css_put(&memcg->css); return ret; } + +/** + * __memcg_kmem_uncharge_memcg: uncharge a kmem page + * @memcg: memcg to uncharge + * @nr_pages: number of pages to uncharge + */ +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} /** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge @@ -2764,14 +2780,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) return; VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); - - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - + __memcg_kmem_uncharge_memcg(memcg, nr_pages); page->mem_cgroup = NULL; /* slab pages do not have PageKmemcg flag set */ From patchwork Tue May 21 20:07:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954455 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D60451395 for ; Tue, 21 May 2019 20:23:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C62E828B05 for ; Tue, 21 May 2019 20:23:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA7C328B0A; Tue, 21 May 2019 20:23:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 070F728B08 for ; Tue, 21 May 2019 20:23:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D72E6B0003; Tue, 21 May 2019 16:23:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 088406B0006; Tue, 21 May 2019 16:23:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBB196B0007; Tue, 21 May 2019 16:23:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id B318D6B0003 for ; Tue, 21 May 2019 16:23:51 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id t1so68130pfa.10 for ; Tue, 21 May 2019 13:23:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=A03AYshFiKbUQ0yCq/bPVpZ5Z4eXewT0wNUpNBIpBX4=; b=dEoe+AK9T1Jgl2Z1kPVc98AYZsSSoqVBY2V0reIyTr9Xeu+CWf4G5oqFqdeB4qdlJQ gmsDLodqo04u0dmS3/c5pFeu6D53+4VzlV5iODKdgeo6YlacwIORvUEhGlftq6/zrGLs LSyQIe9YWNtA6MVkqsrCAKfLrIINNqVMd4VKRByWw5xjH1AwPUWi+h6YOhpXOVKtLG30 Onl4g8DKWZKEZPDQoQPjJsgkQURRZoZP6V+V1U8e7BnhizDjFp4woeg4X/9cGtJdscL9 mMW8BhCXcdeVMhi/uMX2GeOIbEzhBLxFX1tUd3avO6ZWXAA0OBkA/DjQkliSdA4nRkmF j2Mw== X-Gm-Message-State: APjAAAWP+iR35Tfs/piR3NWePt9gi0tGyyfPgFPAau7ma7wgNADXcr1o MgKZHcxeI5AHct7bQ5U3dWKnwwsBxFgupbt9NlTPiCtfihVd61l/VpveIlqm0Tr9461/cw+ScSi xj40GxiVr5wvCfGYpgV7mrTqT9nCNahRsYzOBDdE/tkVqNCDErni26KVWOZBJm+rolA== X-Received: by 2002:a17:902:ac98:: with SMTP id h24mr29224003plr.265.1558470231305; Tue, 21 May 2019 13:23:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqyKzl4gssoe8m141Rv94/GSDqXNCCPmcOAs6b50ocRBGcl9xLdOfiEHGXcBpfRX23nZgsSb X-Received: by 2002:a17:902:ac98:: with SMTP id h24mr29223918plr.265.1558470230354; Tue, 21 May 2019 13:23:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558470230; cv=none; d=google.com; s=arc-20160816; b=xGHvN/aRX6FoRx0GUOQcvNkONt+CVXP0uQVkqYbyOi4tCxxu0hz5sg9s+cbBEx3eqG FyEFVuYAoHPiky3wOid+Zk8OInLs6JU6Rh2Ck1tqSrtrX2zIg3n8COM7TDiagBwEQuJj O7kJpIeeFNrgnUtfGMBVjbRqCOI4RFPPDJBvLwbpQPaPXZdr88sUCj3H3Fw6qm63bCzD QqzpaeNZo629dVY0ScSFSF6J1SeI0mDphZ5T/tOwv2GezljR48YMwvLStrKM7Gh3xr1N I5Y0DOU6Odqw1pjnV0q7A/5+jOucKpz4FVBSVPpmbUkXkfv2Zepne0grXPxpJ2Cp7AtE kauw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=A03AYshFiKbUQ0yCq/bPVpZ5Z4eXewT0wNUpNBIpBX4=; b=h3gYh8FiaTpA1TCl2+rm5kG41tEpA14RlwAoxpt3ce/uxmjgYjXLpyWATFkkIxSGhf 9fM4hAidgJde/8z3tNQ320zglzN67IdvTE+bD7dVHTQdBLf7IdYa4lUGk69zFfiP+kNi kQ7f8keyt4D8YxbSIvEPAxMJ88GJ0gSRYdCchjnd1QCvEfKE55/0gJ801rQxsgKMec3J vayA9j5tev7l56KnyuemTVClBI9oZVPOh7qSyyROVWNBk9DNxiDTD6pdDWf9Oo+x4B0x BEOwJcFqFQRApIvnYaxfsb38N38LKcB58i8p6/s2wkcrJqWRmxwNQABKL2wgGdNEWWhP eYUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=e+Wr4I9+; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id b3si10778762pgd.243.2019.05.21.13.23.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:23:50 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=e+Wr4I9+; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKMxfR014110 for ; Tue, 21 May 2019 13:23:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=A03AYshFiKbUQ0yCq/bPVpZ5Z4eXewT0wNUpNBIpBX4=; b=e+Wr4I9+mlsoWfyPjB0jQ7nRJjqCfBcqS6JZwm2pI4wiDvD52yGgW5DYQF3oi+kutj3k hl+HF/+Ryqjnoo/+5DUIYk2be/NsJ2HN1iKjbH7Pw0CS1g6IgvV7Cbuz11ninGCITUdu 63xwrVTPOpS15HOudXlwggwhiTged/ak5OY= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2smd9cjgj3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 May 2019 13:23:49 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 21 May 2019 13:23:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 0D3C01245FFA7; Tue, 21 May 2019 13:07:50 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 4/7] mm: unify SLAB and SLUB page accounting Date: Tue, 21 May 2019 13:07:32 -0700 Message-ID: <20190521200735.2603003-5-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905210127 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the page accounting code is duplicated in SLAB and SLUB internals. Let's move it into new (un)charge_slab_page helpers in the slab_common.c file. These helpers will be responsible for statistics (global and memcg-aware) and memcg charging. So they are replacing direct memcg_(un)charge_slab() calls. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Christoph Lameter Acked-by: Vladimir Davydov Acked-by: Johannes Weiner --- mm/slab.c | 19 +++---------------- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slub.c | 14 ++------------ 3 files changed, 30 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 83000e46b870..32e6af9ed9af 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1389,7 +1389,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { struct page *page; - int nr_pages; flags |= cachep->allocflags; @@ -1399,17 +1398,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - if (memcg_charge_slab(page, flags, cachep->gfporder, cachep)) { + if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { __free_pages(page, cachep->gfporder); return NULL; } - nr_pages = (1 << cachep->gfporder); - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages); - __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1424,12 +1417,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - unsigned long nr_freed = (1 << order); - - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed); BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); @@ -1438,8 +1425,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping = NULL; if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; - memcg_uncharge_slab(page, order, cachep); + current->reclaim_state->reclaimed_slab += 1 << order; + uncharge_slab_page(page, order, cachep); __free_pages(page, order); } diff --git a/mm/slab.h b/mm/slab.h index 4a261c97c138..c9a31120fa1d 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer, void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **); int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **); +static inline int cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; +} + #ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */ @@ -352,6 +358,25 @@ static inline void memcg_link_cache(struct kmem_cache *s, #endif /* CONFIG_MEMCG_KMEM */ +static __always_inline int charge_slab_page(struct page *page, + gfp_t gfp, int order, + struct kmem_cache *s) +{ + int ret = memcg_charge_slab(page, gfp, order, s); + + if (!ret) + mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + + return ret; +} + +static __always_inline void uncharge_slab_page(struct page *page, int order, + struct kmem_cache *s) +{ + mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + memcg_uncharge_slab(page, order, s); +} + static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) { struct kmem_cache *cachep; diff --git a/mm/slub.c b/mm/slub.c index 8abd2d2a4ae4..13e415cc71b7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1490,7 +1490,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, else page = __alloc_pages_node(node, flags, order); - if (page && memcg_charge_slab(page, flags, order, s)) { + if (page && charge_slab_page(page, flags, order, s)) { __free_pages(page, order); page = NULL; } @@ -1683,11 +1683,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) if (!page) return NULL; - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - 1 << oo_order(oo)); - inc_slabs_node(s, page_to_nid(page), page->objects); return page; @@ -1721,18 +1716,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page) check_object(s, page, p, SLUB_RED_INACTIVE); } - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - -pages); - __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - memcg_uncharge_slab(page, order, s); + uncharge_slab_page(page, order, s); __free_pages(page, order); } From patchwork Tue May 21 20:07:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954457 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94FB81395 for ; Tue, 21 May 2019 20:29:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7BA1E28606 for ; Tue, 21 May 2019 20:29:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F001286BF; Tue, 21 May 2019 20:29:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D21528606 for ; Tue, 21 May 2019 20:29:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1E646B0003; Tue, 21 May 2019 16:29:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C6ACD6B0006; Tue, 21 May 2019 16:29:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0D106B0008; Tue, 21 May 2019 16:29:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f197.google.com (mail-yb1-f197.google.com [209.85.219.197]) by kanga.kvack.org (Postfix) with ESMTP id 821506B0006 for ; Tue, 21 May 2019 16:29:05 -0400 (EDT) Received: by mail-yb1-f197.google.com with SMTP id y185so1830ybc.18 for ; Tue, 21 May 2019 13:29:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=X5XuEH+7k7VZ8ovLoCum2B4VMDZum7qKR9DIlpG5Ceo=; b=eQt3o9BhRaeocNdBf/FpPWTh/3aOdl/s4xt5ZV0FXeJOuCNncphqUYIT/Y+Fg/3wUZ DguFEp4h8oLR7jF3ZJ21PR50GzP+dEW4kp1aOJBUBKCcdO3d6sTWFJ8whWxHXpcwxY+i gCm11Xd0iZxWZoHuuM/nN+hx1L4EY633aCLmpz6m91on2JOXT+6Y12izBf+4XgmR1Sr7 9LGkA3seXSBPoNAFoNuNnpBMWuaHSMXiCU6feUicOZC+INqaIK/0mrSLzFB3mr/h0vOo oiN8QzQnZqSdu3r56Nem4fj8JKmK5KDu/5d/u+vFtnqiU8rGgRVmoN3mGDYREeE8EmgL W4ig== X-Gm-Message-State: APjAAAU/1JAIfbroDpfH46u3Tdu3E6TbWJpuSshb9EOW4z1UG9bzBhmv lnHoGVlLrA3NkndoDW4X8qNVdAbVO9Ciz2i9CiWN5HYt+qPdyzBB8cCZIv6zkbtON+KSEXPiYB3 ySIJCUjDqHA8LK1BWJTEoucHRly8G+owtzXR8I062SNZRKdBQzkf9NGYzXnwwaA9big== X-Received: by 2002:a25:75d4:: with SMTP id q203mr8049074ybc.118.1558470544930; Tue, 21 May 2019 13:29:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqxLBwi58YpVfoUHaecARb8IdXA9t1x6QoOWg4veU+rIodVl5l/GFjiwReD/BhkCWUf13NFW X-Received: by 2002:a25:75d4:: with SMTP id q203mr8049017ybc.118.1558470543522; Tue, 21 May 2019 13:29:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558470543; cv=none; d=google.com; s=arc-20160816; b=Pgnk5PcxUocudI3KfBfEStKFt6CI3I/MuEyeQ3r+4xo9O5AnE0NT9L686h0d6ZRPkf 870YWGzn4JGiQFvOynI0ZetRxWHuSZ31SJeys2YW5gnzDQxLS8CHR7eL7QvC9pDdsdIm uZHE9BmRJarbqg67YRYrxcKMVUnqB70INtE1pABZFEicYkBHZig43GKq4pVrNcpCscBU Kcx8IWlYzsmS/0NVKTqHaV61IeQ6UqeI+w2I9IY3PZ53Tl/TtdM89TLbsNdPg0de4N87 qJcKYaSSZYZdPu53PYBKI0JORMsRYo0hUAJOzJO7ec6LltrPD4qeNB6bV54MI4fVZ77n c81Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=X5XuEH+7k7VZ8ovLoCum2B4VMDZum7qKR9DIlpG5Ceo=; b=irOlG6pD0lKJYPpxSmMyzVfmNc7+gmI1O6M5kIlu5ok+cLkcVFGD9i/ODmPq3xnaH+ F1Ya6cWFKQz1uDjTzRvyEtWhU7/7BcRmSJEgnxdrq14/P+bX/H9wI+WMfq/AT7ho7up/ +wNEMpq0BQv5Eos5S5ud20HfM3a8E1G0LciVs9sdSyEnBsIQLaTnqQoZ/lcL9Fjmfd9r q8p1AgfjEQZCtST7ag3g/zweb8Yj9HkkFTPuIsvphouBk8M3huDDdut5J+SrUehUtjK8 ApWK0L6gSTwBt6qjUTPdkxJdURPFuAiqpILgckClcZIHlDeDGs5LqtHgvmtHgRY8pN7T wE8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=E6nP4bmW; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id d131si5936961ybb.24.2019.05.21.13.29.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:29:03 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=E6nP4bmW; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKROnl013569 for ; Tue, 21 May 2019 13:29:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=X5XuEH+7k7VZ8ovLoCum2B4VMDZum7qKR9DIlpG5Ceo=; b=E6nP4bmWWwQfYCxBTIxOha1MMkIKZR76v+95Uagi7U81x9+kjRaPQ0+6viIkcHj9HzBB mPH5+zvdbP39FIMPFOb6msH7w4cmzfWpu/CXb0LrVcxI9Z+46W5N10FSo6+oZYiHOwRi kVUBHydYWqFwt4DcEiQLvdcoPr0o0H/sb5I= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2smb4mjt33-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 May 2019 13:29:02 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 21 May 2019 13:29:01 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 11E3F1245FFAB; Tue, 21 May 2019 13:07:50 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 5/7] mm: rework non-root kmem_cache lifecycle management Date: Tue, 21 May 2019 13:07:33 -0700 Message-ID: <20190521200735.2603003-6-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905210128 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This commit makes several important changes in the lifecycle of a non-root kmem_cache, which also affect the lifecycle of a memory cgroup. Currently each charged slab page has a page->mem_cgroup pointer to the memory cgroup and holds a reference to it. Kmem_caches are held by the memcg and are released with it. It means that none of kmem_caches are released unless at least one reference to the memcg exists, which is not optimal. So the current scheme can be illustrated as: page->mem_cgroup->kmem_cache. To implement the slab memory reparenting we need to invert the scheme into: page->kmem_cache->mem_cgroup. Let's make every page to hold a reference to the kmem_cache (we already have a stable pointer), and make kmem_caches to hold a single reference to the memory cgroup. To make this possible we need to introduce a new percpu refcounter for non-root kmem_caches. The counter is initialized to the percpu mode, and is switched to atomic mode after deactivation, so we never shutdown an active cache. The counter is bumped for every charged page and also for every running allocation. So the kmem_cache can't be released unless all allocations complete. To shutdown non-active empty kmem_caches, let's reuse the infrastructure of the RCU-delayed work queue, used previously for the deactivation. After the generalization, it's perfectly suited for our needs. Since now we can release a kmem_cache at any moment after the deactivation, let's call sysfs_slab_remove() only from the shutdown path. It makes deactivation path simpler. Because we don't set the page->mem_cgroup pointer, we need to change the way how memcg-level stats is working for slab pages. We can't use mod_lruvec_page_state() helpers anymore, so switch over to mod_lruvec_state(). * I used the following simple approach to test the performance (stolen from another patchset by T. Harding): time find / -name fname-no-exist echo 2 > /proc/sys/vm/drop_caches repeat 10 times Results (I've chosen best results in several runs): orig patched real 0m0.648s real 0m0.593s user 0m0.148s user 0m0.162s sys 0m0.295s sys 0m0.253s real 0m0.581s real 0m0.649s user 0m0.119s user 0m0.136s sys 0m0.254s sys 0m0.250s real 0m0.645s real 0m0.705s user 0m0.138s user 0m0.138s sys 0m0.263s sys 0m0.250s real 0m0.691s real 0m0.718s user 0m0.139s user 0m0.134s sys 0m0.262s sys 0m0.253s real 0m0.654s real 0m0.715s user 0m0.146s user 0m0.128s sys 0m0.247s sys 0m0.261s real 0m0.675s real 0m0.717s user 0m0.129s user 0m0.137s sys 0m0.277s sys 0m0.248s real 0m0.631s real 0m0.719s user 0m0.137s user 0m0.134s sys 0m0.255s sys 0m0.251s real 0m0.622s real 0m0.715s user 0m0.108s user 0m0.124s sys 0m0.279s sys 0m0.264s real 0m0.651s real 0m0.669s user 0m0.139s user 0m0.139s sys 0m0.252s sys 0m0.247s real 0m0.671s real 0m0.632s user 0m0.130s user 0m0.139s sys 0m0.263s sys 0m0.245s So it looks like the difference is not noticeable in this test. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/slab.h | 3 +- mm/memcontrol.c | 57 +++++++++++++++++++++--------- mm/slab.h | 82 +++++++++++++++++++++++++------------------- mm/slab_common.c | 74 +++++++++++++++++++++++---------------- mm/slub.c | 12 +------ 5 files changed, 135 insertions(+), 93 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 47923c173f30..1b54e5f83342 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include /* @@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); -void memcg_destroy_kmem_caches(struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -641,6 +641,7 @@ struct memcg_cache_params { struct mem_cgroup *memcg; struct list_head children_node; struct list_head kmem_caches_node; + struct percpu_ref refcnt; void (*work_fn)(struct kmem_cache *); union { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b2c39f187cbb..1828d82763d8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2610,12 +2610,13 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg, { struct memcg_kmem_cache_create_work *cw; + if (!css_tryget_online(&memcg->css)) + return; + cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN); if (!cw) return; - css_get(&memcg->css); - cw->memcg = memcg; cw->cachep = cachep; INIT_WORK(&cw->work, memcg_kmem_cache_create_func); @@ -2651,20 +2652,35 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; int kmemcg_id; + struct memcg_cache_array *arr; VM_BUG_ON(!is_root_cache(cachep)); if (memcg_kmem_bypass()) return cachep; - memcg = get_mem_cgroup_from_current(); + rcu_read_lock(); + + if (unlikely(current->active_memcg)) + memcg = current->active_memcg; + else + memcg = mem_cgroup_from_task(current); + + if (!memcg || memcg == root_mem_cgroup) + goto out_unlock; + kmemcg_id = READ_ONCE(memcg->kmemcg_id); if (kmemcg_id < 0) - goto out; + goto out_unlock; - memcg_cachep = cache_from_memcg_idx(cachep, kmemcg_id); - if (likely(memcg_cachep)) - return memcg_cachep; + arr = rcu_dereference(cachep->memcg_params.memcg_caches); + + /* + * Make sure we will access the up-to-date value. The code updating + * memcg_caches issues a write barrier to match this (see + * memcg_create_kmem_cache()). + */ + memcg_cachep = READ_ONCE(arr->entries[kmemcg_id]); /* * If we are in a safe context (can wait, and not in interrupt @@ -2677,10 +2693,20 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) * memcg_create_kmem_cache, this means no further allocation * could happen with the slab_mutex held. So it's better to * defer everything. + * + * If the memcg is dying or memcg_cache is about to be released, + * don't bother creating new kmem_caches. Because memcg_cachep + * is ZEROed as the fist step of kmem offlining, we don't need + * percpu_ref_tryget() here. css_tryget_online() check in + * memcg_schedule_kmem_cache_create() will prevent us from + * creation of a new kmem_cache. */ - memcg_schedule_kmem_cache_create(memcg, cachep); -out: - css_put(&memcg->css); + if (unlikely(!memcg_cachep)) + memcg_schedule_kmem_cache_create(memcg, cachep); + else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) + cachep = memcg_cachep; +out_unlock: + rcu_read_unlock(); return cachep; } @@ -2691,7 +2717,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) void memcg_kmem_put_cache(struct kmem_cache *cachep) { if (!is_root_cache(cachep)) - css_put(&cachep->memcg_params.memcg->css); + percpu_ref_put(&cachep->memcg_params.refcnt); } /** @@ -2719,9 +2745,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, cancel_charge(memcg, nr_pages); return -ENOMEM; } - - page->mem_cgroup = memcg; - return 0; } @@ -2744,8 +2767,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg); - if (!ret) + if (!ret) { + page->mem_cgroup = memcg; __SetPageKmemcg(page); + } } css_put(&memcg->css); return ret; @@ -3238,7 +3263,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) memcg_offline_kmem(memcg); if (memcg->kmem_state == KMEM_ALLOCATED) { - memcg_destroy_kmem_caches(memcg); + WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); WARN_ON(page_counter_read(&memcg->kmem)); } diff --git a/mm/slab.h b/mm/slab.h index c9a31120fa1d..b86744c58702 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -173,6 +173,7 @@ void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); +void kmemcg_cache_shutdown(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -248,31 +249,6 @@ static inline const char *cache_name(struct kmem_cache *s) return s->name; } -/* - * Note, we protect with RCU only the memcg_caches array, not per-memcg caches. - * That said the caller must assure the memcg's cache won't go away by either - * taking a css reference to the owner cgroup, or holding the slab_mutex. - */ -static inline struct kmem_cache * -cache_from_memcg_idx(struct kmem_cache *s, int idx) -{ - struct kmem_cache *cachep; - struct memcg_cache_array *arr; - - rcu_read_lock(); - arr = rcu_dereference(s->memcg_params.memcg_caches); - - /* - * Make sure we will access the up-to-date value. The code updating - * memcg_caches issues a write barrier to match this (see - * memcg_create_kmem_cache()). - */ - cachep = READ_ONCE(arr->entries[idx]); - rcu_read_unlock(); - - return cachep; -} - static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) { if (is_root_cache(s)) @@ -280,19 +256,49 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s->memcg_params.root_cache; } +/* + * Charge the slab page belonging to the non-root kmem_cache. + * Can be called for non-root kmem_caches only. + */ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - if (is_root_cache(s)) - return 0; - return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + int ret; + + memcg = s->memcg_params.memcg; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); + if (ret) + return ret; + + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); + + /* transer try_charge() page references to kmem_cache */ + percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + css_put_many(&memcg->css, 1 << order); + + return 0; } +/* + * Uncharge a slab page belonging to a non-root kmem_cache. + * Can be called for non-root kmem_caches only. + */ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { - memcg_kmem_uncharge(page, order); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + + memcg = s->memcg_params.memcg; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); + memcg_kmem_uncharge_memcg(page, order, memcg); + + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } extern void slab_init_memcg_params(struct kmem_cache *); @@ -362,18 +368,24 @@ static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - int ret = memcg_charge_slab(page, gfp, order, s); - - if (!ret) - mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + 1 << order); + return 0; + } - return ret; + return memcg_charge_slab(page, gfp, order, s); } static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { - mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + -(1 << order)); + return; + } + memcg_uncharge_slab(page, order, s); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 4e5b4292a763..8d68de4a2341 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -130,6 +130,9 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr, #ifdef CONFIG_MEMCG_KMEM LIST_HEAD(slab_root_caches); +static DEFINE_SPINLOCK(memcg_kmem_wq_lock); + +static void kmemcg_queue_cache_shutdown(struct percpu_ref *percpu_ref); void slab_init_memcg_params(struct kmem_cache *s) { @@ -145,6 +148,12 @@ static int init_memcg_params(struct kmem_cache *s, struct memcg_cache_array *arr; if (root_cache) { + int ret = percpu_ref_init(&s->memcg_params.refcnt, + kmemcg_queue_cache_shutdown, + 0, GFP_KERNEL); + if (ret) + return ret; + s->memcg_params.root_cache = root_cache; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); @@ -170,6 +179,8 @@ static void destroy_memcg_params(struct kmem_cache *s) { if (is_root_cache(s)) kvfree(rcu_access_pointer(s->memcg_params.memcg_caches)); + else + percpu_ref_exit(&s->memcg_params.refcnt); } static void free_memcg_params(struct rcu_head *rcu) @@ -225,6 +236,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + css_get(&memcg->css); s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); @@ -240,6 +252,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); + css_put(&s->memcg_params.memcg->css); } } #else @@ -708,16 +721,13 @@ static void kmemcg_after_rcu_workfn(struct work_struct *work) put_online_mems(); put_online_cpus(); - - /* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ - css_put(&s->memcg_params.memcg->css); } /* * We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be - * initialized eariler. -*/ + * initialized earlier. + */ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, @@ -727,9 +737,31 @@ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } +static void kmemcg_cache_shutdown_after_rcu(struct kmem_cache *s) +{ + WARN_ON(shutdown_cache(s)); +} + +static void kmemcg_queue_cache_shutdown(struct percpu_ref *percpu_ref) +{ + struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache, + memcg_params.refcnt); + + spin_lock(&memcg_kmem_wq_lock); + if (s->memcg_params.root_cache->memcg_params.dying) + goto unlock; + + WARN_ON(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_shutdown_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); +unlock: + spin_unlock(&memcg_kmem_wq_lock); +} + static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { __kmemcg_cache_deactivate_after_rcu(s); + percpu_ref_kill(&s->memcg_params.refcnt); } static void kmemcg_cache_deactivate(struct kmem_cache *s) @@ -739,9 +771,6 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) if (s->memcg_params.root_cache->memcg_params.dying) return; - /* pin memcg so that @s doesn't get destroyed in the middle */ - css_get(&s->memcg_params.memcg->css); - WARN_ON_ONCE(s->memcg_params.work_fn); s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); @@ -775,28 +804,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) put_online_cpus(); } -void memcg_destroy_kmem_caches(struct mem_cgroup *memcg) -{ - struct kmem_cache *s, *s2; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry_safe(s, s2, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - /* - * The cgroup is about to be freed and therefore has no charges - * left. Hence, all its caches must be empty by now. - */ - BUG_ON(shutdown_cache(s)); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { struct memcg_cache_array *arr; @@ -854,8 +861,15 @@ static int shutdown_memcg_caches(struct kmem_cache *s) static void flush_memcg_workqueue(struct kmem_cache *s) { + /* + * memcg_params.dying is synchronized using slab_mutex AND + * memcg_kmem_wq_lock spinlock, because it's not always + * possible to grab slab_mutex. + */ mutex_lock(&slab_mutex); + spin_lock(&memcg_kmem_wq_lock); s->memcg_params.dying = true; + spin_unlock(&memcg_kmem_wq_lock); mutex_unlock(&slab_mutex); /* diff --git a/mm/slub.c b/mm/slub.c index 13e415cc71b7..0a4ddbeb5ca6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4018,18 +4018,8 @@ void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. - * Even if @s becomes empty after shrinking, we can't know that @s - * doesn't have allocations already in-flight and thus can't - * destroy @s until the associated memcg is released. - * - * However, let's remove the sysfs files for empty caches here. - * Each cache has a lot of interface files which aren't - * particularly useful for empty draining caches; otherwise, we can - * easily end up with millions of unnecessary sysfs files on - * systems which have a lot of memory and transient cgroups. */ - if (!__kmem_cache_shrink(s)) - sysfs_slab_remove(s); + __kmem_cache_shrink(s); } void __kmemcg_cache_deactivate(struct kmem_cache *s) From patchwork Tue May 21 20:07:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954459 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 90FFF933 for ; Tue, 21 May 2019 20:29:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7CAEF28606 for ; Tue, 21 May 2019 20:29:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6CF3B286BF; Tue, 21 May 2019 20:29:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A2E4528606 for ; Tue, 21 May 2019 20:29:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F39F06B0006; Tue, 21 May 2019 16:29:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EB7E46B000A; Tue, 21 May 2019 16:29:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA62F6B0007; Tue, 21 May 2019 16:29:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 66EB36B0003 for ; Tue, 21 May 2019 16:29:05 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id c7so66498pfp.14 for ; Tue, 21 May 2019 13:29:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=KM/nHlU4zhsLfG8c0YB8Ffe5Um+2X27Yn+rA2wKE+IE=; b=E7Xp8NmzywqMAJWlNX1qcEG13ymmuvI5EWHbrFIlCMABacIEkoFhCRc91lPzCJXUiX KeQv+iEChTjhYcpmJUxS2PycDPXL5fGblqW5F36dtsgoRlxyPYDnoR4HJdNHSkVOypXS kkUG9J1lVeyH7kVyGhe7OA103XGgYcAdiz5uj0nJH+vPGofyodERfQ3A24JpbC5gvbUr rHVokYvhJ702FZNfc6ndrCSighZrwgKmZZT0HRPnfd1XThVmoBZkvZIhaLj5ViIfzQIQ zQwytpuz2Y0p5Xr+3Gz+4XNz1QBHVHju6+at5DTblbsVWqWnwCKNHhv0Lbwy/M6H3mkf f7dg== X-Gm-Message-State: APjAAAX0RggjoTVrnJKWr8FVH6dLluNzv/N9qioC4BP29K/qvy1/IXd8 WoC/SlWGU21qgD39D3xPv5j6cqEauu/5lrlT+n8kzS8H4NBnyKtqXvBmfORihcI8UJAJV+VDhGe fFoJraZW7xP6yPntDpjNjx608Sev95IXKvP74xnNLHJBBZd4VYbsp9zAtcMX477HGJQ== X-Received: by 2002:a65:41c6:: with SMTP id b6mr20385715pgq.399.1558470545016; Tue, 21 May 2019 13:29:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqwFKL5cfOe8p+Zv5XPcUEy+F3uD3LsgPw2f0BMvcC5Eg5Foe5S9IOutYzDiygBTzZB9LQ8Z X-Received: by 2002:a65:41c6:: with SMTP id b6mr20385596pgq.399.1558470543841; Tue, 21 May 2019 13:29:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558470543; cv=none; d=google.com; s=arc-20160816; b=Gg8fN8SYeZ4K5OC0x5dH6da/UE8hBqusQpAT5uuflJ+DzXii1AVu3mmgAaMEdb9zPm vUQbK03lNEqFAkuncObi6IYeoGe7DKXVI+N7v/K44TW4XkyPwrGEmATANw6W7XMIielm JjQjj7uSZ6vIBNoJmPFcAs+lVCnnrHOQYbqvA9PkPL3fxpVAPSU3iv9Dwgplc6NbGiDz +XmWFnzCLB/7NKo/KVpmkmlOj/w6mKMAqqiCJxpY0facpLvI/9/qLiVFAiZa5+m5vbhr xM5k7nOgI0pJprgTOL33t6IyqGL5CO1pAaTREBPUy3sExj4LBhGslmOyxm16z18ExpqS SCnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=KM/nHlU4zhsLfG8c0YB8Ffe5Um+2X27Yn+rA2wKE+IE=; b=eXktIJX99weRpOT/IFfqTn2mbuN31jhfgIKOlAhGhn8hog99t4z1iRuP/xRk31hj1b gEdNTDMVsfBu9wbIKVeLoVyggDBCPFTJ0+AMW+qn48m4WTormB3q89NVPd5c9n2aHjg4 7PKBRSRf0/KDxXQi4U8a30lxCA2WJZ6lnEzrrxfUmTIyrkdU7f0F3JeAWxqLW57jHoVC bcAAX7Vv7GHUZIErvroydaPa1qyqQfJh6QSk6cKhD250vmO8faXSuVsmmAeHJvz8yyVX +ka4uAclRZen3QtnoCi/m23l0ZN5AturYA+S5sWkKd2NR9TqFyciXynHUQaaXWIX9+qs JMoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=eQahIBYt; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id t1si23478457pgh.406.2019.05.21.13.29.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:29:03 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=eQahIBYt; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKRnPo018599 for ; Tue, 21 May 2019 13:29:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=KM/nHlU4zhsLfG8c0YB8Ffe5Um+2X27Yn+rA2wKE+IE=; b=eQahIBYtJTjd/zZp2PxdvVJmtCZx0dBSF0FPO/MDcPR3twsLvVsw5xk+Ax9vkT8ZznJi mAEypsAzuvw6xX5ZrV96rxMKtwsrKyfbeFdCxw5W3umAcQ1QE8JT6YMMqzGUeefxUdM5 /1Xirz2vJZ7v0JlBEcvMWZuMRFeDMc8MTxU= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2smd9cjh28-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 May 2019 13:29:03 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 21 May 2019 13:29:01 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 15EA91245FFB0; Tue, 21 May 2019 13:07:50 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 6/7] mm: reparent slab memory on cgroup removal Date: Tue, 21 May 2019 13:07:34 -0700 Message-ID: <20190521200735.2603003-7-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905210128 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application). So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of kmem_caches. Reparenting is performed as a part of the deactivation process. Since the parent cgroup is already charged, everything we need to do is to splice the list of kmem_caches to the parent's kmem_caches list, swap the memcg pointer and drop the css refcounter for each kmem_cache and adjust the parent's css refcounter. Quite simple. Please, note that kmem_cache->memcg_params.memcg isn't a stable pointer anymore. It's safe to read it under rcu_read_lock() or with slab_mutex held. We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere. Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE) aren't used anywhere except count_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/slab.h | 4 ++-- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 21 ++++++++++++++++----- mm/slab_common.c | 21 ++++++++++++++++++--- 4 files changed, 44 insertions(+), 16 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1b54e5f83342..109cab2ad9b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); -void memcg_deactivate_kmem_caches(struct mem_cgroup *); +void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -638,7 +638,7 @@ struct memcg_cache_params { bool dying; }; struct { - struct mem_cgroup *memcg; + struct mem_cgroup __rcu *memcg; struct list_head children_node; struct list_head kmem_caches_node; struct percpu_ref refcnt; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1828d82763d8..de664ff1e310 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3224,15 +3224,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) */ memcg->kmem_state = KMEM_ALLOCATED; - memcg_deactivate_kmem_caches(memcg); - - kmemcg_id = memcg->kmemcg_id; - BUG_ON(kmemcg_id < 0); - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; + memcg_deactivate_kmem_caches(memcg, parent); + + kmemcg_id = memcg->kmemcg_id; + BUG_ON(kmemcg_id < 0); + /* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus @@ -3265,7 +3265,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) if (memcg->kmem_state == KMEM_ALLOCATED) { WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); - WARN_ON(page_counter_read(&memcg->kmem)); } } #else @@ -4677,6 +4676,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) /* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM + INIT_LIST_HEAD(&memcg->kmem_caches); +#endif root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/slab.h b/mm/slab.h index b86744c58702..7ba50e526d82 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -268,10 +268,18 @@ static __always_inline int memcg_charge_slab(struct page *page, struct lruvec *lruvec; int ret; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); + while (memcg && !css_tryget_online(&memcg->css)) + memcg = parent_mem_cgroup(memcg); + rcu_read_unlock(); + + if (unlikely(!memcg)) + return true; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) - return ret; + goto out; lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); @@ -279,8 +287,9 @@ static __always_inline int memcg_charge_slab(struct page *page, /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); css_put_many(&memcg->css, 1 << order); - - return 0; +out: + css_put(&memcg->css); + return ret; } /* @@ -293,10 +302,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct mem_cgroup *memcg; struct lruvec *lruvec; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); memcg_kmem_uncharge_memcg(page, order, memcg); + rcu_read_unlock(); percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 8d68de4a2341..7607a40772aa 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -237,7 +237,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) list_add(&s->root_caches_node, &slab_root_caches); } else { css_get(&memcg->css); - s->memcg_params.memcg = memcg; + rcu_assign_pointer(s->memcg_params.memcg, memcg); list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -252,7 +252,8 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); - css_put(&s->memcg_params.memcg->css); + mem_cgroup_put(rcu_dereference_protected(s->memcg_params.memcg, + lockdep_is_held(&slab_mutex))); } } #else @@ -776,11 +777,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } -void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) +void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg, + struct mem_cgroup *parent) { int idx; struct memcg_cache_array *arr; struct kmem_cache *s, *c; + unsigned int nr_reparented; idx = memcg_cache_id(memcg); @@ -798,6 +801,18 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } + nr_reparented = 0; + list_for_each_entry(s, &memcg->kmem_caches, + memcg_params.kmem_caches_node) { + rcu_assign_pointer(s->memcg_params.memcg, parent); + css_put(&memcg->css); + nr_reparented++; + } + if (nr_reparented) { + list_splice_init(&memcg->kmem_caches, + &parent->kmem_caches); + css_get_many(&parent->css, nr_reparented); + } mutex_unlock(&slab_mutex); put_online_mems(); From patchwork Tue May 21 20:07:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10954461 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F0D4933 for ; Tue, 21 May 2019 20:29:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E13628606 for ; Tue, 21 May 2019 20:29:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 11D1B286BF; Tue, 21 May 2019 20:29:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87C2C28606 for ; Tue, 21 May 2019 20:29:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BDEE6B0007; Tue, 21 May 2019 16:29:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 170C06B0008; Tue, 21 May 2019 16:29:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 010686B000A; Tue, 21 May 2019 16:29:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id BCB286B0007 for ; Tue, 21 May 2019 16:29:06 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id k22so57929pfg.18 for ; Tue, 21 May 2019 13:29:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=1MscbZFIv/dMuN0XCm8C96ivxHN/q8ldFdECslyEkfY=; b=o5cEZ93kya2W+aJxTR6rFrL8ClfreaN+LQF9VT5HjUqO74toQhHmrbT/MsiTE4Z9Pm R+J0F92PLTfYC8/aazcG6Ugc4JQuAmfxnKQCBhImIuy5VQVkvTy7uqZ/FPzGtAV7EBtx gHoDQtwOk57rOJm/TKk047j3WoLFb90gVBtl47H/t8cVpQI/e2Tr93anYEgXetgN6j1D uV+5xyC1AuGXU0g397uHy5Bfbauj5A+eQBAG6UFKxoTBmMEB9+5oxias0XgYiaAYmJ5l lxxIV8om/rLbZ1pp4lSVUpgsavk2e4PeCfeTqedLxIaxxMBcnkGVPYHfHEQJmIbzJxrc Ouhg== X-Gm-Message-State: APjAAAW0ND+jCxGlpT8gjdBMsl2JddFWPa477oKr+0T7yRKOTTZ04lrv cpj6XsNYmsbvCNwCxhQ/ycwmwBM5WkgCnrMtbdDTE2G4/2bzDIIpho1DlQ8N7QLorBTpLPiSkOi J33BYMcXtcrTmxSErWW5oN9T6Tn87lG4PkHJUXfkC8VLUfEfDFTpSEMO21oN0JDo9uA== X-Received: by 2002:a17:902:2be7:: with SMTP id l94mr30909708plb.185.1558470546416; Tue, 21 May 2019 13:29:06 -0700 (PDT) X-Google-Smtp-Source: APXvYqxzrk+aEVlda3XlJpIDGWOK6MiCML8pvAiR3tJjTt2s+jEWbQCpXSYc9pdRDOeDIRvpKzj4 X-Received: by 2002:a17:902:2be7:: with SMTP id l94mr30909614plb.185.1558470545568; Tue, 21 May 2019 13:29:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558470545; cv=none; d=google.com; s=arc-20160816; b=RRqKv2G0ekvcuX6e51oKTh9VzGo4YuFLckXIqBGfvQxJv3SUhFbOAQM8ksvPAOPCOr mPHEvHtS3HjcOAk+piKaI67A4T5Mi3UMA4mnYiCHsaEHVgp1jjLXbKF8KAHYesYEamRX qwlVjaPlCBmMIfmZZKIIziwmP4SWXeIKWga4Kq4FQXtbr3nMAuT+Q38xVZf7GlfKOlAF 9Q/NvwzC4fkQ8k3/wLUHu9bfV8hrfZeV2ZdDcnX3n/j6o7Q+WqXrhDtIO95qsJbof2MY /TyOlc4ENIO5/U5BO2KY6qP8y6/wSKzQzuBm9rsglCZTMx/1RHGFrzkTcGBkOQoybEDB MwoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=1MscbZFIv/dMuN0XCm8C96ivxHN/q8ldFdECslyEkfY=; b=vaKogCuj8XhidIFo7s28GC/upI1HGrMwlWJd/A2DTaMEjKeq6oqLeeOKnGYUksEVdA 06BGXWEd+AbVNwfTd+MKf3FbAWBfDSAZM1huZgQghuce/t0A4f0muVT0qDjRIMUexyzV ZjjERLFP9R9heAZVKWKmywuzFL9AGMlZgqQ2tm6coltmdedZ7OGK8uDd3OVtDC8j4dV6 rXa63nLkLRTR8KGZIOQxxKkbZAFKkfaJuCNrZFVJkPKizJH7x+lmeTvoL21O/lZEKBC1 7PxfoX8N3j5DwJU8f2mb55NYct1zD/NrgxExzFf83zfPMx3b1i7eDqrEGC7m6ztWCOWQ 24xQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="hJH65C/8"; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id x28si25566848pfr.289.2019.05.21.13.29.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 May 2019 13:29:05 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="hJH65C/8"; spf=pass (google.com: domain of prvs=0044fe9fa5=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=0044fe9fa5=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x4LKRnPs018599 for ; Tue, 21 May 2019 13:29:05 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=1MscbZFIv/dMuN0XCm8C96ivxHN/q8ldFdECslyEkfY=; b=hJH65C/82a55FZSVGpq8mYpyG3sGGQ/lAiSIv0ANm1ProvQttMNZEXb3JrItoYBGjUSl aYgP3XaYZU+/7AuTUGj25iNKjOFDGWO9Saium5YzfznUII2HUvy03gT7NA6m46wbo3Ck XeqPLN+b9Q0nG0eQ96hg4TAhmMxCs3ZxEKM= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2smd9cjh28-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 21 May 2019 13:29:04 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 21 May 2019 13:29:01 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 1BAE91245FFB2; Tue, 21 May 2019 13:07:50 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v5 7/7] mm: fix /proc/kpagecgroup interface for slab pages Date: Tue, 21 May 2019 13:07:35 -0700 Message-ID: <20190521200735.2603003-8-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190521200735.2603003-1-guro@fb.com> References: <20190521200735.2603003-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-05-21_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=755 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1905210128 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Switching to an indirect scheme of getting mem_cgroup pointer for !root slab pages broke /proc/kpagecgroup interface for them. Let's fix it by learning page_cgroup_ino() how to get memcg pointer for slab pages. Reported-by: Shakeel Butt Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt --- mm/memcontrol.c | 5 ++++- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slab_common.c | 1 + 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index de664ff1e310..f58454f5cedc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -494,7 +494,10 @@ ino_t page_cgroup_ino(struct page *page) unsigned long ino = 0; rcu_read_lock(); - memcg = READ_ONCE(page->mem_cgroup); + if (PageHead(page) && PageSlab(page)) + memcg = memcg_from_slab_page(page); + else + memcg = READ_ONCE(page->mem_cgroup); while (memcg && !(memcg->css.flags & CSS_ONLINE)) memcg = parent_mem_cgroup(memcg); if (memcg) diff --git a/mm/slab.h b/mm/slab.h index 7ba50e526d82..50fa534c0fc0 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -256,6 +256,26 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s->memcg_params.root_cache; } +/* + * Expects a pointer to a slab page. Please note, that PageSlab() check + * isn't sufficient, as it returns true also for tail compound slab pages, + * which do not have slab_cache pointer set. + * So this function assumes that the page can pass PageHead() and PageSlab() + * checks. + */ +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + struct kmem_cache *s; + + WARN_ON_ONCE(!rcu_read_lock_held()); + + s = READ_ONCE(page->slab_cache); + if (s && !is_root_cache(s)) + return rcu_dereference(s->memcg_params.memcg); + + return NULL; +} + /* * Charge the slab page belonging to the non-root kmem_cache. * Can be called for non-root kmem_caches only. @@ -353,6 +373,11 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s; } +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + return NULL; +} + static inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { diff --git a/mm/slab_common.c b/mm/slab_common.c index 7607a40772aa..e818609c8209 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -254,6 +254,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) list_del(&s->memcg_params.kmem_caches_node); mem_cgroup_put(rcu_dereference_protected(s->memcg_params.memcg, lockdep_is_held(&slab_mutex))); + rcu_assign_pointer(s->memcg_params.memcg, NULL); } } #else