From patchwork Tue Jun 11 23:18:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988333 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EE7814E5 for ; Tue, 11 Jun 2019 23:18:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A97B2881A for ; Tue, 11 Jun 2019 23:18:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4E9522890A; Tue, 11 Jun 2019 23:18:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B9EB2881A for ; Tue, 11 Jun 2019 23:18:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A50CA6B000D; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9FF2B6B000E; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C9166B0269; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 6896A6B000D for ; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id d19so15427629ywb.2 for ; Tue, 11 Jun 2019 16:18:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=53Eg2QAwVCvd3SsQpm1bINM6H+u/Outwh4WdWfAbIvI=; b=k8Fckos3I8BjJrw/xCmEJgzEhPxAxNAk/sDczfNsefKsWn/ptESVA7zUIaAtAexx1G RCIbiwyYaVVMd6CwKkstOosML3sntIrFS/sLnPNv6YAfH2dEcxVUqoDVSu1piNwJfPwj gTXUQ26sdTBitMFdg67WOiaqFPWICgoecaQNKFTy86fdRY9RYb05GRgnqApTT0WJfSQI +/qUgDbJCdkMCkPlweBh9mSFnEnto6Tq6MT9bRFQuEOak7y6P1Vnak6h6+wx0Dj/2UWA msdlnwyi98pVcLmj/7atgUUbhSVLHzd5rPMiIdSacX0t9XJRZ7TuFwWLxIqe3u3C69pT mLYQ== X-Gm-Message-State: APjAAAWEi+yGXplFmX1KIKKU97OinjEwiESFvYArfEENJHShsgpZjkK/ lwf4p/ao7QBlkzoFlkoFMdXQuvdztvFgO/OscPqN6vFqdbfvY7YLNAY4Ln3wD4RvGjymHnZGb/s 2cdYoi47RQouR8rV/seMtmQrMLZpueN9t0SVFOC+vfxC6gxVeekj9PbSaY4EUd2t20w== X-Received: by 2002:a81:3dc8:: with SMTP id k191mr40440511ywa.383.1560295104063; Tue, 11 Jun 2019 16:18:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqyvRfr7ROydJp013I9+bpCnb7uMR1+8x+gMvkgXKdwxKjuvDVGizBoBfcm8TxHLpXBi27kL X-Received: by 2002:a81:3dc8:: with SMTP id k191mr40440484ywa.383.1560295103257; Tue, 11 Jun 2019 16:18:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295103; cv=none; d=google.com; s=arc-20160816; b=yUzf1TsXsjPA3rt/2J14sgjugMcF736CqKRVjqtCTjGg2sOEz4m/PqATis9SFGgFgp KNN9uubhosLUFoOgWE7TYQHA7nm2VfzOb9BcSDqrifP1w097COM/Z6asLQWa7sHCTOQ8 0I/bv/vSGvNuG17hQM1A8uOtmWCyRJLU72Tgj/h69HkVYlBk48UFu5NNDo2oL7oovgG0 6cXeoz/OSWq1zcOAuvdDPSYCEZrMp3p3QfDRzL9LszMlpFkpdJwOqA1oUjatuhdd1gha 5s0oM7FQ2OsevbJt6baIoiXbCmFthJ/qnrLo6aAUGvBlZt9Ls2PW24Fo8uSyHMCMpkH3 1Ijw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=53Eg2QAwVCvd3SsQpm1bINM6H+u/Outwh4WdWfAbIvI=; b=AawbSTZuQq68MDBzk1x62qxkjaZqarfAZ+dmztdfJz5swSskpnM/zjDcWHhwqSmVqx UEo1zat0QFjIeagQVgiOt8L9uwwPc/QPufAiJ2EIiIIAJMCzEGfgXhCE2VM5Lm89G6w3 NPkERSRlkjvdJCHQio9vfHmb4o5Nx0LPX16zDPZUjkCebKvuQEGdFo7YuDolfKrfvWjW xLHo2JreHc7XQ3WUeHbvKEJ5Ra4Ft/uBjrUjro1Ri2hExvoQgJRHAqyB2Ld8AE86JKyp ShxWG/JE8xu+hJDuKJvlkE92aIdPMMBSJtPJ8H5vb/S4XUhbP1xFk1es64Ekftqxtq1U j2gA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=UPa7blHQ; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id a67si2817635yba.200.2019.06.11.16.18.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:23 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=UPa7blHQ; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BNBnCu008431 for ; Tue, 11 Jun 2019 16:18:22 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=53Eg2QAwVCvd3SsQpm1bINM6H+u/Outwh4WdWfAbIvI=; b=UPa7blHQDDpA6oKRYk/eF2XbZhxMHkboJ0hHrxpb/j6G/eimP/p+zoW5df/5GP1Mv68r iyn9HKUFmrennZVSLesmK5RSSMNK/TF2YvD5THeLTsShOxvRiaGBplbBKZA9tz9oBzbn BWPsH//ZSmIWHPI/YgkSfUAHEoO1vL1EafM= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0b-00082601.pphosted.com with ESMTP id 2t2dkmsy3b-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Jun 2019 16:18:22 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:20 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 2A143130CBF67; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 01/10] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Date: Tue, 11 Jun 2019 16:18:04 -0700 Message-ID: <20190611231813.3148843-2-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Initialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params(). Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting. For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vladimir Davydov Acked-by: Johannes Weiner --- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 9e3eee5568b6..a4091f8b3655 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1239,7 +1239,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); + memcg_link_cache(kmem_cache, NULL); slab_state = PARTIAL; /* diff --git a/mm/slab.h b/mm/slab.h index 739099af6cbb..86f7ede21203 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -289,7 +289,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); +extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, void (*deact_fn)(struct kmem_cache *)); @@ -344,7 +344,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache(struct kmem_cache *s) +static inline void memcg_link_cache(struct kmem_cache *s, + struct mem_cgroup *memcg) { } diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..6e00bdf8618d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) } static int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct memcg_cache_array *arr; if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); return 0; @@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs) return ret; } -void memcg_link_cache(struct kmem_cache *s) +void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) { if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } #else static inline int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { return 0; } @@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, memcg, root_cache); + err = init_memcg_params(s, root_cache); if (err) goto out_free_cache; @@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, memcg); out: if (err) return ERR_PTR(err); @@ -997,7 +997,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); s->refcount = 1; return s; } diff --git a/mm/slub.c b/mm/slub.c index 1802c87799ff..9cb2eef62a37 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4213,7 +4213,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); return s; } From patchwork Tue Jun 11 23:18:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988343 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB5CD14DB for ; Tue, 11 Jun 2019 23:18:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA0D52881A for ; Tue, 11 Jun 2019 23:18:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE6F72890A; Tue, 11 Jun 2019 23:18:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3AB1C2881A for ; Tue, 11 Jun 2019 23:18:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1501B6B026E; Tue, 11 Jun 2019 19:18:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0B1EA6B026F; Tue, 11 Jun 2019 19:18:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2696B0270; Tue, 11 Jun 2019 19:18:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id AFA296B026E for ; Tue, 11 Jun 2019 19:18:31 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id u7so10633284pfh.17 for ; Tue, 11 Jun 2019 16:18:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=y8x91J89Hsw0QdQlhcd3vnyvb98p4X6XpzkxVy4k5LU=; b=Bo74WGVBXoDImK33N+KKoUhO/XC+iZOcoYgfRelO4qhxhIXEFcAZcEm6bMNUoPqiNy za+ZZVbx9n7104thOern/Ax5Bh4kLw5CgG6racI2dxz20cA3+oKGzTwx+yVQVQ+GC0f4 Kl5TVxnvb0C0yLcvYp8oAsYtxLCIWUm0mPKIA3MAYy6MUAeImRklklONKGUecPK8/jfg PQcx+Okhsft8Mp4QShnWaaFMsLGQ0kw9OAopM2TzdpvmXXc24YAohsv0lW1vEMGi8xxa /UxfxDjvzt5u+cToqo6+G00TLrFCHPAmInieN7kNAjmOfx15JL8MoiSUl/VXoAlhfLnK gs0A== X-Gm-Message-State: APjAAAXwCrBq9Vxp0aTNZSe/vbD+IMaVjPjo7m4doStfyt6ejbZrvF7p WwgIjCq4nKK9UoY4py2LwQcR1utxdV0N6NvtaQdTvgYSHKk/z8pdDYtCObHMkzPA9318NKB/Z3Z GvK0z5+KFm4Lml8vlpx3tXqG20eWocmfbO5H25wLCS9PQc6DQc0PGaTvlhxWCh1DI2A== X-Received: by 2002:a17:902:20e9:: with SMTP id v38mr37112133plg.62.1560295111274; Tue, 11 Jun 2019 16:18:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqxFzEiguxgh3Drv2n2mZInfZV0cj3sXDsVZClaYz7d5gxgHX0zZgCej1iSTXufLkzOWKdiX X-Received: by 2002:a17:902:20e9:: with SMTP id v38mr37112089plg.62.1560295110466; Tue, 11 Jun 2019 16:18:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295110; cv=none; d=google.com; s=arc-20160816; b=WglBY3Y3NDXXzBRGk1vKqNsUOHbQBDz8onIGUuE0FH1ixL7Mq16lLR0ci4l+jTsOAv A3A0mm0+lnOvWYJObm3wlZ0X7ztpHTUTMTxvFLe6Z+GPfcjaem5UfGAPk8bKRxqqyc05 nSOuu/tyfekn09uK+KPMGBaFuGCvVg3zdPlUQaiTfBzB8NvvfS4aIQ7iU9i544nowdv/ l8lP7BWw0VtP1aWI/2bUBAY/XJCfjuJB19dwU0hVFgKXtFKuZlOQTz6RzJlS+KdhhxMN au4jdlRVvbxZYo20dQA65vMeX0n2FW2meACB1P9u+VgyCzQC1Ro21YXtnI5KMpM1NcKX flXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=y8x91J89Hsw0QdQlhcd3vnyvb98p4X6XpzkxVy4k5LU=; b=DKRtLHbefkIXM0ALR7uAQJG3j6A5MOmLwvbMc4+SGM4VP4aguEVmJ+nGit/+i2Ml94 7o8IENrDV782UpP6+ljTg7YLILsvrow91p/h+Xs9Ej+jfbYz+1tC64J6sNMmIwhrnvOe 1ASQvKcGU9Qyajk6KB3Glkil/YVbb40tBWzbOXukuX7yjLFH71UbzdWoc1TpvS7Cajfx J11Wlfwe0Iq7pK1VtYzT30X9pnDbspDMn7W7/8p2rV1DtJhmBp5QpEj7mntn6cerlDrx lk7n/5xrbu238oP+xWUxjf36V8tAL88kmeJbexjLaWDw/V4tc1YQbXGxWRL6S0wYshLI JC/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=G2316pcB; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id r23si2185330pgk.126.2019.06.11.16.18.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:30 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=G2316pcB; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BN9aF0031322 for ; Tue, 11 Jun 2019 16:18:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=y8x91J89Hsw0QdQlhcd3vnyvb98p4X6XpzkxVy4k5LU=; b=G2316pcBitLm0xhNSOGwtMs4VFS7W8LMn6B4d6jVQ00LyWUySjzJ/QnPAXtvLvPWzx14 r1yxBZ7D8Ze0i4UT5VisczIizNG7xGiA54lkOCttqKQKUx7/R54ChtKV46AkQi4tj8NM /99uNAn0BTGWMUQUcWidfjy1DYIrMekUlxQ= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t2ha1926c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 11 Jun 2019 16:18:29 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 2CD40130CBF69; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 02/10] mm: rename slab delayed deactivation functions and fields Date: Tue, 11 Jun 2019 16:18:05 -0700 Message-ID: <20190611231813.3148843-3-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The delayed work/rcu deactivation infrastructure of non-root kmem_caches can be also used for asynchronous release of these objects. Let's get rid of the word "deactivation" in corresponding names to make the code look better after generalization. It's easier to make the renaming first, so that the generalized code will look consistent from scratch. Let's rename struct memcg_cache_params fields: deact_fn -> work_fn deact_rcu_head -> rcu_head deact_work -> work And RCU/delayed work callbacks in slab common code: kmemcg_deactivate_rcufn -> kmemcg_rcufn kmemcg_deactivate_workfn -> kmemcg_workfn This patch contains no functional changes, only renamings. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- include/linux/slab.h | 6 +++--- mm/slab.h | 2 +- mm/slab_common.c | 30 +++++++++++++++--------------- 3 files changed, 19 insertions(+), 19 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..47923c173f30 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -642,10 +642,10 @@ struct memcg_cache_params { struct list_head children_node; struct list_head kmem_caches_node; - void (*deact_fn)(struct kmem_cache *); + void (*work_fn)(struct kmem_cache *); union { - struct rcu_head deact_rcu_head; - struct work_struct deact_work; + struct rcu_head rcu_head; + struct work_struct work; }; }; }; diff --git a/mm/slab.h b/mm/slab.h index 86f7ede21203..7ef695b91919 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -291,7 +291,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)); + void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 6e00bdf8618d..99489d82ba78 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -691,17 +691,17 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, put_online_cpus(); } -static void kmemcg_deactivate_workfn(struct work_struct *work) +static void kmemcg_workfn(struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work); get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - s->memcg_params.deact_fn(s); + s->memcg_params.work_fn(s); mutex_unlock(&slab_mutex); @@ -712,36 +712,36 @@ static void kmemcg_deactivate_workfn(struct work_struct *work) css_put(&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn(struct rcu_head *head) +static void kmemcg_rcufn(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head); /* - * We need to grab blocking locks. Bounce to ->deact_work. The + * We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be * initialized eariler. */ - INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work); + INIT_WORK(&s->memcg_params.work, kmemcg_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } /** * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a * sched RCU grace period * @s: target kmem_cache - * @deact_fn: deactivation function to call + * @work_fn: deactivation function to call * - * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex + * Schedule @work_fn to be invoked with online cpus, mems and slab_mutex * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @deact_fn is finished. This is to be used from + * alive until @work_fn is finished. This is to be used from * __kmemcg_cache_deactivate(). */ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)) + void (*work_fn)(struct kmem_cache *)) { if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.deact_fn)) + WARN_ON_ONCE(s->memcg_params.work_fn)) return; if (s->memcg_params.root_cache->memcg_params.dying) @@ -750,8 +750,8 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; - call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + s->memcg_params.work_fn = work_fn; + call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) From patchwork Tue Jun 11 23:18:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988351 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B939A14DB for ; Tue, 11 Jun 2019 23:18:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A83382881A for ; Tue, 11 Jun 2019 23:18:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9C9C02890A; Tue, 11 Jun 2019 23:18:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 017252881A for ; Tue, 11 Jun 2019 23:18:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 375826B0272; Tue, 11 Jun 2019 19:18:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 288E36B0274; Tue, 11 Jun 2019 19:18:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1FAF6B0272; Tue, 11 Jun 2019 19:18:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id A586B6B0274 for ; Tue, 11 Jun 2019 19:18:35 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id r142so9360370pfc.2 for ; Tue, 11 Jun 2019 16:18:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=wvy1LTE78mhoSYH9KKkPh58hA9wYMnN4JWdHvONVtxM=; b=iic3lCOgjxWFiTCxaWN00l2M6VnoZ7t4Pi07M95DbmMrWJiduQKjfclsoo4j/zS4Nx mSSDqAw0UAzzbyEfqyQu/bEVpJy04r+UJZRde6Q9kTJX/I0s2NHEXrFjpcnhn0YtZqJg Td7U+c6UIY37zTDH8Sj96fAurY7pHifO19EuQqwF4a1qLIHcQRXNfO3WF38/PJOOjk9o 0czbOvJHy4OEAfv+amUbBM85aDuNSY87I/+IJhsHlKSXxrVYKHwxyzbP9nnQ0fPlEW5d KFmBOYp+F1KeBGmQpKzH2JHv0UulcZ1TmZGR6YxIx32tmGQDHGLeRMbAhpz552VylBuQ mDsw== X-Gm-Message-State: APjAAAU8N+/rDOnDsgGCz7YLalaXij3JYwSNSd4Yox1x2qzZxe5wYng4 KWyL0KYL6DQh09Ktlg5M4/y0yJTLiKUhSYju6Q7vVLZnGntk+Jd0wVbN6shmkVb6b5fIMf+++jU LcxFPntsp26qimm70SkxTL17K1uG/aRciv8e8z8qyK2dNNuFt2Phn5Vtt8ot1BfI8yA== X-Received: by 2002:a62:1c91:: with SMTP id c139mr75443181pfc.25.1560295115326; Tue, 11 Jun 2019 16:18:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqyn+awHncTrAOI6CJKRImoccRicBgTyuLYR4dMiQvJKQ7xkZuaGqv14u3qH05ErQ1tovzIw X-Received: by 2002:a62:1c91:: with SMTP id c139mr75443077pfc.25.1560295113817; Tue, 11 Jun 2019 16:18:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295113; cv=none; d=google.com; s=arc-20160816; b=aTI4eEW7BlQjFWgP7x4Itt3mgBCzYEUAfN0yVgKX8XxRVEnk+E1vZhXmudNQ1wR8Iq m0oRsVJ4XSFZ+KYgWjKNCTwYKDE7PTQNfRc1vABRT4ZY7TOIH7zkyjIsy831bs/bJTQu XDDFXsluPEqDxuTZvQeBTMMq1TlC8xaOHsdIetbm1qP2ugPee8rEce94RSYte4+CkOqw Rn3w+FL/y2UarAonm+rJKBWQ0RCgp3lN1G1IzPGz/7dOhLk11wIDBacLKLD1B+ln5JcR PrO1lJNHmHzY112y0vCewgoQip1D7issnkYMu5dDTNgbKRu/b1wUt319bSTOThfOOPHq lYEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=wvy1LTE78mhoSYH9KKkPh58hA9wYMnN4JWdHvONVtxM=; b=PWSBj+sSQw4bIwiTyv+QED9BOzQniymmsztfrHFIMpZXfRgx6esjPQId46PZtb9YtK jZZ9KMR5F8qIEqdn6sOBqCxsGmbKYlpAzYP1OWOggrbEPAzEv71QfZwEJRo6JelrTUuF YaehGzFEq8NHs/zjZoLcMkdAgAr0ndu2Z80OqwoWxdJh/BMledMd6eHJmFwxHoh3J28Y lCPNbyORXf7ztswElf5uM9LqTVX4IMC96k9lGJ+i3Inx08KxBhLIMMoPgTgnSOQ+v/XV 3jWnpDrlQIlVgil56RXioXvlIKXIbGP/ZSkJ3wuC0GKiGaP6HklWmG9PJp6kg+lAQvLn 4J/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LSsD9PMp; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id y17si13090218plp.59.2019.06.11.16.18.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:33 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=LSsD9PMp; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BN9aF7031322 for ; Tue, 11 Jun 2019 16:18:33 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=wvy1LTE78mhoSYH9KKkPh58hA9wYMnN4JWdHvONVtxM=; b=LSsD9PMpW8fbsnC/E4xQtfhl4K7ThMyR8L2jfRX5XAtFIJOzguxeNNi20t8xHS9gi+AF BoQ1bcDQMVcFdpTOHkadn/SrIM943nw0YShfdEIhZGgfvOY9IyTTPxkQGU+OqKGycW/M WTHtfuR62ysJI/7iNO7NcLAL+qauqwy8VMk= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t2ha1926c-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 11 Jun 2019 16:18:33 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:22 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 30FC6130CBF6B; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 03/10] mm: generalize postponed non-root kmem_cache deactivation Date: Tue, 11 Jun 2019 16:18:06 -0700 Message-ID: <20190611231813.3148843-4-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches release, but requires generalization for SLAB case. Introduce kmemcg_cache_deactivate() function, which calls allocator-specific __kmem_cache_deactivate() and schedules execution of __kmem_cache_deactivate_after_rcu() with all necessary locks in a worker context after an rcu grace period. Here is the new calling scheme: kmemcg_cache_deactivate() __kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_rcufn() rcu kmemcg_workfn() work __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specific instead of: __kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_rcufn() rcu kmemcg_workfn() work kmemcg_cache_deact_after_rcu() SLUB-only For consistency, all allocator-specific functions start with "__". Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- mm/slab.c | 4 ++++ mm/slab.h | 3 +-- mm/slab_common.c | 27 ++++++++------------------- mm/slub.c | 8 +------- 4 files changed, 14 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index a4091f8b3655..4b865393ebb4 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2252,6 +2252,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) { __kmem_cache_shrink(cachep); } + +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ +} #endif int __kmem_cache_shutdown(struct kmem_cache *cachep) diff --git a/mm/slab.h b/mm/slab.h index 7ef695b91919..dc83583ee9dd 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -290,8 +291,6 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); -extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 99489d82ba78..5e7638f495d1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -708,7 +708,7 @@ static void kmemcg_workfn(struct work_struct *work) put_online_mems(); put_online_cpus(); - /* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ + /* done, put the ref from kmemcg_cache_deactivate() */ css_put(&s->memcg_params.memcg->css); } @@ -726,31 +726,21 @@ static void kmemcg_rcufn(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } -/** - * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a - * sched RCU grace period - * @s: target kmem_cache - * @work_fn: deactivation function to call - * - * Schedule @work_fn to be invoked with online cpus, mems and slab_mutex - * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @work_fn is finished. This is to be used from - * __kmemcg_cache_deactivate(). - */ -void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*work_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate(struct kmem_cache *s) { if (WARN_ON_ONCE(is_root_cache(s)) || WARN_ON_ONCE(s->memcg_params.work_fn)) return; + __kmemcg_cache_deactivate(s); + if (s->memcg_params.root_cache->memcg_params.dying) return; /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.work_fn = work_fn; + s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); } @@ -773,7 +763,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) if (!c) continue; - __kmemcg_cache_deactivate(c); + kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } mutex_unlock(&slab_mutex); @@ -866,11 +856,10 @@ static void flush_memcg_workqueue(struct kmem_cache *s) mutex_unlock(&slab_mutex); /* - * SLUB deactivates the kmem_caches through call_rcu. Make + * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ - if (IS_ENABLED(CONFIG_SLUB)) - rcu_barrier(); + rcu_barrier(); /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB diff --git a/mm/slub.c b/mm/slub.c index 9cb2eef62a37..ae3b1e49ecec 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4022,7 +4022,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) } #ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. @@ -4048,12 +4048,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s) */ slub_set_cpu_partial(s, 0); s->min_partial = 0; - - /* - * s->cpu_partial is checked locklessly (see put_cpu_partial), so - * we have to make sure the change is visible before shrinking. - */ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); } #endif /* CONFIG_MEMCG */ From patchwork Tue Jun 11 23:18:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988345 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DAC7914E5 for ; Tue, 11 Jun 2019 23:18:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9EBF2881A for ; Tue, 11 Jun 2019 23:18:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE0E22890A; Tue, 11 Jun 2019 23:18:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3CE062881A for ; Tue, 11 Jun 2019 23:18:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 399906B026F; Tue, 11 Jun 2019 19:18:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 34B286B0270; Tue, 11 Jun 2019 19:18:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1528F6B0271; Tue, 11 Jun 2019 19:18:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id C08926B026F for ; Tue, 11 Jun 2019 19:18:33 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id k36so5579326pgl.7 for ; Tue, 11 Jun 2019 16:18:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=JhIs3/HCydWffB+uV1RGxA08DRLtLPsTTqEKxSMP1fY=; b=FPQyjv2cnWI6EAyxTmFSpModVy06aoloDB2bDSvKEEmv4/hByG4Ieg27Uv7uH7lNZs VBxMZGiOi1VW4cG32baIyoFnx46IAcLgU06ODkwYtCLkcH+ovgeMAYr9siN/0FP+oYGw GApoj9qmh+k4t8wPGkFR5Uk0kr4m43jGoMLSOmiNNpamUj2ioYRgxcLYJWwd0DWSePjZ 9a9leq4yM4+2Y8FkQku9xRZ/UCCxvJ63D3LcaPiIkVhiXdzOJLKMg4jcocJ4iF0mtVun CWwDycK1VuTlprWGHWFdT43DJe0J8AfpMg+Kq4GRbrUTtzbKGMOnErmhR9FND2b6IOX5 Vxvg== X-Gm-Message-State: APjAAAU5ZOvO9maACVJQ1L5svhViWiocchjF04vF43qzj92vG6Pwxmpa 8grL5/HheFrw+zq8gkmdUW/icqdTzbjCXwsgsG9f1oxyEpX/RkprC89NSKb4lO+ylk8ft8pFvL2 3QJBEXDGitboPNoceUb3n++AYpM26qkjXwDBF9fbkYDpHoyPkRQaCXlamVd1YOE3DsQ== X-Received: by 2002:a17:90a:208e:: with SMTP id f14mr29835233pjg.57.1560295113387; Tue, 11 Jun 2019 16:18:33 -0700 (PDT) X-Google-Smtp-Source: APXvYqwCcBDxKA/OlqIjEf3nLHenpBNSWwsXFz4BxDAoGX/lN0GDtrHFV6GGQ2ZOk8i1dlt4xC6n X-Received: by 2002:a17:90a:208e:: with SMTP id f14mr29835185pjg.57.1560295112586; Tue, 11 Jun 2019 16:18:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295112; cv=none; d=google.com; s=arc-20160816; b=G8D/Ij2P4Jf8tJAOftLwqQIaaxDUM6vjRuGTzINo4xwfxXc6RZ5rtYDlmsTgkWxB8T o7Zdc92+472E8ao6KWCLlCdq2/Bijem9PZ5838HyYX5ri9eE1bpl/7TMO9jIFZEnSivY E+7b1M6YQY1qiojOjk+9vU9FnHV7xvPv9t+MPKFUGhF/WVE36Swy/bjFLPwOoQEEHapM K5JIvme+iamgF+v2gqz0qKSNQAx9M+enR6dO0Ltk4E9wtQiAKudDIwg+GKPIWNXPMdCC vsIwtbJCMRtm8NO/6sJ6zoAcB7KAAUz5yTfQG9/PpnxxjygFaKJjKt0KaeVMXNMPYKYA H/bA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=JhIs3/HCydWffB+uV1RGxA08DRLtLPsTTqEKxSMP1fY=; b=k1E+3yiwQyVQhIlRsqwCWxU2q+eL8prUqBaJFmS1ycmon657R9uVByNgA+nn1aZ5uU t4ifNXvZbQOca0jh+SyBCpoxr0tIdJOhpEJqieSGzZLJNzWWLgOI5wHuspASN7seg/h8 7KGsw22VBPkpnTsLupgKxKl70ogftq/cByCnr9HEwJImeQk5EhoypvUwFpWX6g9Q3T8E v5fgTCslmlfUZM8XRF952ypVrmxhzCnbiwCdrTblYBSUk9IPUYxoz9FVxr3kSJO78oDK lN5603K791+hAlGdaEO02+C06Vf94iGsq0p7kft5OU9n/42Z5MF8ubnBANm0QpGoZ4pB dBRA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Zk3BS+Qo; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id h3si2721012pgv.403.2019.06.11.16.18.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:32 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Zk3BS+Qo; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BNAEP6024730 for ; Tue, 11 Jun 2019 16:18:32 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=JhIs3/HCydWffB+uV1RGxA08DRLtLPsTTqEKxSMP1fY=; b=Zk3BS+QoAeRg4WWCaz05ocuVThlNe41m7dPL12u8XO0/tdXd0hGBEwMvHnFYHMreoq+Z vKyranxiKgeotgqqiTLUQHMhhBrp9CLOU5qIcgjkIWnDW1xx41kANpgts1x+5GG0cFQP FmeXRqW/Nx9lpVsVLV4vcMTRwEiWmZEI4qQ= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t2n4j04ug-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Jun 2019 16:18:32 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 35E75130CBF6D; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 04/10] mm: introduce __memcg_kmem_uncharge_memcg() Date: Tue, 11 Jun 2019 16:18:07 -0700 Message-ID: <20190611231813.3148843-5-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=939 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work. This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_uncharge_memcg() if memcg_kmem_enabled() check is passed. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vladimir Davydov --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3ca57bacfdd2..9abf31bbe53a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1304,6 +1304,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge(struct page *page, int order); int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; @@ -1345,6 +1347,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, return __memcg_kmem_charge_memcg(page, gfp, order, memcg); return 0; } + +static inline void memcg_kmem_uncharge_memcg(struct page *page, int order, + struct mem_cgroup *memcg) +{ + if (memcg_kmem_enabled()) + __memcg_kmem_uncharge_memcg(memcg, 1 << order); +} + /* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function diff --git a/mm/memcontrol.c b/mm/memcontrol.c index be9344777b29..8eaf553b67f1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2812,6 +2812,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) css_put(&memcg->css); return ret; } + +/** + * __memcg_kmem_uncharge_memcg: uncharge a kmem page + * @memcg: memcg to uncharge + * @nr_pages: number of pages to uncharge + */ +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} /** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge @@ -2826,14 +2842,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) return; VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); - - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - + __memcg_kmem_uncharge_memcg(memcg, nr_pages); page->mem_cgroup = NULL; /* slab pages do not have PageKmemcg flag set */ From patchwork Tue Jun 11 23:18:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988339 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86CAA14E5 for ; Tue, 11 Jun 2019 23:18:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 743E32881A for ; Tue, 11 Jun 2019 23:18:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 68C332890A; Tue, 11 Jun 2019 23:18:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B34262881A for ; Tue, 11 Jun 2019 23:18:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 659146B0269; Tue, 11 Jun 2019 19:18:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 60CBB6B026B; Tue, 11 Jun 2019 19:18:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 481D26B026C; Tue, 11 Jun 2019 19:18:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 1E3C16B0269 for ; Tue, 11 Jun 2019 19:18:27 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id 75so9372323ywb.3 for ; Tue, 11 Jun 2019 16:18:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=qH2mK7ADNJ2npb+novhRBQUAsWfqGb1hE4WTZyKm8WI=; b=X123vEtMYA1a+N07tfdJ27r5XtApqazNw2WSzQBauHKFFCOzQjupBSKBHoPnzM/XxZ JGXALLcTvqaXgdAnyGWNliUq/1dQY0OaHcSt3XdKUV4eb9Tt+wiVYFutVk8izWdXm8cv eKwPPT19VJaUP4CUYyt1g5Ie+ugTQyzRpy4jTnt2UpgnF/7iqjrwt9+zf1dG6aKNYCAr zap9vuPvnBhrFl70X/7gLe/Cx2amEwtgyqCnifoJ3LqT65iweYCogk20/9wcB1NvJt3Z apZFCFdDNuhb4qennsWtnFckAdPlOXNTQ62fcOWsMXTWdM6fMPzbGEl8SliFcLGqGBrI ztmA== X-Gm-Message-State: APjAAAUvQ574Pr8DEzJGT1Z8lkgrXmENZ5g0AiKnwAbP+N6H3dvHVUdL pbdu0HN2mbBM871OA6qkmBLEcHCugEezts8x9dM5JovatuuMA/e1PeSaxeS6emrqNwAzafk4wjA tlj5viI0+oLmmQ/2rRyOTwgDhB3m887lyzeW3VsoQC4tEWwIsVT0N7iqCHuRonAcLxA== X-Received: by 2002:a81:148b:: with SMTP id 133mr29023487ywu.394.1560295106871; Tue, 11 Jun 2019 16:18:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqz5nIE+0wmIZsT6HJkyBMlpYDH9e5/b0egV2HaX/Asvo4kI3POmZJR1gloDNS2XsWVk2STB X-Received: by 2002:a81:148b:: with SMTP id 133mr29023451ywu.394.1560295106175; Tue, 11 Jun 2019 16:18:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295106; cv=none; d=google.com; s=arc-20160816; b=Kq5dktcHmQpX0KPSf2X31qX853kd5fe6Q5xklwitvY9glueJgacz+07jlpPU956jok xE2NlA6IwrhMO+emwFkJ+S84eO9GZRBPPx3jhQYoP8I1T/R7mmAJs6w0ZtY1fMG3uX+t sL/IkAOHHbnaqg9tei+6d31YpthEAiyNiNocKQaO51U0SszNa8IhxyMob4nFfjOG0h/D fHwnSRlZJZ4mPqbHJJWVvCaXnSRiaXJ5heS3FlJcYq+bFMEMHZ8xCYnntPdG5JEyW5n2 Cje8oVyo8vdb4nt8hnzUMV6+3phmupyIyhWI/yaRtsaZ75vW5ArUENhT6B7v5+KucR8z 5sqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=qH2mK7ADNJ2npb+novhRBQUAsWfqGb1hE4WTZyKm8WI=; b=pK0nXQJGSm1o7C6jIHtm5nnRKKybgVVZmIwJ92GTbbaXcsRg8nhDHknsCMIZ84fVW/ 5/0pLVQGbp/oPrf33o4Rv024+eA5PqK7qujJezer26tzZkfEzCQSUaYhC3UWZc+RQEsZ YmLE7TNgJbmqZsh/LhBX4gXxNt0uTXPzGkgV86VUUgu7Uep2pSCzOzalgRKkWGyvQqRt MFAAiNxics/7vTTbPCVsGloMabnfgbMm/qLzgNlDHGTgY4BnbKzjh99gn1IOuN9sR+29 TDo/bI5q26bfoAL4o6OPU97wQmy8/UhMb4A/RO4SGv5iLYqNczO6gYYaprDotXUN9uAL h1sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Z0bvFpiu; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id b5si5016487ywf.221.2019.06.11.16.18.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:26 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=Z0bvFpiu; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BNBnD4008431 for ; Tue, 11 Jun 2019 16:18:26 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=qH2mK7ADNJ2npb+novhRBQUAsWfqGb1hE4WTZyKm8WI=; b=Z0bvFpiudRjXKo4lqflXk3QO0pA8BeE7EHfgcY+YiyX1VvUivQx92uxfFm+t8YnHrsRg S7A25vFea7iz3ZQq64EMk2FbRpNBIpmupB6aLFiY8x+MA6eLdN+2gZY/xwmVtndcZjcr jdfqQla+d3aCNp2fr3iDPym1O2jLiAwIQrU= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0b-00082601.pphosted.com with ESMTP id 2t2dkmsy3b-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Jun 2019 16:18:25 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 3A2DC130CBF6F; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 05/10] mm: unify SLAB and SLUB page accounting Date: Tue, 11 Jun 2019 16:18:08 -0700 Message-ID: <20190611231813.3148843-6-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the page accounting code is duplicated in SLAB and SLUB internals. Let's move it into new (un)charge_slab_page helpers in the slab_common.c file. These helpers will be responsible for statistics (global and memcg-aware) and memcg charging. So they are replacing direct memcg_(un)charge_slab() calls. Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Christoph Lameter Acked-by: Vladimir Davydov Acked-by: Johannes Weiner --- mm/slab.c | 19 +++---------------- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slub.c | 14 ++------------ 3 files changed, 30 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 4b865393ebb4..b417824a9b15 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1360,7 +1360,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { struct page *page; - int nr_pages; flags |= cachep->allocflags; @@ -1370,17 +1369,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - if (memcg_charge_slab(page, flags, cachep->gfporder, cachep)) { + if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { __free_pages(page, cachep->gfporder); return NULL; } - nr_pages = (1 << cachep->gfporder); - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages); - __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1395,12 +1388,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - unsigned long nr_freed = (1 << order); - - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed); BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); @@ -1409,8 +1396,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping = NULL; if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; - memcg_uncharge_slab(page, order, cachep); + current->reclaim_state->reclaimed_slab += 1 << order; + uncharge_slab_page(page, order, cachep); __free_pages(page, order); } diff --git a/mm/slab.h b/mm/slab.h index dc83583ee9dd..46623a576a3c 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer, void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **); int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **); +static inline int cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; +} + #ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */ @@ -361,6 +367,25 @@ static inline struct kmem_cache *virt_to_cache(const void *obj) return page->slab_cache; } +static __always_inline int charge_slab_page(struct page *page, + gfp_t gfp, int order, + struct kmem_cache *s) +{ + int ret = memcg_charge_slab(page, gfp, order, s); + + if (!ret) + mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + + return ret; +} + +static __always_inline void uncharge_slab_page(struct page *page, int order, + struct kmem_cache *s) +{ + mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + memcg_uncharge_slab(page, order, s); +} + static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) { struct kmem_cache *cachep; diff --git a/mm/slub.c b/mm/slub.c index ae3b1e49ecec..6a5174b51cd6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1488,7 +1488,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, else page = __alloc_pages_node(node, flags, order); - if (page && memcg_charge_slab(page, flags, order, s)) { + if (page && charge_slab_page(page, flags, order, s)) { __free_pages(page, order); page = NULL; } @@ -1681,11 +1681,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) if (!page) return NULL; - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - 1 << oo_order(oo)); - inc_slabs_node(s, page_to_nid(page), page->objects); return page; @@ -1719,18 +1714,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page) check_object(s, page, p, SLUB_RED_INACTIVE); } - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - -pages); - __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - memcg_uncharge_slab(page, order, s); + uncharge_slab_page(page, order, s); __free_pages(page, order); } From patchwork Tue Jun 11 23:18:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988335 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A976714E5 for ; Tue, 11 Jun 2019 23:18:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 990942881A for ; Tue, 11 Jun 2019 23:18:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8CEF32890A; Tue, 11 Jun 2019 23:18:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A46A2881A for ; Tue, 11 Jun 2019 23:18:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE5196B000E; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B85F46B026B; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 963866B0010; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id 6A7616B000E for ; Tue, 11 Jun 2019 19:18:24 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id z4so14031786ybo.4 for ; Tue, 11 Jun 2019 16:18:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=nplAGvztvw22AKN56QTMhELHdT1v5h5cxoHCBLrrDqI=; b=Wgbha9YNwxewOWDhuv55Ml7NHS9uV0S8HYrj1PBkkSQ/DHc7zsAo0MZ9qvBf6AtSoI 09auUxnG5eWnefHH7CYxtR+KeQBmBMgNQtsDdCmJ4hhMmm0J1y4sTn5gal1J6oF0meUz jR5HUuQgyDW0DWa5kHzg8cQIw8bklE/QpwrPbvedaObu/uVqv+mB7QSBXms5XHTnlkIZ UqalJZKi68+VP9MPM+jNDJfO0UqZLiedWYENg413Scls2iillaNUoiyAllydg8M/nY4j jiMn6kEcLeMPWj6lLxjykXXp8STthDwcfqhHOMeQDnoRpCHf+kSaQXMjM6fgzLUylrQU WZfA== X-Gm-Message-State: APjAAAXN42IPToEaEct6mzXOyloMiFIXdTH4rTvbvxNdfYPzr4gWMJuE KrJm95wEdItoczRbe5qMf07UEAD+xpMtTtfspjzCfBqDEC5Az1r9zNL1025xNqedlusYWhxGaTN hplTwntEl4PIzJ1EAi+TcZfRq+XHqEkIfgVf/MIDA3iobpbMaI9SR40bbWUTE279rOQ== X-Received: by 2002:a25:ef43:: with SMTP id w3mr37970268ybm.411.1560295104072; Tue, 11 Jun 2019 16:18:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqxhMXPNE6ia+3g6BWUCgIIx7vzgvsJ2964VFFep1dmtVrrjS+mNazHLJU9CYR0P3e2Qb52t X-Received: by 2002:a25:ef43:: with SMTP id w3mr37970248ybm.411.1560295103350; Tue, 11 Jun 2019 16:18:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295103; cv=none; d=google.com; s=arc-20160816; b=RFT0ykg11BQyj+D1zCH31e5iCyfIObZm3Ev0Tt1rpkthD8KS9z/lqxaOEhGZ8ZfgK0 KwWwDsgXqKx3GvXfpNk0Nzt1vzcbwJNr61SZyoXTIg0UsgQOG1eD5FA5m5QMvXdBfoXB ZHQ336D9kae2XNqZf+raep5maip871AAJ9fXhwT9REN6PkJL1P301FuXea0aCZAIgCUi mZgPgH/WISJjyySQyKu4xoNLLwxmCPO8XYTlKNN7thr2V/UpOF5qdf/H+1d40C9QLNT3 iJMHkH/n8G6siJ5+bDHFOlRHVBMWUuCnjWQ9wbr48rmaieHC6W4fDIA8hRl55GkztwAR Qyjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=nplAGvztvw22AKN56QTMhELHdT1v5h5cxoHCBLrrDqI=; b=wgsIrm+Dt+0aF7x6pSkeQk3cr1R1CDrK4HOdCvMe8Ztvz3BQsBExdDQvkAbYp0YS5M 1WeLasvbKNRq8USaw7PA871dl+JMAi89BTL9TtTj8vgSQkGAefhvuZCPqCpb7DWSB0VE vFT5qHhNjBPzElgFVyNyaaCaoLJS8yftMztc+azqXQ7Dnko1VesFvLCwrU95rhpY+imA PPRbm9LPhvBufjdQXTjqWxaLgeEwhPEp0zBzBn+pTzwDBxZTbj9pNZUGL8UjvkG+GcAJ 3nlMxD7yUXj4fAQcS0Im3m5M0L72nxBInrax6LM7N9Ua2/vdrLosWYyQVimqy8NsRUis XDcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=nGjePryA; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id l80si4736539ywl.337.2019.06.11.16.18.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:23 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=nGjePryA; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BN8Ubt022343 for ; Tue, 11 Jun 2019 16:18:23 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=nplAGvztvw22AKN56QTMhELHdT1v5h5cxoHCBLrrDqI=; b=nGjePryAVos7TNaTWm7Id2mKsFVe++WTr9Nx9tvWflQTJRXzdEZ7izOhgXCWOrU/hMd0 6nQ0ltd9AFZiFoYBID3dSUrtudltOGCtZ5QcdFLz1SG3Fb3izAuKX7H2AHxa0jysu9Yi M8M71Rx5vvpj7yVk3ONEglGL1MVrxKcR5iE= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t2f4rhk2p-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Jun 2019 16:18:23 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 3ED11130CBF71; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 06/10] mm: don't check the dying flag on kmem_cache creation Date: Tue, 11 Jun 2019 16:18:09 -0700 Message-ID: <20190611231813.3148843-7-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=777 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There is no point in checking the root_cache->memcg_params.dying flag on kmem_cache creation path. New allocations shouldn't be performed using a dead root kmem_cache, so no new memcg kmem_cache creation can be scheduled after the flag is set. And if it was scheduled before, flush_memcg_workqueue() will wait for it anyway. So let's drop this check to simplify the code. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- mm/slab_common.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 5e7638f495d1..9383104651cd 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -640,7 +640,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, * The memory cgroup could have been offlined while the cache * creation work was pending. */ - if (memcg->kmem_state != KMEM_ONLINE || root_cache->memcg_params.dying) + if (memcg->kmem_state != KMEM_ONLINE) goto out_unlock; idx = memcg_cache_id(memcg); From patchwork Tue Jun 11 23:18:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988341 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8FE314E5 for ; Tue, 11 Jun 2019 23:18:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D859F2881A for ; Tue, 11 Jun 2019 23:18:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CCA032890A; Tue, 11 Jun 2019 23:18:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59AD72881A for ; Tue, 11 Jun 2019 23:18:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BAF5D6B026C; Tue, 11 Jun 2019 19:18:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B60A36B026D; Tue, 11 Jun 2019 19:18:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C9036B026E; Tue, 11 Jun 2019 19:18:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id 62D5C6B026C for ; Tue, 11 Jun 2019 19:18:30 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id f69so14383168ywb.21 for ; Tue, 11 Jun 2019 16:18:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=Xx5Luiw4ossPD+e7sUfpgznJxtqNhDquLK7q4vM5gv8=; b=ij0cJvRF+zNTBSr1c32DcWrLVEaT6W47q9ZlynnIdIa8ucb5+sRzufdmi4KxLxm4qF vOT2+DYbIe24Z9iWvADew7prXnLWOLk5UE3PG0Y1dBVJ767kKGoMYjaiTtlwpPsw2oHo 8A03jnbspFTbeWHCbxTaoJOp8b6uFBPsImffuMWdRRIzq/6gwEQnzyEzsEHaTZq1C4oT NQmof/ODT+zDjc2ACMQWPFbSZBgzvmOo5KYWny1wqo5UfRQ2/R/TGOwlkyZPmyQuv+/q gUo6tP4PHGeUcW2pRBZyz/nU+StbtH4aMHWiwcJ/36FipoaHijX7QXKXxyyNj6edwbL8 opwA== X-Gm-Message-State: APjAAAXW4wo+5XtsS5FPZke/hAQNrx/UvN74Q3BJFYdUPMIxoRl37Ute VOq1lBBRMleiFyn/cB8elbIiSCEMkuS/nBbZpcvEZgzU2r+rf+jsm/CU7Lrw/NIZbVfbjoo5tb1 ElXItTH11XwTNcQpZkYstFbMHGDiKlcEG1CGBRJ4XvWxHDoQhbSACkv2/jE1OeYFWwA== X-Received: by 2002:a0d:ea91:: with SMTP id t139mr34324223ywe.119.1560295110164; Tue, 11 Jun 2019 16:18:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqw6TKAD3z5VX7b8LMuSxS38UkUZxM1H2yRWKran40pPt0SoA2E3SrofMaYtJBXNuJ2Ay5wt X-Received: by 2002:a0d:ea91:: with SMTP id t139mr34324203ywe.119.1560295109655; Tue, 11 Jun 2019 16:18:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295109; cv=none; d=google.com; s=arc-20160816; b=EAjdyKsKT5z6zMX6EK8FJRlvqnZ+Xl+846X5B2Qw5R+d5jyg3UciwuZwyJZrb9uwxc jGExaQ7xCaW5IAU7CPLM6/lcTRSRDL8Pxno075ZvBKC7POxQnyzzVGLyYH8VEGumRwZ3 u6ijlWYn9L/Q9MqrzMY0Qdz+Nuka4Bedc8Op8/eykR4JegiLyEXUuZySlXG3dh9NrBGF kjxzjH8HUxUn/eXxCKAeWUDwhIggqGMq4zGOL0q/4eS0PiXhFfSjTF02FMh6p6qNy+8K vZ1w0tS3KRHMaMN4kgjhWJeGB2Pq1i3CObU0EfrWIR/7TibskihgHq9jsZhgWum+8YrX VF8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=Xx5Luiw4ossPD+e7sUfpgznJxtqNhDquLK7q4vM5gv8=; b=oAqLPeoLkhsDYd55NTneW8WYyo80HagFsULvi1MpeXKZbqOyNlzvtG3HBmCrpfHfmW VOezkvkQH6KQVUL8wBJ1yyFdlgsLKmGckpKx1vePq1+6HutAtWOdBpOpBxTBdvuOjoGI fiFr1y9jwe28ysKVyc9ywkp0MHE+uGLcGHhnYygtfYHHgsrIct/QP5D5iu9/77DgKaaw nMd+jXqYfEN2RiPArsuUFbEPQy5x8IrLictAcW95v90IYppfWEPapEcSCNqCLRhNY4KF kMqLhMGsu/XM5Bwkxv1fMV5vWloVdpYaN48mSd7DMkjksTVbmF1DIP84tlMhR/rtlSe+ w7QA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=A657KbJo; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id 206si4531913ywv.82.2019.06.11.16.18.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:29 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=A657KbJo; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BNBnDC008431 for ; Tue, 11 Jun 2019 16:18:29 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=Xx5Luiw4ossPD+e7sUfpgznJxtqNhDquLK7q4vM5gv8=; b=A657KbJouvq/Ev7aTKFK7x7621+5wRCN625mZI/d0duLQMf3Y0bLvGhFDQTyvbDjFDij I0L3vq6qbr2k/DGA8SpeLWInH4njDu4xksjnSt/AlX68oOG+nGwkwtx9ZsNp/0jmB2YT ftJFna0Bi2j88VEk1lI8RiIdyVr5ebgANFU= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0b-00082601.pphosted.com with ESMTP id 2t2dkmsy3b-18 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Jun 2019 16:18:29 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 42C22130CBF73; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 07/10] mm: synchronize access to kmem_cache dying flag using a spinlock Date: Tue, 11 Jun 2019 16:18:10 -0700 Message-ID: <20190611231813.3148843-8-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=734 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the memcg_params.dying flag and the corresponding workqueue used for the asynchronous deactivation of kmem_caches is synchronized using the slab_mutex. It makes impossible to check this flag from the irq context, which will be required in order to implement asynchronous release of kmem_caches. So let's switch over to the irq-save flavor of the spinlock-based synchronization. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- mm/slab_common.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 9383104651cd..1e5eaf84bf08 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -130,6 +130,7 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr, #ifdef CONFIG_MEMCG_KMEM LIST_HEAD(slab_root_caches); +static DEFINE_SPINLOCK(memcg_kmem_wq_lock); void slab_init_memcg_params(struct kmem_cache *s) { @@ -734,14 +735,22 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) __kmemcg_cache_deactivate(s); + /* + * memcg_kmem_wq_lock is used to synchronize memcg_params.dying + * flag and make sure that no new kmem_cache deactivation tasks + * are queued (see flush_memcg_workqueue() ). + */ + spin_lock_irq(&memcg_kmem_wq_lock); if (s->memcg_params.root_cache->memcg_params.dying) - return; + goto unlock; /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); +unlock: + spin_unlock_irq(&memcg_kmem_wq_lock); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) @@ -851,9 +860,9 @@ static int shutdown_memcg_caches(struct kmem_cache *s) static void flush_memcg_workqueue(struct kmem_cache *s) { - mutex_lock(&slab_mutex); + spin_lock_irq(&memcg_kmem_wq_lock); s->memcg_params.dying = true; - mutex_unlock(&slab_mutex); + spin_unlock_irq(&memcg_kmem_wq_lock); /* * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make From patchwork Tue Jun 11 23:18:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988337 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DBD6714DB for ; Tue, 11 Jun 2019 23:18:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CA1FB2881A for ; Tue, 11 Jun 2019 23:18:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE1212890A; Tue, 11 Jun 2019 23:18:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 976112881A for ; Tue, 11 Jun 2019 23:18:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71DE96B0010; Tue, 11 Jun 2019 19:18:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6CC8C6B0269; Tue, 11 Jun 2019 19:18:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51DCE6B026B; Tue, 11 Jun 2019 19:18:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 1233C6B0010 for ; Tue, 11 Jun 2019 19:18:25 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id 14so10056657pgo.14 for ; Tue, 11 Jun 2019 16:18:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=cVERFm6oPDK3UMABxkEQ0fgUIK7kTz8oGAMIws2M7BA=; b=JdugSU8hf5mo4/QvcTDIB9TsN0zKJNSLzwMumpfrwz8SCtp9hX6Sfx8oG7aAkco/F1 pqpR80NaegrIhV1agaOIBZJ7pcE0YfuCmLTyidkk3cbza/jcZo540MM+QxbjnLmMmmkT mgh78WnBgJt0oX+4z+kWd3X2TUHXTP6iC1+P/sbnek9CSeEhaUZQt3/BU8t3eQW44+w7 Y8ejVTtW6QYkAdOexu6lgyytL5iY0mIt8zT1FQwB1x98XCmLv6u6eGGNJY+PHnD0RV60 Hm7PHy902zQ6yoqjZ9/4pK6xaB/n/46Z3agq4qjCqQEgiAc72Oj1BatF2ZTr0FFuMlAW By0Q== X-Gm-Message-State: APjAAAXilBe6vrZew27KMELIAHEbWy5n3Q4v+G4dMupdWlOuzsuGDbIz KRN6M+g2IyBS4elxv+0sDOlIpiOT/6xjjBkZgB5BaWoqwYtGLy9P6dmZ/xGiDG3Do4/sJGsj8ui m7d/57xti16Pfhaz84a0zw9ljtqCglaNM333y8tUiMkcs3/+hVVnHjAMHglkIuKepAQ== X-Received: by 2002:a17:902:6bcb:: with SMTP id m11mr50841697plt.318.1560295104677; Tue, 11 Jun 2019 16:18:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqz/8DLTSLtIyJ0ibK9wHx9tjKV+UPDonRf/a4f7uAqzAJrDPrkG+U4NU9DO1EkSAqOp6F79 X-Received: by 2002:a17:902:6bcb:: with SMTP id m11mr50841627plt.318.1560295103324; Tue, 11 Jun 2019 16:18:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295103; cv=none; d=google.com; s=arc-20160816; b=wZQFfiFptYUnKsoKNBiTkKCS4jsILHMAGorSawcR/KYOFFJAKwpFDLWQNCFN8zlZAk 0qzqyQnnPnWQ28UCiDFEi+b5zimBxJv8TtB2VUh81KlJKNA5rXZWs3st+4bhDvgsuRGp 7MDS+75RnBWV9kvEvSPuvTQMATMWj/xZNlC4541rCCbt6i9W5IQ8YsCa4Ky8G9ILgNYs Miru6CmqCD3VQQlwIDIr/AOkZFMqHa/Jc1xin2LeiV/h3y6cqIsVeye0I94gpbVlc7x2 IEVXlLQVbPQ36+YeSO5rsYlD0Naqfv8yrkpVgnbiTHWYOoLhiK2mec3zgEGj9VDd3jVJ odIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=cVERFm6oPDK3UMABxkEQ0fgUIK7kTz8oGAMIws2M7BA=; b=y3aYM0zUpGpOjYfbaiDO5PHEzEmQK7sbuTAd5l8LjAQxDyHBLZtLSxrbH5OizKGCwg HUSELtZM3/bVIEtXS0z5UZT5ffDZEHhXjxvT2UInPgBd+JePNm2y+ZW7BQRrJLOars/S /Bhg8knJk384xU4ASsa4+nRa2W3uzn24CwjoBNok+IVG9YhFgFc4FN6OJWKde7MF4Pku Xy4HZllOqfFj3uVnXTiawUXrBUNwYnWNY2tfiQazwgyqZJUujZowg0IK95BkAaBvoRAi nCZhnQFg0u6gcwG1owWpzhOeR30M15JvrlfGHwVxeU2qg1PWVkz36bX2+2IlDHkKbtSX 2lbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=XgpLqiWs; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id k9si5585152pfh.56.2019.06.11.16.18.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:23 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=XgpLqiWs; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BN8HYO026256 for ; Tue, 11 Jun 2019 16:18:22 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=cVERFm6oPDK3UMABxkEQ0fgUIK7kTz8oGAMIws2M7BA=; b=XgpLqiWsiO3p0N4uRswthR0JG7u+FU1Nt+CsmmC61aubWzG6NjJOMhSAUpRoZEP23sZF yYwLPZecyQsgh41reeDMn1VSVlMHjw1iEASX6FYQE6UMsGtBTzwI0wel1pRF5SqcNuPW PF4O4MxRU/Jytnj9z1BnLaX43De2oA2xxkg= Received: from mail.thefacebook.com (mailout.thefacebook.com [199.201.64.23]) by mx0a-00082601.pphosted.com with ESMTP id 2t2jma8q5s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Jun 2019 16:18:22 -0700 Received: from mx-out.facebook.com (2620:10d:c081:10::13) by mail.thefacebook.com (2620:10d:c081:35::127) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 46A68130CBF75; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 08/10] mm: rework non-root kmem_cache lifecycle management Date: Tue, 11 Jun 2019 16:18:11 -0700 Message-ID: <20190611231813.3148843-9-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently each charged slab page holds a reference to the cgroup to which it's charged. Kmem_caches are held by the memcg and are released all together with the memory cgroup. It means that none of kmem_caches are released unless at least one reference to the memcg exists, which is very far from optimal. Let's rework it in a way that allows releasing individual kmem_caches as soon as the cgroup is offline, the kmem_cache is empty and there are no pending allocations. To make it possible, let's introduce a new percpu refcounter for non-root kmem caches. The counter is initialized to the percpu mode, and is switched to the atomic mode during kmem_cache deactivation. The counter is bumped for every charged page and also for every running allocation. So the kmem_cache can't be released unless all allocations complete. To shutdown non-active empty kmem_caches, let's reuse the work queue, previously used for the kmem_cache deactivation. Once the reference counter reaches 0, let's schedule an asynchronous kmem_cache release. * I used the following simple approach to test the performance (stolen from another patchset by T. Harding): time find / -name fname-no-exist echo 2 > /proc/sys/vm/drop_caches repeat 10 times Results: orig patched real 0m1.455s real 0m1.355s user 0m0.206s user 0m0.219s sys 0m0.855s sys 0m0.807s real 0m1.487s real 0m1.699s user 0m0.221s user 0m0.256s sys 0m0.806s sys 0m0.948s real 0m1.515s real 0m1.505s user 0m0.183s user 0m0.215s sys 0m0.876s sys 0m0.858s real 0m1.291s real 0m1.380s user 0m0.193s user 0m0.198s sys 0m0.843s sys 0m0.786s real 0m1.364s real 0m1.374s user 0m0.180s user 0m0.182s sys 0m0.868s sys 0m0.806s real 0m1.352s real 0m1.312s user 0m0.201s user 0m0.212s sys 0m0.820s sys 0m0.761s real 0m1.302s real 0m1.349s user 0m0.205s user 0m0.203s sys 0m0.803s sys 0m0.792s real 0m1.334s real 0m1.301s user 0m0.194s user 0m0.201s sys 0m0.806s sys 0m0.779s real 0m1.426s real 0m1.434s user 0m0.216s user 0m0.181s sys 0m0.824s sys 0m0.864s real 0m1.350s real 0m1.295s user 0m0.200s user 0m0.190s sys 0m0.842s sys 0m0.811s So it looks like the difference is not noticeable in this test. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- include/linux/slab.h | 3 +- mm/memcontrol.c | 50 +++++++++++++++++++++------- mm/slab.h | 44 +++++++----------------- mm/slab_common.c | 79 ++++++++++++++++++++++++++------------------ 4 files changed, 99 insertions(+), 77 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 47923c173f30..1b54e5f83342 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include /* @@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); -void memcg_destroy_kmem_caches(struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -641,6 +641,7 @@ struct memcg_cache_params { struct mem_cgroup *memcg; struct list_head children_node; struct list_head kmem_caches_node; + struct percpu_ref refcnt; void (*work_fn)(struct kmem_cache *); union { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8eaf553b67f1..43a42bc3ed3f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2672,12 +2672,13 @@ static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg, { struct memcg_kmem_cache_create_work *cw; + if (!css_tryget_online(&memcg->css)) + return; + cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN); if (!cw) return; - css_get(&memcg->css); - cw->memcg = memcg; cw->cachep = cachep; INIT_WORK(&cw->work, memcg_kmem_cache_create_func); @@ -2712,6 +2713,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) { struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; + struct memcg_cache_array *arr; int kmemcg_id; VM_BUG_ON(!is_root_cache(cachep)); @@ -2719,14 +2721,28 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) if (memcg_kmem_bypass()) return cachep; - memcg = get_mem_cgroup_from_current(); + rcu_read_lock(); + + if (unlikely(current->active_memcg)) + memcg = current->active_memcg; + else + memcg = mem_cgroup_from_task(current); + + if (!memcg || memcg == root_mem_cgroup) + goto out_unlock; + kmemcg_id = READ_ONCE(memcg->kmemcg_id); if (kmemcg_id < 0) - goto out; + goto out_unlock; + + arr = rcu_dereference(cachep->memcg_params.memcg_caches); - memcg_cachep = cache_from_memcg_idx(cachep, kmemcg_id); - if (likely(memcg_cachep)) - return memcg_cachep; + /* + * Make sure we will access the up-to-date value. The code updating + * memcg_caches issues a write barrier to match the data dependency + * barrier inside READ_ONCE() (see memcg_create_kmem_cache()). + */ + memcg_cachep = READ_ONCE(arr->entries[kmemcg_id]); /* * If we are in a safe context (can wait, and not in interrupt @@ -2739,10 +2755,20 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) * memcg_create_kmem_cache, this means no further allocation * could happen with the slab_mutex held. So it's better to * defer everything. + * + * If the memcg is dying or memcg_cache is about to be released, + * don't bother creating new kmem_caches. Because memcg_cachep + * is ZEROed as the fist step of kmem offlining, we don't need + * percpu_ref_tryget_live() here. css_tryget_online() check in + * memcg_schedule_kmem_cache_create() will prevent us from + * creation of a new kmem_cache. */ - memcg_schedule_kmem_cache_create(memcg, cachep); -out: - css_put(&memcg->css); + if (unlikely(!memcg_cachep)) + memcg_schedule_kmem_cache_create(memcg, cachep); + else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) + cachep = memcg_cachep; +out_unlock: + rcu_read_unlock(); return cachep; } @@ -2753,7 +2779,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) void memcg_kmem_put_cache(struct kmem_cache *cachep) { if (!is_root_cache(cachep)) - css_put(&cachep->memcg_params.memcg->css); + percpu_ref_put(&cachep->memcg_params.refcnt); } /** @@ -3300,7 +3326,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) memcg_offline_kmem(memcg); if (memcg->kmem_state == KMEM_ALLOCATED) { - memcg_destroy_kmem_caches(memcg); + WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); WARN_ON(page_counter_read(&memcg->kmem)); } diff --git a/mm/slab.h b/mm/slab.h index 46623a576a3c..5d2b8511e6fb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -248,31 +248,6 @@ static inline const char *cache_name(struct kmem_cache *s) return s->name; } -/* - * Note, we protect with RCU only the memcg_caches array, not per-memcg caches. - * That said the caller must assure the memcg's cache won't go away by either - * taking a css reference to the owner cgroup, or holding the slab_mutex. - */ -static inline struct kmem_cache * -cache_from_memcg_idx(struct kmem_cache *s, int idx) -{ - struct kmem_cache *cachep; - struct memcg_cache_array *arr; - - rcu_read_lock(); - arr = rcu_dereference(s->memcg_params.memcg_caches); - - /* - * Make sure we will access the up-to-date value. The code updating - * memcg_caches issues a write barrier to match this (see - * memcg_create_kmem_cache()). - */ - cachep = READ_ONCE(arr->entries[idx]); - rcu_read_unlock(); - - return cachep; -} - static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) { if (is_root_cache(s)) @@ -284,14 +259,25 @@ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { + int ret; + if (is_root_cache(s)) return 0; - return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + + ret = memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + if (ret) + return ret; + + percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + + return 0; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { + if (!is_root_cache(s)) + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); memcg_kmem_uncharge(page, order); } @@ -323,12 +309,6 @@ static inline const char *cache_name(struct kmem_cache *s) return s->name; } -static inline struct kmem_cache * -cache_from_memcg_idx(struct kmem_cache *s, int idx) -{ - return NULL; -} - static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) { return s; diff --git a/mm/slab_common.c b/mm/slab_common.c index 1e5eaf84bf08..6b7750f7ea33 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -132,6 +132,8 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr, LIST_HEAD(slab_root_caches); static DEFINE_SPINLOCK(memcg_kmem_wq_lock); +static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref); + void slab_init_memcg_params(struct kmem_cache *s) { s->memcg_params.root_cache = NULL; @@ -146,6 +148,12 @@ static int init_memcg_params(struct kmem_cache *s, struct memcg_cache_array *arr; if (root_cache) { + int ret = percpu_ref_init(&s->memcg_params.refcnt, + kmemcg_cache_shutdown, + 0, GFP_KERNEL); + if (ret) + return ret; + s->memcg_params.root_cache = root_cache; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); @@ -171,6 +179,8 @@ static void destroy_memcg_params(struct kmem_cache *s) { if (is_root_cache(s)) kvfree(rcu_access_pointer(s->memcg_params.memcg_caches)); + else + percpu_ref_exit(&s->memcg_params.refcnt); } static void free_memcg_params(struct rcu_head *rcu) @@ -226,6 +236,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + css_get(&memcg->css); s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); @@ -241,6 +252,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); + css_put(&s->memcg_params.memcg->css); } } #else @@ -678,7 +690,7 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, } /* - * Since readers won't lock (see cache_from_memcg_idx()), we need a + * Since readers won't lock (see memcg_kmem_get_cache()), we need a * barrier here to ensure nobody will see the kmem_cache partially * initialized. */ @@ -703,14 +715,12 @@ static void kmemcg_workfn(struct work_struct *work) mutex_lock(&slab_mutex); s->memcg_params.work_fn(s); + s->memcg_params.work_fn = NULL; mutex_unlock(&slab_mutex); put_online_mems(); put_online_cpus(); - - /* done, put the ref from kmemcg_cache_deactivate() */ - css_put(&s->memcg_params.memcg->css); } static void kmemcg_rcufn(struct rcu_head *head) @@ -727,10 +737,39 @@ static void kmemcg_rcufn(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } +static void kmemcg_cache_shutdown_fn(struct kmem_cache *s) +{ + WARN_ON(shutdown_cache(s)); +} + +static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref) +{ + struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache, + memcg_params.refcnt); + unsigned long flags; + + spin_lock_irqsave(&memcg_kmem_wq_lock, flags); + if (s->memcg_params.root_cache->memcg_params.dying) + goto unlock; + + WARN_ON(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_shutdown_fn; + INIT_WORK(&s->memcg_params.work, kmemcg_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); + +unlock: + spin_unlock_irqrestore(&memcg_kmem_wq_lock, flags); +} + +static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ + __kmemcg_cache_deactivate_after_rcu(s); + percpu_ref_kill(&s->memcg_params.refcnt); +} + static void kmemcg_cache_deactivate(struct kmem_cache *s) { - if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.work_fn)) + if (WARN_ON_ONCE(is_root_cache(s))) return; __kmemcg_cache_deactivate(s); @@ -744,10 +783,8 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) if (s->memcg_params.root_cache->memcg_params.dying) goto unlock; - /* pin memcg so that @s doesn't get destroyed in the middle */ - css_get(&s->memcg_params.memcg->css); - - s->memcg_params.work_fn = __kmemcg_cache_deactivate_after_rcu; + WARN_ON_ONCE(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); unlock: spin_unlock_irq(&memcg_kmem_wq_lock); @@ -781,28 +818,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) put_online_cpus(); } -void memcg_destroy_kmem_caches(struct mem_cgroup *memcg) -{ - struct kmem_cache *s, *s2; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry_safe(s, s2, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - /* - * The cgroup is about to be freed and therefore has no charges - * left. Hence, all its caches must be empty by now. - */ - BUG_ON(shutdown_cache(s)); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { struct memcg_cache_array *arr; From patchwork Tue Jun 11 23:18:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988353 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E90D814E5 for ; Tue, 11 Jun 2019 23:18:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D75F22881A for ; Tue, 11 Jun 2019 23:18:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CB7312890A; Tue, 11 Jun 2019 23:18:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 149852881A for ; Tue, 11 Jun 2019 23:18:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B390C6B0273; Tue, 11 Jun 2019 19:18:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AED3D6B0274; Tue, 11 Jun 2019 19:18:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 966E46B0275; Tue, 11 Jun 2019 19:18:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 580AD6B0273 for ; Tue, 11 Jun 2019 19:18:36 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id i3so8617647plb.8 for ; Tue, 11 Jun 2019 16:18:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=Tffvx74eRN7sOmy9DcF9GcEFiD3Crn+QmEOXHigsbRc=; b=Oia89jcl6XIvV8Wkf/ZTlD8twROOuRAhxvIkeabGOtosF9FKMZNNr9fmWJjn4RHrxk Bvax7IxVjMzAEMUvcuOw9e4oI7AXj3w/tqXb1yg8K3ej9yesV2dprMB17PGXLMCphuG+ zIrKiWe2UTzCnpjU/mQEva3nnGlklxkfAjTems8OVQs7Lo83D9PtRIlHiS443/rhqOWI KMHhbixDd+7QaQvQYI0XMIsS18qdJfNZ4ydASp8t807zCWZ76zBsqHf+RltcyXP1xRIZ Kcz38PeZ1wU9k6Fov+qMUeTE3cCkY10ZdwqPEOT+e3A0mG0IfPijtnU2/+CBHLOeOD2X wxiQ== X-Gm-Message-State: APjAAAW8qv127rJX1wFPLQXCnHi0HyPvPlTmB/c6pwRdZI3PdLRf8+1D JFIqBfKh75WW9HxvXFT7NF1GcAUuoTFOrJ+TJa6klaqlAC66c6rDn4OHlhqy12bLLXrbAB1mkqH GnRePr7eVUjsnOZdyqaq2CfOGyk0hrr5JIIxH+1AxddTqBFPepY9NYR+Or4r0u1Uzug== X-Received: by 2002:a62:4d03:: with SMTP id a3mr5193386pfb.2.1560295115993; Tue, 11 Jun 2019 16:18:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqzs1LTHV5VEXJS4fSVulNoHezwjKHL2SOIXGq8yrnTwo2pyB1ouIVjbGvAPO4xGCOTsNT4M X-Received: by 2002:a62:4d03:: with SMTP id a3mr5193312pfb.2.1560295115130; Tue, 11 Jun 2019 16:18:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295115; cv=none; d=google.com; s=arc-20160816; b=BPivI6xzWVfmy9IyeRzsvBTXWNQ3JpVLtUDt2dB61jKMWiUCtoqsm3Dn0tezaRZFfN OoEcUqZV5Mw3mJbEjOulYnow0EpG/h2ZS8kY9yk1BJO0B048M2h2wMoiP8lbR81flgbn PtTMN0SnPKLEssSIrq2tfxjMjAO/d2KulYoPNeQ/vou8PRuKyq/riOqANzpAD8ZxMPFw kE/VxTo2PL6Ql4pvv5Tdr/CabauWIRTDOb6aAXFTE4t9NLg+rdcpLHhPPApBt19WUkXz HbFsyfUVbNdGlUGqCjJx+K3nqGq7giU8vR4mI9oEKnLHhx7rS1eWhcRLwxQ7nGbCNMsK 11Ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=Tffvx74eRN7sOmy9DcF9GcEFiD3Crn+QmEOXHigsbRc=; b=GgpBFQ9QOXjQOIuDVR7YPVZVvCwwLrpKtyWMzpnfvcAVksZJM8dp+wBmt+fG1iQEfl 5fcifblvcrNEdnP46KQYRbPDBrkxyPHOkcg+vXhpjNRCxryMWUZxDteLXEuVCw3MIo0w 2spI+Kqk9+OMkIefOuzS+pqzFQLeeTjtIxBM2so5dK5mDEulp2tuG0hQvSHNS3qS+/To Sqf7fwcZNyervdbV2wrpfZAJWsgFElNYEjsYep0ejhEroh68Bmmm7dH+YLGoqtGDgV6z tNpkLQjAWXwvDNjuIQEJ4pTZUqowEsj33MBB4qecAy0B91/sfkhrSx0zgG6rIoHl2cRV qyDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=D2jzOp2P; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id k4si13109732pgq.293.2019.06.11.16.18.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:35 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=D2jzOp2P; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BN9aFA031322 for ; Tue, 11 Jun 2019 16:18:34 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=Tffvx74eRN7sOmy9DcF9GcEFiD3Crn+QmEOXHigsbRc=; b=D2jzOp2PflUbmHqEVfc7rVBqWuylvhKlJDwk4z50trsh5OnSL8V0XdAJx156r+1jIQ87 pWGXmoZ5D/ZMEHXRJhwunP7szIiqp221o3oIPGl0Gtd0fVuOUf1r7cC6euRTFPP0d+VD 0tRIEwxDkaym8d3Uox+TFQ1JXaEQOcnVSBE= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t2ha1926c-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 11 Jun 2019 16:18:34 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:22 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 4A8F8130CBF77; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 09/10] mm: stop setting page->mem_cgroup pointer for slab pages Date: Tue, 11 Jun 2019 16:18:12 -0700 Message-ID: <20190611231813.3148843-10-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=655 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Every slab page charged to a non-root memory cgroup has a pointer to the memory cgroup and holds a reference to it, which protects a non-empty memory cgroup from being released. At the same time the page has a pointer to the corresponding kmem_cache, and also hold a reference to the kmem_cache. And kmem_cache by itself holds a reference to the cgroup. So there is clearly some redundancy, which allows to stop setting the page->mem_cgroup pointer and rely on getting memcg pointer indirectly via kmem_cache. Further it will allow to change this pointer easier, without a need to go over all charged pages. So let's stop setting page->mem_cgroup pointer for slab pages, and stop using the css refcounter directly for protecting the memory cgroup from going away. Instead rely on kmem_cache as an intermediate object. Make sure that vmstats and shrinker lists are working as previously, as well as /proc/kpagecgroup interface. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- mm/list_lru.c | 3 +- mm/memcontrol.c | 12 ++++---- mm/slab.h | 74 ++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 70 insertions(+), 19 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 927d85be32f6..0f1f6b06b7f3 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -12,6 +12,7 @@ #include #include #include +#include "slab.h" #ifdef CONFIG_MEMCG_KMEM static LIST_HEAD(list_lrus); @@ -63,7 +64,7 @@ static __always_inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr) if (!memcg_kmem_enabled()) return NULL; page = virt_to_head_page(ptr); - return page->mem_cgroup; + return memcg_from_slab_page(page); } static inline struct list_lru_one * diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 43a42bc3ed3f..25e72779fd33 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -486,7 +486,10 @@ ino_t page_cgroup_ino(struct page *page) unsigned long ino = 0; rcu_read_lock(); - memcg = READ_ONCE(page->mem_cgroup); + if (PageHead(page) && PageSlab(page)) + memcg = memcg_from_slab_page(page); + else + memcg = READ_ONCE(page->mem_cgroup); while (memcg && !(memcg->css.flags & CSS_ONLINE)) memcg = parent_mem_cgroup(memcg); if (memcg) @@ -2807,9 +2810,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, cancel_charge(memcg, nr_pages); return -ENOMEM; } - - page->mem_cgroup = memcg; - return 0; } @@ -2832,8 +2832,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg); - if (!ret) + if (!ret) { + page->mem_cgroup = memcg; __SetPageKmemcg(page); + } } css_put(&memcg->css); return ret; diff --git a/mm/slab.h b/mm/slab.h index 5d2b8511e6fb..7ead47cb9338 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -255,30 +255,67 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s->memcg_params.root_cache; } +/* + * Expects a pointer to a slab page. Please note, that PageSlab() check + * isn't sufficient, as it returns true also for tail compound slab pages, + * which do not have slab_cache pointer set. + * So this function assumes that the page can pass PageHead() and PageSlab() + * checks. + */ +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + struct kmem_cache *s; + + s = READ_ONCE(page->slab_cache); + if (s && !is_root_cache(s)) + return s->memcg_params.memcg; + + return NULL; +} + +/* + * Charge the slab page belonging to the non-root kmem_cache. + * Can be called for non-root kmem_caches only. + */ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { + struct mem_cgroup *memcg; + struct lruvec *lruvec; int ret; - if (is_root_cache(s)) - return 0; - - ret = memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + memcg = s->memcg_params.memcg; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) return ret; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); + + /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + css_put_many(&memcg->css, 1 << order); return 0; } +/* + * Uncharge a slab page belonging to a non-root kmem_cache. + * Can be called for non-root kmem_caches only. + */ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { - if (!is_root_cache(s)) - percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); - memcg_kmem_uncharge(page, order); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + + memcg = s->memcg_params.memcg; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); + memcg_kmem_uncharge_memcg(page, order, memcg); + + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } extern void slab_init_memcg_params(struct kmem_cache *); @@ -314,6 +351,11 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) return s; } +static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +{ + return NULL; +} + static inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { @@ -351,18 +393,24 @@ static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - int ret = memcg_charge_slab(page, gfp, order, s); - - if (!ret) - mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + 1 << order); + return 0; + } - return ret; + return memcg_charge_slab(page, gfp, order, s); } static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { - mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + -(1 << order)); + return; + } + memcg_uncharge_slab(page, order, s); } From patchwork Tue Jun 11 23:18:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10988349 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B59014DB for ; Tue, 11 Jun 2019 23:18:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A5172881A for ; Tue, 11 Jun 2019 23:18:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2E85D2890A; Tue, 11 Jun 2019 23:18:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 665732881A for ; Tue, 11 Jun 2019 23:18:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E21296B0276; Tue, 11 Jun 2019 19:18:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D82576B0273; Tue, 11 Jun 2019 19:18:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAF9D6B0275; Tue, 11 Jun 2019 19:18:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 78E296B0273 for ; Tue, 11 Jun 2019 19:18:35 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id l4so10660049pff.5 for ; Tue, 11 Jun 2019 16:18:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=4a5S038n6rmjsorMyNQnK5HbJ3nnI0HE3O8xMf4e+Mg=; b=Rhszao5aL+aHzvKz067UVBS0GjZYnGREXYV8bQ2wfwcE/WIJdVYywfbOwBd4qsJuBe 3CQzfiJOuGe7REOmi17N5Pk5hg/giO8ZzbMq0ED1uwseZTOJxPLlHH0u3ei3J7+Cunpb fPAntSQjAt55fYYQfumM62rFewPlvxNT8Lotij40wNgwikuOwnZZu8CM6hvaC7xaCFz2 bofYfcxfByZt3//NSt9j7/o6XAwPanoR58wP9vKO5KLPZ+hy35VL3HxlfzFfPN59tJDd vP17Dyg5ecPX87+AM3ILx9wBwBMjD3yC1Oqlz2+wlBYXN3P2jPQhE8kwNdleq3PRVzyD 4HWg== X-Gm-Message-State: APjAAAVpdTjknZkb+CB1jP13nqAs8aY6myJWvlPGloS9Cv9m27rA0WYk i6LiKjY45UDbbxOJvJsoXnFc0OkEbP3/AxhlimhcDyQY1d0gkKu0QfdjOz0+GRpkHTDuvBcJBb+ q9plYvudATuqcdC/tAzVAwxGoBhmjQMrlfsrqAHbku5BBhti62YWjqbChEAUTWqfOjg== X-Received: by 2002:a17:902:934b:: with SMTP id g11mr66021972plp.245.1560295115115; Tue, 11 Jun 2019 16:18:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqw91mnuV5frHUcISfPtOOfB6cB1W21YUrTCZlvXo/zc/bXyT6BEjyQgPpZmJsMJFpXFDXBC X-Received: by 2002:a17:902:934b:: with SMTP id g11mr66021919plp.245.1560295114237; Tue, 11 Jun 2019 16:18:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560295114; cv=none; d=google.com; s=arc-20160816; b=CHiXHgqm/238PJUkpfZzCeNbhdyWx5bjw85iwrXG7/aQ9ZN2GLObmTN06REGk41EG8 hCh9dN3rcxPpP3zJV+V3EqAcdnd295oqPmfOFvZS+yL8oyt3GQ4tkROIEBTwaCuOPxOX Eg4p/w1xBRGm2jt/gYMXYDnMRYpJ5Qhm44aVqc2VL+AHq7GkXmBdc1l2Aaw1OOjny2MZ Yn2Plk5/R1KJWRGSjW4JEa1DVPWINO2qaPdfanO2EjwKiOLlFbck0hDNeVNHLC9wI9Lm Ocy/3cRmyT8uffKtOFlh3dVQUUMnl5NwnuFtno5w449NmK5P/uWq2TDfPNBdCEv3R95s ggag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=4a5S038n6rmjsorMyNQnK5HbJ3nnI0HE3O8xMf4e+Mg=; b=pSCkxRGNL9dAjb6Z2fdUSdPdLdtZiT2MDBI0ya4rgcBMItmbmifvRtUaS/VvtOCdhL ARo2S+mAvZuDO3tBUyetO4HS77to1FLn0lO8Bf4/2df2f/wrSnWbo2K0n6M07bOM295r OlcLGGgRQpWvphwAZa4ple4041fCDHLv6atnOO0jXGdOLhWn7Yms5mt4u3vK1J4vY9Jl x6Cbs3pZeKW4SsYfsyXCRmKgS68kgX6GobG9eKmYiE5wXFO4YF/wQg1OaVATTZlAO6QE gJMDfwrxHU/s05viU7hzCeFzxtMEep0soEcQXD8r65inelR/vnVzUUXpKhRVpEQ9dcHL 0XsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="Vy9ofn/a"; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id v20si13859752pgk.58.2019.06.11.16.18.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Jun 2019 16:18:34 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b="Vy9ofn/a"; spf=pass (google.com: domain of prvs=106579ac2e=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=106579ac2e=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5BN9aF8031322 for ; Tue, 11 Jun 2019 16:18:33 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=4a5S038n6rmjsorMyNQnK5HbJ3nnI0HE3O8xMf4e+Mg=; b=Vy9ofn/ahsuhhMv8Ehj9mot6wdzLhc9P9NBL7Qw9JOQoLAMg0ZqAPLjubeQNvHRDEn9s k8w6jVD/XV9NkNTAjp/eZQwWq+uaQGKLF7zdnc3oVoZYQgqlxziwSL1WLK+67XQLdMh0 p5WXCS6kOSg8obpLnSTnX146VAaocR4Pz3Y= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 2t2ha1926c-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 11 Jun 2019 16:18:33 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 11 Jun 2019 16:18:22 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 4EAFF130CBF79; Tue, 11 Jun 2019 16:18:20 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton , Vladimir Davydov CC: , , , Johannes Weiner , Shakeel Butt , Waiman Long , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v7 10/10] mm: reparent memcg kmem_caches on cgroup removal Date: Tue, 11 Jun 2019 16:18:13 -0700 Message-ID: <20190611231813.3148843-11-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611231813.3148843-1-guro@fb.com> References: <20190611231813.3148843-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-11_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906110151 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's reparent non-root kmem_caches on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application). Since the parent cgroup is already charged, everything we need to do is to splice the list of kmem_caches to the parent's kmem_caches list, swap the memcg pointer, drop the css refcounter for each kmem_cache and adjust the parent's css refcounter. Please, note that kmem_cache->memcg_params.memcg isn't a stable pointer anymore. It's safe to read it under rcu_read_lock(), cgroup_mutex held, or any other way that protects the memory cgroup from being released. We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere. Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE) aren't used anywhere except count_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything. Signed-off-by: Roman Gushchin Acked-by: Vladimir Davydov Reviewed-by: Shakeel Butt --- include/linux/slab.h | 2 +- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 26 ++++++++++++++++++++------ mm/slab_common.c | 19 +++++++++++++++++-- 4 files changed, 46 insertions(+), 15 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1b54e5f83342..fecf40b7be69 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); -void memcg_deactivate_kmem_caches(struct mem_cgroup *); +void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 25e72779fd33..db46a9dc37ab 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3289,15 +3289,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) */ memcg->kmem_state = KMEM_ALLOCATED; - memcg_deactivate_kmem_caches(memcg); - - kmemcg_id = memcg->kmemcg_id; - BUG_ON(kmemcg_id < 0); - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; + memcg_deactivate_kmem_caches(memcg, parent); + + kmemcg_id = memcg->kmemcg_id; + BUG_ON(kmemcg_id < 0); + /* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus @@ -3330,7 +3330,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) if (memcg->kmem_state == KMEM_ALLOCATED) { WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); - WARN_ON(page_counter_read(&memcg->kmem)); } } #else @@ -4777,6 +4776,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) /* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM + INIT_LIST_HEAD(&memcg->kmem_caches); +#endif root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/slab.h b/mm/slab.h index 7ead47cb9338..a4c9b9d042de 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -261,6 +261,9 @@ static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) * which do not have slab_cache pointer set. * So this function assumes that the page can pass PageHead() and PageSlab() * checks. + * + * The kmem_cache can be reparented asynchronously. The caller must ensure + * the memcg lifetime, e.g. by taking rcu_read_lock() or cgroup_mutex. */ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) { @@ -268,7 +271,7 @@ static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) s = READ_ONCE(page->slab_cache); if (s && !is_root_cache(s)) - return s->memcg_params.memcg; + return READ_ONCE(s->memcg_params.memcg); return NULL; } @@ -285,10 +288,18 @@ static __always_inline int memcg_charge_slab(struct page *page, struct lruvec *lruvec; int ret; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = READ_ONCE(s->memcg_params.memcg); + while (memcg && !css_tryget_online(&memcg->css)) + memcg = parent_mem_cgroup(memcg); + rcu_read_unlock(); + + if (unlikely(!memcg)) + return true; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) - return ret; + goto out; lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); @@ -296,8 +307,9 @@ static __always_inline int memcg_charge_slab(struct page *page, /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); css_put_many(&memcg->css, 1 << order); - - return 0; +out: + css_put(&memcg->css); + return ret; } /* @@ -310,10 +322,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct mem_cgroup *memcg; struct lruvec *lruvec; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = READ_ONCE(s->memcg_params.memcg); lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); memcg_kmem_uncharge_memcg(page, order, memcg); + rcu_read_unlock(); percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 6b7750f7ea33..91e8c739dc97 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -252,7 +252,8 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); - css_put(&s->memcg_params.memcg->css); + mem_cgroup_put(s->memcg_params.memcg); + WRITE_ONCE(s->memcg_params.memcg, NULL); } } #else @@ -790,11 +791,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) spin_unlock_irq(&memcg_kmem_wq_lock); } -void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) +void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg, + struct mem_cgroup *parent) { int idx; struct memcg_cache_array *arr; struct kmem_cache *s, *c; + unsigned int nr_reparented; idx = memcg_cache_id(memcg); @@ -812,6 +815,18 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } + nr_reparented = 0; + list_for_each_entry(s, &memcg->kmem_caches, + memcg_params.kmem_caches_node) { + WRITE_ONCE(s->memcg_params.memcg, parent); + css_put(&memcg->css); + nr_reparented++; + } + if (nr_reparented) { + list_splice_init(&memcg->kmem_caches, + &parent->kmem_caches); + css_get_many(&parent->css, nr_reparented); + } mutex_unlock(&slab_mutex); put_online_mems();