From patchwork Tue Apr 23 21:31:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10914151 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 926AC1390 for ; Wed, 24 Apr 2019 05:25:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 815F028701 for ; Wed, 24 Apr 2019 05:25:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74C4828867; Wed, 24 Apr 2019 05:25:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E1E4828701 for ; Wed, 24 Apr 2019 05:25:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A978D6B0005; Wed, 24 Apr 2019 01:25:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A1D9D6B0006; Wed, 24 Apr 2019 01:25:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BD5C6B0007; Wed, 24 Apr 2019 01:25:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 3B3E26B0005 for ; Wed, 24 Apr 2019 01:25:03 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id n25so9234562edd.5 for ; Tue, 23 Apr 2019 22:25:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=76gfXMG+0uRElSBO/APtur6FtcIZXTee5d1iEwxVGMw=; b=fnVxplyPRbhFkVVhTMWMs+cStcG7Khuf/EwJW3FQCfTJnN4HFBNpqbo6OiYNUWWlpD PtfRyqm+mRlqcYeKLO2yPjqhfrS9H5Iute+OTgAzTpNyIQF7vNx1GsxibIrUwnP0RD0k CqJ4WsjvJWM8T1l3TtZ0BTeI3Jw35ayvDz83I90h5nkRp+h8fPZvZx9PlZ4iyx29A7Bz tCQvzSgyWaKVU2C6yjmB7DXqd9SjbxWnA5jo8/U2PL1SEHFyAyjmJ9VS/mZgsGKY5MvF 9yKAykyUsrS5xk8yiykjYcd2+SKKmR3WRq5E0AU7R8k3wlMkEGkqGKrU8zYg4sFk+5kj FJqg== X-Gm-Message-State: APjAAAXCqKeCCO/HttL3AUu3OlpCotUfSuGBxS+br4EjX7T63OC/4Ni+ 6LrPJKEahwDqnLSLSvLkmwaz9vwa9lL69+XSrmLuLsimmPjJ/vhWLBYxo2pGL2puJQIQd2Xs8ur phgwhdheD/DYROk+H/uwSGQclvnUcOsbXVP6ga3EHNWUACedbvxkXg0guBLSBPDGU2A== X-Received: by 2002:a50:ed0a:: with SMTP id j10mr17970631eds.188.1556083502687; Tue, 23 Apr 2019 22:25:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqwd4xJeciem+O3fFWexFUaBIY8kFLIPZEcFobtdMCz7IHmS5eZS46JXwoW6gVAk0hwC6ALe X-Received: by 2002:a50:ed0a:: with SMTP id j10mr17970599eds.188.1556083501835; Tue, 23 Apr 2019 22:25:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556083501; cv=none; d=google.com; s=arc-20160816; b=L9x1moumQFJOzG7RznqPWUQDQoPLtFehR+TgXGN84T3RuSoLYe79Nob0wth4PuMzXs Kt1vIqedaYNcL9LlrjRpVjZPBrZjhqRdjkKPtzhqmVmlG/s8m8jzzxxiAuWDIZqF15U7 TU8cDuoLMDCejV4dvISODqPE0DtbHVusH494XF2LCv6Rx3HJ4/ovsFZYH1gNfGOqDPMy VSybqPG6vzXjogQIsncyTYjglKe4zi9joPKZMCsKKT6cJC6CrbUdETJPikYQUHn9gHYh I+jI/lfB48wuojtN8nGSplCCHdH8z69k+e3vXm7ArUM6K1JGjhXK7MM7y2q4ODgF6s6q VfSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=76gfXMG+0uRElSBO/APtur6FtcIZXTee5d1iEwxVGMw=; b=dzp9NgEKXyVQ27Q7qg5vZ/wbZR3yZEgiS+pqbGu7IisD/nCuAlUxtmq+WOIvnCa/1R 3vZfXIr7Y+yKqo6UvokZ7GqfRbvb+lO6bdXLTMiewNzgz+xW/fsRlKGDtONv/E1sbc2J 7YxlJlbO/o4w6mqe/x8CQZ9UbmIR/Gd8JjwAxseVKp1wnDtaw3CPehFAASkLrzJIyPf9 aAC7PqJvIQ0A9GlDwJjurIgEqUIc1CVqg92rHj76HBbsBg/OlPDoGlT5axh1FpxA1QXc wftTqqbeDnF9x8Q1E7B+7EBwCRbZjXTia+c28DXcPeMIbvOQl8xt8L1PPFuz0m+w0s7d upGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fbPk8N3D; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id l18si1306397ejp.166.2019.04.23.22.25.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Apr 2019 22:25:01 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=fbPk8N3D; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3O5NIM3024315 for ; Tue, 23 Apr 2019 22:25:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=76gfXMG+0uRElSBO/APtur6FtcIZXTee5d1iEwxVGMw=; b=fbPk8N3Dow+holCAQ3pCf4QAbbqddsS6NPrkcJh5/QiNWaTDi9YD90sMkGICKxU3HUQ7 Rvyu2D2pzHyc6dLas8EtypUUHuMAoloq4yfc5bcS8GewMfvZxKcwHG1ruQn+fVFu5/Ro cuCf3AlXQXLrs9x25vA93N1ythjfbpb4Sxc= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2s2695bb1e-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 23 Apr 2019 22:25:00 -0700 Received: from mx-out.facebook.com (2620:10d:c0a1:3::13) by mail.thefacebook.com (2620:10d:c021:18::171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 23 Apr 2019 22:24:52 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id BFBD71142D2E0; Tue, 23 Apr 2019 14:31:36 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v2 1/6] mm: postpone kmem_cache memcg pointer initialization to memcg_link_cache() Date: Tue, 23 Apr 2019 14:31:28 -0700 Message-ID: <20190423213133.3551969-2-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190423213133.3551969-1-guro@fb.com> References: <20190423213133.3551969-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-24_03:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Initialize kmem_cache->memcg_params.memcg pointer in memcg_link_cache() rather than in init_memcg_params(). Once kmem_cache will hold a reference to the memory cgroup, it will simplify the refcounting. For non-root kmem_caches memcg_link_cache() is always called before the kmem_cache becomes visible to a user, so it's safe. Signed-off-by: Roman Gushchin --- mm/slab.c | 2 +- mm/slab.h | 5 +++-- mm/slab_common.c | 14 +++++++------- mm/slub.c | 2 +- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index b1eefe751d2a..57a332f524cf 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1268,7 +1268,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); + memcg_link_cache(kmem_cache, NULL); slab_state = PARTIAL; /* diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..6a562ca72bca 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -289,7 +289,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); +extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, void (*deact_fn)(struct kmem_cache *)); @@ -344,7 +344,8 @@ static inline void slab_init_memcg_params(struct kmem_cache *s) { } -static inline void memcg_link_cache(struct kmem_cache *s) +static inline void memcg_link_cache(struct kmem_cache *s, + struct mem_cgroup *memcg) { } diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..6e00bdf8618d 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -140,13 +140,12 @@ void slab_init_memcg_params(struct kmem_cache *s) } static int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct memcg_cache_array *arr; if (root_cache) { s->memcg_params.root_cache = root_cache; - s->memcg_params.memcg = memcg; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); return 0; @@ -221,11 +220,12 @@ int memcg_update_all_caches(int num_memcgs) return ret; } -void memcg_link_cache(struct kmem_cache *s) +void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) { if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -244,7 +244,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } #else static inline int init_memcg_params(struct kmem_cache *s, - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { return 0; } @@ -384,7 +384,7 @@ static struct kmem_cache *create_cache(const char *name, s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, memcg, root_cache); + err = init_memcg_params(s, root_cache); if (err) goto out_free_cache; @@ -394,7 +394,7 @@ static struct kmem_cache *create_cache(const char *name, s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, memcg); out: if (err) return ERR_PTR(err); @@ -997,7 +997,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); s->refcount = 1; return s; } diff --git a/mm/slub.c b/mm/slub.c index a34fbe1f6ede..2b9244529d76 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4224,7 +4224,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); + memcg_link_cache(s, NULL); return s; } From patchwork Tue Apr 23 21:31:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10913779 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A375414DB for ; Tue, 23 Apr 2019 22:16:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87F8C2886B for ; Tue, 23 Apr 2019 22:16:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74FC32887B; Tue, 23 Apr 2019 22:16:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACA0D2886B for ; Tue, 23 Apr 2019 22:16:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 741E06B0005; Tue, 23 Apr 2019 18:16:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6CAD36B0008; Tue, 23 Apr 2019 18:16:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 544046B000A; Tue, 23 Apr 2019 18:16:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id F2F676B0005 for ; Tue, 23 Apr 2019 18:16:53 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id j9so4392688eds.17 for ; Tue, 23 Apr 2019 15:16:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=MtkthRXb9xr3Z6+sHE+ddzmShZROkNDy015XAdhrxgc=; b=DVuNuMrORuki7pFKtsaq5pq7aYGeNw+vfH8vI+z4ug3Z4T16RszmuUXEDX26H7gs7+ YsFxNIH+nv2retqyQDENrjxaBABMAmTnOmVXPgEMwTHXfvQhe3wFfz7+0aISLAKrMXXt m8CMi1aE0GvO6w4MfEm3G3dvQyoHTBOlDgQRVv9LTzzhQd+uG+4HQXPR93Va0+HSHanE Hduf7kogD8+PkPX/i2kg3WEqm7rtBCZ0lkuNybW5dOk+YSsTa9g6H1cw+lcxXGi2ytl5 3AM33LiCGIqlq7YELIcH6Z4VvSAMW4aySdm5iAB/wmuHfB9x8v8ksHf+TJ3wNNdxIgWs SuFA== X-Gm-Message-State: APjAAAWMfGMswfH3qSwC66GMv0JVMFuZ/D3EAkaO7uF9Ej8RfGJtUYQa rKP6tbCO4Akg7VCM6pD3BRsOL3yzaptXS0+dXLgWslMonZrPLshrxK7KIYzLKKhPG0qrvGOtdbE l/QvWL2csEoNSkS2LXvU7DzyNJWffrWmn41Tsnf2HwJZq8x1gOmHev1xWqRb+A5WGhw== X-Received: by 2002:a17:906:a3c7:: with SMTP id ca7mr440923ejb.20.1556057813411; Tue, 23 Apr 2019 15:16:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqzDtOoB9fclIDkt0eUmIdfIGP6wm/yy6ZQE3bqZTDkZNm2bATCbD7IH+SobE2921mtArDH+ X-Received: by 2002:a17:906:a3c7:: with SMTP id ca7mr440873ejb.20.1556057812252; Tue, 23 Apr 2019 15:16:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556057812; cv=none; d=google.com; s=arc-20160816; b=YlJOeOpK0YPrNM+setgFYgwSBNHVSOm9GnGS4yJLkUhYF97cu87UHpTYF54gNOykNR EdinB81j1INOr3E7kFnZIEmoGWPwwEffrP+NI8SVAhN4RLDWGFDXDeM/JXoG3npO7YLp py3A0zEtQaqQuy1CrVvUVeoZdj2sGjH+lrfeKcsz8rnfOM5kaC8DqQxXU/MLSqdWOsG6 7A2ablFtwX6SWjBTSlOpBFuA42gq5+6O0poLjH1uX3zUgIaHKSX857Ct6MEjudEe6EyK U2v1S1Xx1LSMMEHCmMdF2Zu2ieGZobJ+AxOi+ZY4n3lvayCMFTJ7G+TzQFDhMkGpuCfM Lfqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=MtkthRXb9xr3Z6+sHE+ddzmShZROkNDy015XAdhrxgc=; b=UbrZQ6oOTbh+p4ns9+20nH827/QM2cv8Im3hNhywsNRERUWDUCOyK1qpRL21gtF6ni 5XQjqA6dzTJfb76thc+V8dlOYaIECbTG2MMnowNBioxk2xK3tnmUqEOOrVHnl1wrub3U 3Vfq4frsz6KUs3W8/NjZBsShcUMGBmHSj2oRESSPc52VK53nk04plFkq8lc9pDfIBfeO ClieW4RcORQjAnP0C8j0w7P9WXI+JjDxs1ETQAXThnAehyT5vfKFPFo7vRkP4Qk+x/lX VSxNm27lvmz+2YOmAh2+SEh+9+NH/gx5EHSIWkqU4PdpNiir6idjYlwaSrT1cq1nY/30 bPkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=lmYZkXiA; spf=pass (google.com: domain of prvs=901699794a=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=901699794a=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id t4si1940479ejt.96.2019.04.23.15.16.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Apr 2019 15:16:52 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=901699794a=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=lmYZkXiA; spf=pass (google.com: domain of prvs=901699794a=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=901699794a=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3NM9gFt002693 for ; Tue, 23 Apr 2019 15:16:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=MtkthRXb9xr3Z6+sHE+ddzmShZROkNDy015XAdhrxgc=; b=lmYZkXiARc+GMISjiGAr6LDhoLFvNr4MU96SCAqjRj3y2G9lSSUe2X+6OHuS6ro69Jlg YiIWjRbLD+cUv0QKwgO1yC0ZvNeMUi23hEWlCv4h5B9/aRkn8ULTBmk8FSTVZwUmHmHG dlMRoglP+3++t1x0A/Eoo3ereq0i5UHobIM= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2s25j31y0r-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 23 Apr 2019 15:16:50 -0700 Received: from mx-out.facebook.com (2620:10d:c0a1:3::13) by mail.thefacebook.com (2620:10d:c021:18::175) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 23 Apr 2019 15:16:48 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id C2E9C1142D2E2; Tue, 23 Apr 2019 14:31:36 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v2 2/6] mm: generalize postponed non-root kmem_cache deactivation Date: Tue, 23 Apr 2019 14:31:29 -0700 Message-ID: <20190423213133.3551969-3-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190423213133.3551969-1-guro@fb.com> References: <20190423213133.3551969-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-23_08:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently SLUB uses a work scheduled after an RCU grace period to deactivate a non-root kmem_cache. This mechanism can be reused for kmem_caches reparenting, but requires some generalization. Let's decouple all infrastructure (rcu callback, work callback) from the SLUB-specific code, so it can be used with SLAB as well. Also, let's rename some functions to make the code look simpler. All SLAB/SLUB-specific functions start with "__". Remove "deact_" prefix from the corresponding struct fields. Here is the graph of a new calling scheme: kmemcg_cache_deactivate() __kmemcg_cache_deactivate() SLAB/SLUB-specific kmemcg_schedule_work_after_rcu() rcu kmemcg_after_rcu_workfn() work kmemcg_cache_deactivate_after_rcu() __kmemcg_cache_deactivate_after_rcu() SLAB/SLUB-specific instead of: __kmemcg_cache_deactivate() SLAB/SLUB-specific slab_deactivate_memcg_cache_rcu_sched() SLUB-only kmemcg_deactivate_rcufn SLUB-only, rcu kmemcg_deactivate_workfn SLUB-only, work kmemcg_cache_deact_after_rcu() SLUB-only Signed-off-by: Roman Gushchin --- include/linux/slab.h | 6 ++--- mm/slab.c | 4 +++ mm/slab.h | 3 ++- mm/slab_common.c | 62 ++++++++++++++++++++------------------------ mm/slub.c | 8 +----- 5 files changed, 38 insertions(+), 45 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..47923c173f30 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -642,10 +642,10 @@ struct memcg_cache_params { struct list_head children_node; struct list_head kmem_caches_node; - void (*deact_fn)(struct kmem_cache *); + void (*work_fn)(struct kmem_cache *); union { - struct rcu_head deact_rcu_head; - struct work_struct deact_work; + struct rcu_head rcu_head; + struct work_struct work; }; }; }; diff --git a/mm/slab.c b/mm/slab.c index 57a332f524cf..14466a73d057 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -2317,6 +2317,10 @@ void __kmemcg_cache_deactivate(struct kmem_cache *cachep) { __kmem_cache_shrink(cachep); } + +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) +{ +} #endif int __kmem_cache_shutdown(struct kmem_cache *cachep) diff --git a/mm/slab.h b/mm/slab.h index 6a562ca72bca..4a261c97c138 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -172,6 +172,7 @@ int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -291,7 +292,7 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); extern void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)); + void (*work_fn)(struct kmem_cache *)); #else /* CONFIG_MEMCG_KMEM */ diff --git a/mm/slab_common.c b/mm/slab_common.c index 6e00bdf8618d..4e5b4292a763 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -691,17 +691,18 @@ void memcg_create_kmem_cache(struct mem_cgroup *memcg, put_online_cpus(); } -static void kmemcg_deactivate_workfn(struct work_struct *work) +static void kmemcg_after_rcu_workfn(struct work_struct *work) { struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.deact_work); + memcg_params.work); get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - s->memcg_params.deact_fn(s); + s->memcg_params.work_fn(s); + s->memcg_params.work_fn = NULL; mutex_unlock(&slab_mutex); @@ -712,37 +713,28 @@ static void kmemcg_deactivate_workfn(struct work_struct *work) css_put(&s->memcg_params.memcg->css); } -static void kmemcg_deactivate_rcufn(struct rcu_head *head) +/* + * We need to grab blocking locks. Bounce to ->work. The + * work item shares the space with the RCU head and can't be + * initialized eariler. +*/ +static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.deact_rcu_head); + memcg_params.rcu_head); - /* - * We need to grab blocking locks. Bounce to ->deact_work. The - * work item shares the space with the RCU head and can't be - * initialized eariler. - */ - INIT_WORK(&s->memcg_params.deact_work, kmemcg_deactivate_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.deact_work); + INIT_WORK(&s->memcg_params.work, kmemcg_after_rcu_workfn); + queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } -/** - * slab_deactivate_memcg_cache_rcu_sched - schedule deactivation after a - * sched RCU grace period - * @s: target kmem_cache - * @deact_fn: deactivation function to call - * - * Schedule @deact_fn to be invoked with online cpus, mems and slab_mutex - * held after a sched RCU grace period. The slab is guaranteed to stay - * alive until @deact_fn is finished. This is to be used from - * __kmemcg_cache_deactivate(). - */ -void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, - void (*deact_fn)(struct kmem_cache *)) +static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { - if (WARN_ON_ONCE(is_root_cache(s)) || - WARN_ON_ONCE(s->memcg_params.deact_fn)) - return; + __kmemcg_cache_deactivate_after_rcu(s); +} + +static void kmemcg_cache_deactivate(struct kmem_cache *s) +{ + __kmemcg_cache_deactivate(s); if (s->memcg_params.root_cache->memcg_params.dying) return; @@ -750,8 +742,9 @@ void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s, /* pin memcg so that @s doesn't get destroyed in the middle */ css_get(&s->memcg_params.memcg->css); - s->memcg_params.deact_fn = deact_fn; - call_rcu(&s->memcg_params.deact_rcu_head, kmemcg_deactivate_rcufn); + WARN_ON_ONCE(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) @@ -773,7 +766,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) if (!c) continue; - __kmemcg_cache_deactivate(c); + kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } mutex_unlock(&slab_mutex); @@ -866,11 +859,12 @@ static void flush_memcg_workqueue(struct kmem_cache *s) mutex_unlock(&slab_mutex); /* - * SLUB deactivates the kmem_caches through call_rcu. Make + * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make * sure all registered rcu callbacks have been invoked. */ - if (IS_ENABLED(CONFIG_SLUB)) - rcu_barrier(); +#ifndef CONFIG_SLOB + rcu_barrier(); +#endif /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB diff --git a/mm/slub.c b/mm/slub.c index 2b9244529d76..195f61785c7d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4033,7 +4033,7 @@ int __kmem_cache_shrink(struct kmem_cache *s) } #ifdef CONFIG_MEMCG -static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) +void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. @@ -4059,12 +4059,6 @@ void __kmemcg_cache_deactivate(struct kmem_cache *s) */ slub_set_cpu_partial(s, 0); s->min_partial = 0; - - /* - * s->cpu_partial is checked locklessly (see put_cpu_partial), so - * we have to make sure the change is visible before shrinking. - */ - slab_deactivate_memcg_cache_rcu_sched(s, kmemcg_cache_deact_after_rcu); } #endif /* CONFIG_MEMCG */ From patchwork Tue Apr 23 21:31:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10914077 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C1791515 for ; Wed, 24 Apr 2019 05:11:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B96128A48 for ; Wed, 24 Apr 2019 05:11:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F5B128A4A; Wed, 24 Apr 2019 05:11:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E917A28A48 for ; Wed, 24 Apr 2019 05:11:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03EF06B0005; Wed, 24 Apr 2019 01:11:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F2FEA6B0006; Wed, 24 Apr 2019 01:11:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF67F6B000A; Wed, 24 Apr 2019 01:11:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 8FA706B0005 for ; Wed, 24 Apr 2019 01:11:21 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id j9so4828772eds.17 for ; Tue, 23 Apr 2019 22:11:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=cjltM3PwRHKlunBeTZVTehXyFJS0UTo3dIGGM8A+7Ff2ACOjgTGZXzcKRxYzNwgtkF oObzCEHRggbD8XyI2x46eZrJyz3F8wzSh6EJjQo9xyz2GGFqG6cj7fuBrdnxLu+J0bAa ULQFuL3FSgp7RCxRYdHUAOR7aqafm92LiZqxHRg2n/D1ddsZecWS2bJAoD790BtiMo53 7acJX6FGtlB55YUEGRbBsCj3FAVDWAqqYCYVLvIlM2EHVzr5ma0MAW8+WiHbhAeIDvGA VaIVkIIUbp6nhd4RyHCn92LkBLs6bYIhJ/it+AZhjM3GS62EQEpxzjTeDqS+fd9J2GLv IIpg== X-Gm-Message-State: APjAAAUooD+Y+/+45bcazBtoOvIr9atqiJnhcjXAaEF5oCnW4hASi7WS jDXxshl/qI7ZqZiymVRs7tEcLmtfef3PZYA9SVuNJR4iECj25oumo8sv56NGqgE4pdBhaoA8pFG w/plliIxEgTBpGA/V+PFCN0MboxK+tm/ZEBzhjQryk/IDDxJC+3wWU0Bh674urCd3Ng== X-Received: by 2002:a05:6402:1711:: with SMTP id y17mr18465496edu.275.1556082680946; Tue, 23 Apr 2019 22:11:20 -0700 (PDT) X-Google-Smtp-Source: APXvYqxsGOD0a+64wsJf500PZNfUlJhRnWqICumtmO2OPEkD1Pog/vIntga4FgC8+Uk1rHWz/0I0 X-Received: by 2002:a05:6402:1711:: with SMTP id y17mr18465448edu.275.1556082679982; Tue, 23 Apr 2019 22:11:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556082679; cv=none; d=google.com; s=arc-20160816; b=zgEYEkwk65LAfwD4bf6QPNedUhy/jiVMue16ji/kDCPa8Uic6qH0x2JPV373nTt4oc CgjcT5foFSg7D97hkmspui5fteXfQYXxCSnzyEIcMsMqqq5cYH/hwGxeG9tK/rqC+cGR FrOr73vhIhr6tzK2vDIU3XwEQ5Kg+LCf5PhlOSVrL2Wf25SebgDCQ6rk2l95W98Db34K Zg1YpN/6bvflHjPXnW3K98I+Ev74Hi5Hi79WLRD2Bm2h7dOv/AlrPhYKXVQ71VUHkurp zX4TD2/JSF7LHs2Q7oOAxHAmdX+9bKeuoXkfuXkJ0mtkItDancm7oriwNMHIgwGkLNQ3 8mIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=TpYYUlTouhrhmukNKO8QEiiPzQeBbWpWsCNk16vqs7quyU173qqryJQqMHM5igOI3J fE0D0eFKEV9SnNNmzvw0vorlgiBkP/XsRYSS/3yQJwSrdqBarambIlfoV1dgnJ8QL8/c 2tmtnKdVrpgWaMU90FLZl9OSYdPYk4m5a9JuXXcmpH/owPVbrJPP4KehHKpJYXSYB8/U jH2+AWFjdlHDYxjo/zjUoTgE6TRBnjh6fvyLnAMhRs/zHfPmdCH0ln1BFUR3/oKrseCl 47d9IRgm3Opc2XB2iM7vh3b0dcMYVl//Y6HHCgPaMdLI07PIiMSif8Vsle1bGh8CfryV N+FA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=UmOzXPgo; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id y12si2709772ejq.177.2019.04.23.22.11.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Apr 2019 22:11:19 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=UmOzXPgo; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3O5AZ7i031564 for ; Tue, 23 Apr 2019 22:11:18 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=cRmr36eXTaKgScK2fa3yPK15EeGvQLf0Lc+faeDoG3Q=; b=UmOzXPgoQyMb6775Oz2yU4LqMZg7j6hKF1EyxksST2XqEVT04qJqd9NI/yWg3n/yIqH/ a5TMQXYG/BcjWC4uW47FfZCqbTOyhI3fFCgDBKNwLiJZUmz//uzvCGM79Jb8FC1VCzJi 0YoFkJ587li2/ZqhKmpKSmSY/V+8v9wZ7Fw= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2s28tp2e8v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 23 Apr 2019 22:11:18 -0700 Received: from mx-out.facebook.com (2620:10d:c0a1:3::13) by mail.thefacebook.com (2620:10d:c021:18::176) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 23 Apr 2019 22:11:16 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id C6D241142D2E4; Tue, 23 Apr 2019 14:31:36 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v2 3/6] mm: introduce __memcg_kmem_uncharge_memcg() Date: Tue, 23 Apr 2019 14:31:30 -0700 Message-ID: <20190423213133.3551969-4-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190423213133.3551969-1-guro@fb.com> References: <20190423213133.3551969-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-24_03:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's separate the page counter modification code out of __memcg_kmem_uncharge() in a way similar to what __memcg_kmem_charge() and __memcg_kmem_charge_memcg() work. This will allow to reuse this code later using a new memcg_kmem_uncharge_memcg() wrapper, which calls __memcg_kmem_unchare_memcg() if memcg_kmem_enabled() check is passed. Signed-off-by: Roman Gushchin --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 25 +++++++++++++++++-------- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 36bdfe8e5965..deb209510902 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1298,6 +1298,8 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge(struct page *page, int order); int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, struct mem_cgroup *memcg); +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; @@ -1339,6 +1341,14 @@ static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, return __memcg_kmem_charge_memcg(page, gfp, order, memcg); return 0; } + +static inline void memcg_kmem_uncharge_memcg(struct page *page, int order, + struct mem_cgroup *memcg) +{ + if (memcg_kmem_enabled()) + __memcg_kmem_uncharge_memcg(memcg, 1 << order); +} + /* * helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 48a8f1c35176..b2c39f187cbb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2750,6 +2750,22 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) css_put(&memcg->css); return ret; } + +/** + * __memcg_kmem_uncharge_memcg: uncharge a kmem page + * @memcg: memcg to uncharge + * @nr_pages: number of pages to uncharge + */ +void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + page_counter_uncharge(&memcg->kmem, nr_pages); + + page_counter_uncharge(&memcg->memory, nr_pages); + if (do_memsw_account()) + page_counter_uncharge(&memcg->memsw, nr_pages); +} /** * __memcg_kmem_uncharge: uncharge a kmem page * @page: page to uncharge @@ -2764,14 +2780,7 @@ void __memcg_kmem_uncharge(struct page *page, int order) return; VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - page_counter_uncharge(&memcg->kmem, nr_pages); - - page_counter_uncharge(&memcg->memory, nr_pages); - if (do_memsw_account()) - page_counter_uncharge(&memcg->memsw, nr_pages); - + __memcg_kmem_uncharge_memcg(memcg, nr_pages); page->mem_cgroup = NULL; /* slab pages do not have PageKmemcg flag set */ From patchwork Tue Apr 23 21:31:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10914065 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12798922 for ; Wed, 24 Apr 2019 04:44:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E92782850F for ; Wed, 24 Apr 2019 04:44:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DC09F28A0B; Wed, 24 Apr 2019 04:44:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 377422850F for ; Wed, 24 Apr 2019 04:44:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B76236B0005; Wed, 24 Apr 2019 00:44:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B25556B0006; Wed, 24 Apr 2019 00:44:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9ECEF6B0007; Wed, 24 Apr 2019 00:44:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id 786546B0005 for ; Wed, 24 Apr 2019 00:44:24 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id g186so13958659ybg.16 for ; Tue, 23 Apr 2019 21:44:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=DYe1Kczls7Vz3TOywv3SCxyloB0ylRoW+6/MG/k28Aw=; b=DVvyT5foRXkai1jZUNTHYzJfnglGC6zr7bhUiv+jE/Yu3PO5XKd2sy0a+HgA0dHBrB 1L2nFezAbi3lJkRyDxVA+oqOQL8s8uxp/gkrTybkW/+cfzpmzlffJYOnRVpg1mv6YIiQ QpSIQ8FLQfpHNGWRWYbCmA4DWv2t1Srr4fl+yT8xokNUsvuETjJbcBG2lfvO+Htmcop2 gJfWgqOUUZ1zDPBDx/nSIt7Y9HRaLaW4y8hMsQVFo6ZEb1mutkCBUnoATl4Utl8hiNSC 0xD5xquinggebXhU9ve/obWup/cdD8KlW17ksS4JH3flsrAH9o4Km63HDnaI+uHChykW 2/XA== X-Gm-Message-State: APjAAAW5/GtFR8LXwXax5ctmZA0jeeSS4S+M7+ElpIe1rheRx96CnHza rJVJIXnjbazx13yI70tteGdhg4gIEkVVbqHsiNa2X9tnkYrRtcI7bSux9YLcppq84uL0OLMSQgc ugSa0Y/dRNY6+c2/fji52E4P0BAuS0HjgcySemTO4Y9TpbKf0niuBGbdW0wbPmEAVdg== X-Received: by 2002:a81:7c56:: with SMTP id x83mr1592385ywc.303.1556081064201; Tue, 23 Apr 2019 21:44:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqwQXH2RDkpXNDVEtztn4YY1peBN+RZLoAKasrUhPt+QZ3z9ABPNs88J9WPjX+krAZxAffZn X-Received: by 2002:a81:7c56:: with SMTP id x83mr1592343ywc.303.1556081063192; Tue, 23 Apr 2019 21:44:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556081063; cv=none; d=google.com; s=arc-20160816; b=lccI+HC4KvR1L0kFI5khL7R1hnbWR9Fwb+0h7rfo/qFsVZEAGgDJ42GrvBbIqKUkQ9 f0SAwUIDB91ILTL3UwoiBljqsNNhsEVllOJG7ueC3g0ENd6xmMu8j/X31+/hC//AHNcd /qJ1tirtIYXOlZviMRXGM4+OCHM6tMWMnfcjqoayp5q4acGitsiamvIRffLxVNaDcnmp O2DQ+AopgzTYXxCpEdt4WNpqg36nDHew761ecc2ow6bZg+NZohAAEZY/5dySsrtDKZii NqX436MwqrPBOOeaTeK1MP4SbBxvURjo7TFVKGhEol/5axBHTtmwRWphTWN0/aRriTKp K7qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=DYe1Kczls7Vz3TOywv3SCxyloB0ylRoW+6/MG/k28Aw=; b=RG9R8xaRm2xIibxg2v2oP/RWpInJm99QG+f0/ZrVHwoJVEy4Re5a65MW1U9L2UNb2J DJJOi0IBhYDKFff1kdIdVpddflhmYOjLVvCYnWGTNkGsrqLaXyOcPUZiLxDR2x5mNiER Pw50GEdUPEuyfT7G5Sf+INyV7FoRmmAfM/xG7tUqitBtZ655fBiS3efygU3cGw8+vfVK d8AP94lo4yq7f5lB7jDZ3XxHtQGQo0RYNsbH/Cda/DhR8sbwdxNtJuTdpTPQxsLp261T elyKLCiNqYzXrt2+mmp+KdJbX7i4/93gjwzM+UkBdEcNP9TDwU82IQofHEZNvmhpQxBt FIiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=bHFYCX9d; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id j74si12375977ybj.472.2019.04.23.21.44.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Apr 2019 21:44:23 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=bHFYCX9d; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.16.0.27/8.16.0.27) with SMTP id x3O4ftRX022118 for ; Tue, 23 Apr 2019 21:44:22 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=DYe1Kczls7Vz3TOywv3SCxyloB0ylRoW+6/MG/k28Aw=; b=bHFYCX9dNv+cHnVzRdmj18VbL4bDaxiJJyLgc94iTPCfRd9wNi3dL/rByW9imQZa6iGr kACIO/RjRo+g9bmaNvlqG0vBB/KS4n/qab2mlGo2lBZ6N+pY8AiRkfvXFNklVhQHI0sR qBwXvEKBnK4lJsv6nu2Q33NJFu6LmyVS0aY= Received: from maileast.thefacebook.com ([199.201.65.23]) by m0001303.ppops.net with ESMTP id 2s28ke1upn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 23 Apr 2019 21:44:22 -0700 Received: from mx-out.facebook.com (2620:10d:c0a1:3::13) by mail.thefacebook.com (2620:10d:c021:18::174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 23 Apr 2019 21:44:21 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id CBFA01142D2E6; Tue, 23 Apr 2019 14:31:36 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v2 4/6] mm: unify SLAB and SLUB page accounting Date: Tue, 23 Apr 2019 14:31:31 -0700 Message-ID: <20190423213133.3551969-5-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190423213133.3551969-1-guro@fb.com> References: <20190423213133.3551969-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-24_02:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently the page accounting code is duplicated in SLAB and SLUB internals. Let's move it into new (un)charge_slab_page helpers in the slab_common.c file. These helpers will be responsible for statistics (global and memcg-aware) and memcg charging. So they are replacing direct memcg_(un)charge_slab() calls. Signed-off-by: Roman Gushchin --- mm/slab.c | 19 +++---------------- mm/slab.h | 22 ++++++++++++++++++++++ mm/slub.c | 14 ++------------ 3 files changed, 27 insertions(+), 28 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 14466a73d057..53e6b2687102 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1389,7 +1389,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid) { struct page *page; - int nr_pages; flags |= cachep->allocflags; @@ -1399,17 +1398,11 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, return NULL; } - if (memcg_charge_slab(page, flags, cachep->gfporder, cachep)) { + if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { __free_pages(page, cachep->gfporder); return NULL; } - nr_pages = (1 << cachep->gfporder); - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, nr_pages); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, nr_pages); - __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1424,12 +1417,6 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, static void kmem_freepages(struct kmem_cache *cachep, struct page *page) { int order = cachep->gfporder; - unsigned long nr_freed = (1 << order); - - if (cachep->flags & SLAB_RECLAIM_ACCOUNT) - mod_lruvec_page_state(page, NR_SLAB_RECLAIMABLE, -nr_freed); - else - mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE, -nr_freed); BUG_ON(!PageSlab(page)); __ClearPageSlabPfmemalloc(page); @@ -1438,8 +1425,8 @@ static void kmem_freepages(struct kmem_cache *cachep, struct page *page) page->mapping = NULL; if (current->reclaim_state) - current->reclaim_state->reclaimed_slab += nr_freed; - memcg_uncharge_slab(page, order, cachep); + current->reclaim_state->reclaimed_slab += 1 << order; + uncharge_slab_page(page, order, cachep); __free_pages(page, order); } diff --git a/mm/slab.h b/mm/slab.h index 4a261c97c138..0f5c5444acf1 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -205,6 +205,12 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer, void __kmem_cache_free_bulk(struct kmem_cache *, size_t, void **); int __kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **); +static inline int cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; +} + #ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */ @@ -352,6 +358,22 @@ static inline void memcg_link_cache(struct kmem_cache *s, #endif /* CONFIG_MEMCG_KMEM */ +static __always_inline int charge_slab_page(struct page *page, + gfp_t gfp, int order, + struct kmem_cache *s) +{ + memcg_charge_slab(page, gfp, order, s); + mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); + return 0; +} + +static __always_inline void uncharge_slab_page(struct page *page, int order, + struct kmem_cache *s) +{ + mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + memcg_uncharge_slab(page, order, s); +} + static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) { struct kmem_cache *cachep; diff --git a/mm/slub.c b/mm/slub.c index 195f61785c7d..90563c0b3b5f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1499,7 +1499,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, else page = __alloc_pages_node(node, flags, order); - if (page && memcg_charge_slab(page, flags, order, s)) { + if (page && charge_slab_page(page, flags, order, s)) { __free_pages(page, order); page = NULL; } @@ -1692,11 +1692,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) if (!page) return NULL; - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - 1 << oo_order(oo)); - inc_slabs_node(s, page_to_nid(page), page->objects); return page; @@ -1730,18 +1725,13 @@ static void __free_slab(struct kmem_cache *s, struct page *page) check_object(s, page, p, SLUB_RED_INACTIVE); } - mod_lruvec_page_state(page, - (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE, - -pages); - __ClearPageSlabPfmemalloc(page); __ClearPageSlab(page); page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - memcg_uncharge_slab(page, order, s); + uncharge_slab_page(page, order, s); __free_pages(page, order); } From patchwork Tue Apr 23 21:31:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10913781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 879241515 for ; Tue, 23 Apr 2019 22:24:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 721512886B for ; Tue, 23 Apr 2019 22:24:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61A8A28952; Tue, 23 Apr 2019 22:24:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FD152886B for ; Tue, 23 Apr 2019 22:24:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCE0B6B0005; Tue, 23 Apr 2019 18:24:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D54BB6B0008; Tue, 23 Apr 2019 18:24:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCE0E6B000A; Tue, 23 Apr 2019 18:24:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 620966B0005 for ; Tue, 23 Apr 2019 18:24:09 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id z29so8765487edb.4 for ; Tue, 23 Apr 2019 15:24:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=S4eDFS8Wd5P/PTGcdDOvQC7oyq3i19rfVFEECZZTV6I=; b=nc0+bbKGxl+WPMzmDj3EwV+pCLF6Rc5V+oLCfg9DEvO0pLKW0r64yLKG1ktFyl7oL6 dGBbEvrvn1tybd62/zf5Sst4z3WB1DsF6fOhMY/YSNnEfrlgug1qL0/tOS7ERnk2SSTV bQXG4HLEqJ7cISLymqVVVOIcaZ3NKyMzr9FCqLvBvQKaCq0ftzP35lapySZR0MHyLn07 CDBAyKlXX3LT9N1+SxkAonHLE5PdeAkYLfU9X/E/FSCK2fMMbRC7xOO4WkgXojPHcioW H5Q6YAcwQP5mg2m0aGXelbQRLGbbOoNypAFThP7NK/fidgQP1lIWO/2ujcsqow/j28xS JjZA== X-Gm-Message-State: APjAAAU+AVQjHC1ThbsoXvapOd5T5k4o2iFWqN8l9hnCZgr4PnIcr1Xu NeVwQXsv3yya6/tHQDggYM843/3PnBxCocRa51Pe/YqeOMpEq/JQsq9zEKtU12Gm6+OhIAcsfuT pbozIAIcs4IejvBzSRhvcjMlPTGjh2xd0Pob8FvIXKKhBOohO6/MRXSqNm3tqYyd5Zw== X-Received: by 2002:a50:9008:: with SMTP id b8mr13790294eda.115.1556058248827; Tue, 23 Apr 2019 15:24:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqyZ94EVSkoTACm5QL0BNT97NSfZlBee6kTSTub0BM0nSsI/8a2erVg/1PasGGBUhqdWUrC9 X-Received: by 2002:a50:9008:: with SMTP id b8mr13790245eda.115.1556058247481; Tue, 23 Apr 2019 15:24:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556058247; cv=none; d=google.com; s=arc-20160816; b=gKn5IBlOJZVlVPmwIE5HLP7FUnzQtWDHL/aKw/KhqkdbFsHITgk8GawgyfbkupFcK4 G7Kb/fyRoA8wcOHRI6Uv9ue0/pMB8oCBciXON72DX3VkbGpmGqwGQZldggxQ8ekMn9Ox SJNJC7LJGYhzMyME36SdNqXkxCz+JQL26JOWrUyvMidBHfNfYw8JclOBBlbyXwjHGjsH vZZ2V82adDV1cFfIOAHKjBKNWIYxzT51dGjsgYWguujbUz5hw+Q5Bu6G/+ESdb8NTziM dGzdyiUbYT+Evn1WU5/NQx67tLwW8g4FPQ9UxZOcwbSCDjSTS39I43et5LKjDlcuV58g ZsWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=S4eDFS8Wd5P/PTGcdDOvQC7oyq3i19rfVFEECZZTV6I=; b=k8HVTSvnS5P3BBwUdLserxEmJVXe8+AXRQEvRB+KLerkkdDt390EfabrcefmvdTn7I 888MQni0xNSjk/ZsvBu+1KngfpKCJVkMw2zmf2x1dU1b8GPoFcTUdGbw5cf7cu7qfoX6 FA87vhPwlT21aFN+kzUJ8USTiaKtIswGMSMTlepoaZD6yNIZk1AaROcO0hk8IjUUEXAE Fj9DFDGbGBvYlY29BeIP5Y17JC5+vtJK9RN/9fdANss/NLVrbzkO5Aoc8waqb7L4+6Lt RBR9W97PJh4PuYJZ+x8PSGsdBnviNN7c5z/iuQcN6Xp3DTvtWHJChdb1NiA18W8jLzZo ql1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=XPHnb6N2; spf=pass (google.com: domain of prvs=901699794a=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=901699794a=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com. [67.231.145.42]) by mx.google.com with ESMTPS id a3si1521053ejp.64.2019.04.23.15.24.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Apr 2019 15:24:07 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=901699794a=guro@fb.com designates 67.231.145.42 as permitted sender) client-ip=67.231.145.42; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=XPHnb6N2; spf=pass (google.com: domain of prvs=901699794a=guro@fb.com designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=901699794a=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3NMJ9td001022 for ; Tue, 23 Apr 2019 15:24:05 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=S4eDFS8Wd5P/PTGcdDOvQC7oyq3i19rfVFEECZZTV6I=; b=XPHnb6N2T/9n6q1OiaIGsPfohRhReLbw435iNCJqBwVzyNuuAe1PpnXg53nmFvd3Ed+y AM58jTdunvEw94XeXpE9z27pg6QkF90fFSmqZLLNHPI8U2bgclWn3wmS4cRIvVY+zk8F l5fX9JIyWDWZ4wstGzzwkFC1RX4qGnyDV/I= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2s203pucg8-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 23 Apr 2019 15:24:05 -0700 Received: from mx-out.facebook.com (2620:10d:c0a1:3::13) by mail.thefacebook.com (2620:10d:c021:18::172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA) id 15.1.1713.5; Tue, 23 Apr 2019 15:23:52 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id D178D1142D2E8; Tue, 23 Apr 2019 14:31:36 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v2 5/6] mm: rework non-root kmem_cache lifecycle management Date: Tue, 23 Apr 2019 14:31:32 -0700 Message-ID: <20190423213133.3551969-6-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190423213133.3551969-1-guro@fb.com> References: <20190423213133.3551969-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-23_08:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This commit makes several important changes in the lifecycle of a non-root kmem_cache, which also affect the lifecycle of a memory cgroup. Currently each charged slab page has a page->mem_cgroup pointer to the memory cgroup and holds a reference to it. Kmem_caches are held by the memcg and are released with it. It means that none of kmem_caches are released unless at least one reference to the memcg exists, which is not optimal. So the current scheme can be illustrated as: page->mem_cgroup->kmem_cache. To implement the slab memory reparenting we need to invert the scheme into: page->kmem_cache->mem_cgroup. Let's make every page to hold a reference to the kmem_cache (we already have a stable pointer), and make kmem_caches to hold a single reference to the memory cgroup. To make this possible we need to introduce a new percpu refcounter for non-root kmem_caches. The counter is initialized to the percpu mode, and is switched to atomic mode after deactivation, so we never shutdown an active cache. The counter is bumped for every charged page and also for every running allocation. So the kmem_cache can't be released unless all allocations complete. To shutdown non-active empty kmem_caches, let's reuse the infrastructure of the RCU-delayed work queue, used previously for the deactivation. After the generalization, it's perfectly suited for our needs. Since now we can release a kmem_cache at any moment after the deactivation, let's call sysfs_slab_remove() only from the shutdown path. It makes deactivation path simpler. Because we don't set the page->mem_cgroup pointer, we need to change the way how memcg-level stats is working for slab pages. We can't use mod_lruvec_page_state() helpers anymore, so switch over to mod_lruvec_state(). * I used the following simple approach to test the performance (stolen from another patchset by T. Harding): time find / -name fname-no-exist echo 2 > /proc/sys/vm/drop_caches repeat several times Results (I've chosen best results in several runs): orig patched real 0m0.700s 0m0.722s user 0m0.114s 0m0.120s sys 0m0.317s 0m0.324s real 0m0.729s 0m0.746s user 0m0.110s 0m0.139s sys 0m0.320s 0m0.317s real 0m0.745s 0m0.719s user 0m0.108s 0m0.124s sys 0m0.320s 0m0.323s So it looks like the difference is not noticeable in this test. Signed-off-by: Roman Gushchin --- include/linux/slab.h | 3 ++- mm/memcontrol.c | 16 ++++++----- mm/slab.h | 48 +++++++++++++++++++++++++++------ mm/slab_common.c | 63 +++++++++++++++++++++++--------------------- mm/slub.c | 12 +-------- 5 files changed, 85 insertions(+), 57 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 47923c173f30..1b54e5f83342 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -16,6 +16,7 @@ #include #include #include +#include /* @@ -152,7 +153,6 @@ int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); -void memcg_destroy_kmem_caches(struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -641,6 +641,7 @@ struct memcg_cache_params { struct mem_cgroup *memcg; struct list_head children_node; struct list_head kmem_caches_node; + struct percpu_ref refcnt; void (*work_fn)(struct kmem_cache *); union { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b2c39f187cbb..c9896105d8d5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2663,8 +2663,11 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) goto out; memcg_cachep = cache_from_memcg_idx(cachep, kmemcg_id); - if (likely(memcg_cachep)) + if (likely(memcg_cachep)) { + percpu_ref_get(&memcg_cachep->memcg_params.refcnt); + css_put(&memcg->css); return memcg_cachep; + } /* * If we are in a safe context (can wait, and not in interrupt @@ -2691,7 +2694,7 @@ struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) void memcg_kmem_put_cache(struct kmem_cache *cachep) { if (!is_root_cache(cachep)) - css_put(&cachep->memcg_params.memcg->css); + percpu_ref_put(&cachep->memcg_params.refcnt); } /** @@ -2719,9 +2722,6 @@ int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order, cancel_charge(memcg, nr_pages); return -ENOMEM; } - - page->mem_cgroup = memcg; - return 0; } @@ -2744,8 +2744,10 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order) memcg = get_mem_cgroup_from_current(); if (!mem_cgroup_is_root(memcg)) { ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg); - if (!ret) + if (!ret) { + page->mem_cgroup = memcg; __SetPageKmemcg(page); + } } css_put(&memcg->css); return ret; @@ -3238,7 +3240,7 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) memcg_offline_kmem(memcg); if (memcg->kmem_state == KMEM_ALLOCATED) { - memcg_destroy_kmem_caches(memcg); + WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); WARN_ON(page_counter_read(&memcg->kmem)); } diff --git a/mm/slab.h b/mm/slab.h index 0f5c5444acf1..61110b3035e7 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -173,6 +173,7 @@ void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void __kmemcg_cache_deactivate(struct kmem_cache *s); void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); +void kmemcg_cache_shutdown(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); struct seq_file; @@ -284,15 +285,37 @@ static __always_inline int memcg_charge_slab(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - if (is_root_cache(s)) - return 0; - return memcg_kmem_charge_memcg(page, gfp, order, s->memcg_params.memcg); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + int ret; + + memcg = s->memcg_params.memcg; + ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); + if (ret) + return ret; + + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); + + /* transer try_charge() page references to kmem_cache */ + percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); + css_put_many(&memcg->css, 1 << order); + + return 0; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct kmem_cache *s) { - memcg_kmem_uncharge(page, order); + struct mem_cgroup *memcg; + struct lruvec *lruvec; + + memcg = s->memcg_params.memcg; + lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); + memcg_kmem_uncharge_memcg(page, order, memcg); + + percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } extern void slab_init_memcg_params(struct kmem_cache *); @@ -362,15 +385,24 @@ static __always_inline int charge_slab_page(struct page *page, gfp_t gfp, int order, struct kmem_cache *s) { - memcg_charge_slab(page, gfp, order, s); - mod_lruvec_page_state(page, cache_vmstat_idx(s), 1 << order); - return 0; + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + 1 << order); + return 0; + } + + return memcg_charge_slab(page, gfp, order, s); } static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { - mod_lruvec_page_state(page, cache_vmstat_idx(s), -(1 << order)); + if (is_root_cache(s)) { + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), + -(1 << order)); + return; + } + memcg_uncharge_slab(page, order, s); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 4e5b4292a763..995920222127 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -45,6 +45,8 @@ static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work); static DECLARE_WORK(slab_caches_to_rcu_destroy_work, slab_caches_to_rcu_destroy_workfn); +static void kmemcg_queue_cache_shutdown(struct percpu_ref *percpu_ref); + /* * Set of flags that will prevent slab merging */ @@ -145,6 +147,12 @@ static int init_memcg_params(struct kmem_cache *s, struct memcg_cache_array *arr; if (root_cache) { + int ret = percpu_ref_init(&s->memcg_params.refcnt, + kmemcg_queue_cache_shutdown, + 0, GFP_KERNEL); + if (ret) + return ret; + s->memcg_params.root_cache = root_cache; INIT_LIST_HEAD(&s->memcg_params.children_node); INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); @@ -170,6 +178,8 @@ static void destroy_memcg_params(struct kmem_cache *s) { if (is_root_cache(s)) kvfree(rcu_access_pointer(s->memcg_params.memcg_caches)); + else + percpu_ref_exit(&s->memcg_params.refcnt); } static void free_memcg_params(struct rcu_head *rcu) @@ -225,6 +235,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) if (is_root_cache(s)) { list_add(&s->root_caches_node, &slab_root_caches); } else { + css_get(&memcg->css); s->memcg_params.memcg = memcg; list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); @@ -240,6 +251,7 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); + css_put(&s->memcg_params.memcg->css); } } #else @@ -708,16 +720,13 @@ static void kmemcg_after_rcu_workfn(struct work_struct *work) put_online_mems(); put_online_cpus(); - - /* done, put the ref from slab_deactivate_memcg_cache_rcu_sched() */ - css_put(&s->memcg_params.memcg->css); } /* * We need to grab blocking locks. Bounce to ->work. The * work item shares the space with the RCU head and can't be - * initialized eariler. -*/ + * initialized earlier. + */ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) { struct kmem_cache *s = container_of(head, struct kmem_cache, @@ -727,9 +736,28 @@ static void kmemcg_schedule_work_after_rcu(struct rcu_head *head) queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); } +static void kmemcg_cache_shutdown_after_rcu(struct kmem_cache *s) +{ + WARN_ON(shutdown_cache(s)); +} + +static void kmemcg_queue_cache_shutdown(struct percpu_ref *percpu_ref) +{ + struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache, + memcg_params.refcnt); + + if (s->memcg_params.root_cache->memcg_params.dying) + return; + + WARN_ON(s->memcg_params.work_fn); + s->memcg_params.work_fn = kmemcg_cache_shutdown_after_rcu; + call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); +} + static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { __kmemcg_cache_deactivate_after_rcu(s); + percpu_ref_kill(&s->memcg_params.refcnt); } static void kmemcg_cache_deactivate(struct kmem_cache *s) @@ -739,9 +767,6 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) if (s->memcg_params.root_cache->memcg_params.dying) return; - /* pin memcg so that @s doesn't get destroyed in the middle */ - css_get(&s->memcg_params.memcg->css); - WARN_ON_ONCE(s->memcg_params.work_fn); s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); @@ -775,28 +800,6 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) put_online_cpus(); } -void memcg_destroy_kmem_caches(struct mem_cgroup *memcg) -{ - struct kmem_cache *s, *s2; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry_safe(s, s2, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - /* - * The cgroup is about to be freed and therefore has no charges - * left. Hence, all its caches must be empty by now. - */ - BUG_ON(shutdown_cache(s)); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { struct memcg_cache_array *arr; diff --git a/mm/slub.c b/mm/slub.c index 90563c0b3b5f..5bfd899d201c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4027,18 +4027,8 @@ void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) { /* * Called with all the locks held after a sched RCU grace period. - * Even if @s becomes empty after shrinking, we can't know that @s - * doesn't have allocations already in-flight and thus can't - * destroy @s until the associated memcg is released. - * - * However, let's remove the sysfs files for empty caches here. - * Each cache has a lot of interface files which aren't - * particularly useful for empty draining caches; otherwise, we can - * easily end up with millions of unnecessary sysfs files on - * systems which have a lot of memory and transient cgroups. */ - if (!__kmem_cache_shrink(s)) - sysfs_slab_remove(s); + __kmem_cache_shrink(s); } void __kmemcg_cache_deactivate(struct kmem_cache *s) From patchwork Tue Apr 23 21:31:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 10914075 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6FBD71390 for ; Wed, 24 Apr 2019 04:59:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A1A928A0D for ; Wed, 24 Apr 2019 04:59:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A77228A11; Wed, 24 Apr 2019 04:59:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 594C328A0D for ; Wed, 24 Apr 2019 04:59:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 464F96B000A; Wed, 24 Apr 2019 00:59:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 414336B000C; Wed, 24 Apr 2019 00:59:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DD2A6B000D; Wed, 24 Apr 2019 00:59:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f70.google.com (mail-yw1-f70.google.com [209.85.161.70]) by kanga.kvack.org (Postfix) with ESMTP id 048B56B000A for ; Wed, 24 Apr 2019 00:59:42 -0400 (EDT) Received: by mail-yw1-f70.google.com with SMTP id o17so1825728ywd.22 for ; Tue, 23 Apr 2019 21:59:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:smtp-origin-hostprefix:from :smtp-origin-hostname:to:cc:smtp-origin-cluster:subject:date :message-id:in-reply-to:references:mime-version; bh=VEdpMvj5kP5nuI6Ztt7mzIbqR1F5iJNOXyr/8yu0PUA=; b=FcFiQX+uAZgmAQIW6sdoD13yXpSTl3Gs/1kVAkgK2lxtpvbkUjcFF790d9q8REXSxX d508PDc1lqZDSvcsEfXRiYd8Q58hCZkyhRvlEvgRjK0gWbbE9liChp2UR4tkPW5HPIFX 2RgmJeFbbbuXao4wIcPH9P5bf5zFX/9UrFJWStQzFHMtggIVEmMVxvzanpS9vdSWqQhM HS9a7tuOXIC7jXOQMGac8q+z545zrEm22THOgAs3Wrm7i5f1CoV2/o1ImvsOPbzPiR0T MZz319YQOpw8aDpFNB3VetbDyfC99wqJMvfaL/mDsyspHCXtySGIXZQd1oq5iwDvjYXN JZ8A== X-Gm-Message-State: APjAAAUfMYdZNid3euXV3b9jcrImf0REgUFxNCwHOXmBLK5gfuH3FKz3 3Ckk3IunIBWOEhiQeQL5Q4xbHQV1yYBUHomEhZrA4e3DG+Bbs2NGl8egZ9MxA5/Pcj3ZY/PLjPs N7t5j1QTHqMbJajkPKvBa3/Mr1Y2amfQzPEM6RLEzgWWZvsxLLf3jR3ZHWijFxGc7vA== X-Received: by 2002:a25:2a13:: with SMTP id q19mr23091754ybq.243.1556081981737; Tue, 23 Apr 2019 21:59:41 -0700 (PDT) X-Google-Smtp-Source: APXvYqy3xGyLR2STh4KYTB52c1HNTyBkPH9NvXwe7QthwUXV9wMuNlc2+zanAR3lG6EBN6xMPrJ9 X-Received: by 2002:a25:2a13:: with SMTP id q19mr23091720ybq.243.1556081980929; Tue, 23 Apr 2019 21:59:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556081980; cv=none; d=google.com; s=arc-20160816; b=MmtFxFe7klt2H2QipGDh/dZOU6uTp8/wC21GUcpCMi9s43eZBlG+ZHPxr3jPtVNtlt Qli9ycgPhcaZmsbkb7tH3YaiVCMh6+LbylQiIiQIK0nUByFo1/BWgHT5W5KFQwa35q2m OwMy34Fxt6eNbNbn/lxynZyxCK+jmvYmftpISfKS8sD0k4tKmCPniBmGTh5IlXqfPecJ JUjyeDVyOcOqYzEQCqUiDhYwEckJYL3ZvqDt6de87YDl01K4biFgtqC8NpPW4Q9E33aG 96k2DQ7mrQxZnOX7aG9dA5Zo6GhjYuZ922dMnUUPC26tkdKgX7BN/Ru6C2m9pylLjiIU X4bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject :smtp-origin-cluster:cc:to:smtp-origin-hostname:from :smtp-origin-hostprefix:dkim-signature; bh=VEdpMvj5kP5nuI6Ztt7mzIbqR1F5iJNOXyr/8yu0PUA=; b=qwO6EkrL1xxtZugIF2YVLCIRNBNPQGPUg3RoyW9xceTG1rSpaNgJe8L73MgswkpuY1 xWhSg5FVCJ20c0d1JVZqmywfFP763nJp7a32MqltNa20jsHqWGo3tSL+K6rs4nZJpA7O PTbscsCxz4Mz0RHb++7PWaoaQ9hKbwGimoX9V2K0dD2RQ9xF0Kn1cwm56TCxIzqaMzG5 sOhmIyrpWvAEuGp52W8rsE4EIsb3NF87aYtNBTGErnSeDilflpf9dU+fpt0F/M3uae+C FT7JP2OzByav+9MTom8bAEpkIyf7bOOzZbi0YDjUtZk2JqrDz41urskt3av9peWnFx3Y AZmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=DfRDcwz9; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com. [67.231.153.30]) by mx.google.com with ESMTPS id g129si3597461ywb.266.2019.04.23.21.59.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Apr 2019 21:59:40 -0700 (PDT) Received-SPF: pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.153.30 as permitted sender) client-ip=67.231.153.30; Authentication-Results: mx.google.com; dkim=pass header.i=@fb.com header.s=facebook header.b=DfRDcwz9; spf=pass (google.com: domain of prvs=90171118fe=guro@fb.com designates 67.231.153.30 as permitted sender) smtp.mailfrom="prvs=90171118fe=guro@fb.com"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=fb.com Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.27/8.16.0.27) with SMTP id x3O4xNBR031269 for ; Tue, 23 Apr 2019 21:59:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=facebook; bh=VEdpMvj5kP5nuI6Ztt7mzIbqR1F5iJNOXyr/8yu0PUA=; b=DfRDcwz9GNFsTOvKcjdYUIClLw2lmbenktFncRuI/FTa8XUZ6VyjEodtAqNCMQl29Bhy 1DALn7surcj2kJNETxq0jo4Hn5+pQ8NquZhMWHyKLNswPeUbmqeulLzItHee2hTbDMAc zeVPjI2V9zlLEp1iFsjjb8L51n0Wi8QqcKY= Authentication-Results: fb.com; spf=pass smtp.mailfrom=guro@fb.com Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net with ESMTP id 2s2du18m6d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 23 Apr 2019 21:59:40 -0700 Received: from mx-out.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Tue, 23 Apr 2019 21:59:39 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id D5B971142D2EA; Tue, 23 Apr 2019 14:31:36 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: Andrew Morton CC: , , , Johannes Weiner , Michal Hocko , Rik van Riel , Shakeel Butt , Christoph Lameter , Vladimir Davydov , , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH v2 6/6] mm: reparent slab memory on cgroup removal Date: Tue, 23 Apr 2019 14:31:33 -0700 Message-ID: <20190423213133.3551969-7-guro@fb.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190423213133.3551969-1-guro@fb.com> References: <20190423213133.3551969-1-guro@fb.com> X-FB-Internal: Safe MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-24_03:,, signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904240042 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Let's reparent memcg slab memory on memcg offlining. This allows us to release the memory cgroup without waiting for the last outstanding kernel object (e.g. dentry used by another application). So instead of reparenting all accounted slab pages, let's do reparent a relatively small amount of kmem_caches. Reparenting is performed as a part of the deactivation process. Since the parent cgroup is already charged, everything we need to do is to splice the list of kmem_caches to the parent's kmem_caches list, swap the memcg pointer and drop the css refcounter for each kmem_cache and adjust the parent's css refcounter. Quite simple. Please, note that kmem_cache->memcg_params.memcg isn't a stable pointer anymore. It's safe to read it under rcu_read_lock() or with slab_mutex held. We can race with the slab allocation and deallocation paths. It's not a big problem: parent's charge and slab global stats are always correct, and we don't care anymore about the child usage and global stats. The child cgroup is already offline, so we don't use or show it anywhere. Local slab stats (NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE) aren't used anywhere except count_shadow_nodes(). But even there it won't break anything: after reparenting "nodes" will be 0 on child level (because we're already reparenting shrinker lists), and on parent level page stats always were 0, and this patch won't change anything. Signed-off-by: Roman Gushchin --- include/linux/slab.h | 4 ++-- mm/memcontrol.c | 14 ++++++++------ mm/slab.h | 14 +++++++++----- mm/slab_common.c | 23 ++++++++++++++++++++--- 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1b54e5f83342..109cab2ad9b4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -152,7 +152,7 @@ void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); -void memcg_deactivate_kmem_caches(struct mem_cgroup *); +void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *); /* * Please use this macro to create slab caches. Simply specify the @@ -638,7 +638,7 @@ struct memcg_cache_params { bool dying; }; struct { - struct mem_cgroup *memcg; + struct mem_cgroup __rcu *memcg; struct list_head children_node; struct list_head kmem_caches_node; struct percpu_ref refcnt; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c9896105d8d5..27ae253922da 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3201,15 +3201,15 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) */ memcg->kmem_state = KMEM_ALLOCATED; - memcg_deactivate_kmem_caches(memcg); - - kmemcg_id = memcg->kmemcg_id; - BUG_ON(kmemcg_id < 0); - parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; + memcg_deactivate_kmem_caches(memcg, parent); + + kmemcg_id = memcg->kmemcg_id; + BUG_ON(kmemcg_id < 0); + /* * Change kmemcg_id of this cgroup and all its descendants to the * parent's id, and then move all entries from this cgroup's list_lrus @@ -3242,7 +3242,6 @@ static void memcg_free_kmem(struct mem_cgroup *memcg) if (memcg->kmem_state == KMEM_ALLOCATED) { WARN_ON(!list_empty(&memcg->kmem_caches)); static_branch_dec(&memcg_kmem_enabled_key); - WARN_ON(page_counter_read(&memcg->kmem)); } } #else @@ -4654,6 +4653,9 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) /* The following stuff does not apply to the root */ if (!parent) { +#ifdef CONFIG_MEMCG_KMEM + INIT_LIST_HEAD(&memcg->kmem_caches); +#endif root_mem_cgroup = memcg; return &memcg->css; } diff --git a/mm/slab.h b/mm/slab.h index 61110b3035e7..68c5fc6e557e 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -289,10 +289,11 @@ static __always_inline int memcg_charge_slab(struct page *page, struct lruvec *lruvec; int ret; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); ret = memcg_kmem_charge_memcg(page, gfp, order, memcg); if (ret) - return ret; + goto out; lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order); @@ -300,8 +301,9 @@ static __always_inline int memcg_charge_slab(struct page *page, /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); css_put_many(&memcg->css, 1 << order); - - return 0; +out: + rcu_read_unlock(); + return ret; } static __always_inline void memcg_uncharge_slab(struct page *page, int order, @@ -310,10 +312,12 @@ static __always_inline void memcg_uncharge_slab(struct page *page, int order, struct mem_cgroup *memcg; struct lruvec *lruvec; - memcg = s->memcg_params.memcg; + rcu_read_lock(); + memcg = rcu_dereference(s->memcg_params.memcg); lruvec = mem_cgroup_lruvec(page_pgdat(page), memcg); mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order)); memcg_kmem_uncharge_memcg(page, order, memcg); + rcu_read_unlock(); percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 995920222127..36673a43ed31 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -236,7 +236,7 @@ void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) list_add(&s->root_caches_node, &slab_root_caches); } else { css_get(&memcg->css); - s->memcg_params.memcg = memcg; + rcu_assign_pointer(s->memcg_params.memcg, memcg); list_add(&s->memcg_params.children_node, &s->memcg_params.root_cache->memcg_params.children); list_add(&s->memcg_params.kmem_caches_node, @@ -251,7 +251,8 @@ static void memcg_unlink_cache(struct kmem_cache *s) } else { list_del(&s->memcg_params.children_node); list_del(&s->memcg_params.kmem_caches_node); - css_put(&s->memcg_params.memcg->css); + mem_cgroup_put(rcu_dereference_protected(s->memcg_params.memcg, + lockdep_is_held(&slab_mutex))); } } #else @@ -772,11 +773,13 @@ static void kmemcg_cache_deactivate(struct kmem_cache *s) call_rcu(&s->memcg_params.rcu_head, kmemcg_schedule_work_after_rcu); } -void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) +void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg, + struct mem_cgroup *parent) { int idx; struct memcg_cache_array *arr; struct kmem_cache *s, *c; + unsigned int nr_reparented; idx = memcg_cache_id(memcg); @@ -794,6 +797,20 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg) kmemcg_cache_deactivate(c); arr->entries[idx] = NULL; } + if (memcg != parent) { + nr_reparented = 0; + list_for_each_entry(s, &memcg->kmem_caches, + memcg_params.kmem_caches_node) { + rcu_assign_pointer(s->memcg_params.memcg, parent); + css_put(&memcg->css); + nr_reparented++; + } + if (nr_reparented) { + list_splice_init(&memcg->kmem_caches, + &parent->kmem_caches); + css_get_many(&parent->css, nr_reparented); + } + } mutex_unlock(&slab_mutex); put_online_mems();