From patchwork Mon Mar 30 02:32:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 11464483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21A1713A4 for ; Mon, 30 Mar 2020 02:33:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C6B4220735 for ; Mon, 30 Mar 2020 02:33:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=joelfernandes.org header.i=@joelfernandes.org header.b="DDiCsr4b" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6B4220735 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=joelfernandes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A9E656B0074; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A28DA6B0075; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 852DF6B0078; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id 69EA26B0074 for ; Sun, 29 Mar 2020 22:33:42 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 692AB8248D7C for ; Mon, 30 Mar 2020 02:33:42 +0000 (UTC) X-FDA: 76650457884.10.order61_5a86e2e8d962b X-Spam-Summary: 2,0,0,808fb5177e6df9ab,d41d8cd98f00b204,joel@joelfernandes.org,,RULES_HIT:4:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2638:2693:2907:2914:2918:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:5007:6117:6119:6261:6653:6742:7514:7576:7875:7903:8603:9010:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13141:13161:13229:13230:13868:13894:14096:14394:21080:21324:21433:21444:21451:21627:21740:21789:21987:21990:30034:30045:30054:30070:30091,0,RBL:209.85.219.66:@joelfernandes.org:.lbl8.mailshell.net-66.100.201.201 62.2.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: order61_5a86e2e8d962b X-Filterd-Recvd-Size: 15403 Received: from mail-qv1-f66.google.com (mail-qv1-f66.google.com [209.85.219.66]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Mar 2020 02:33:41 +0000 (UTC) Received: by mail-qv1-f66.google.com with SMTP id c28so8180643qvb.10 for ; Sun, 29 Mar 2020 19:33:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9O4vayA2XFJge9v6ySy8FCcWCe4sfIKtielmms+HdCM=; b=DDiCsr4b1kQ1XOVbiJHuWE2tSr7GVhfYZ3CVtapaOGyv93A54AwPAbKXIE+Zhjk1ck jBLtStB+C08LRv3D1lnqZAY9ylWA8GQs38UiehGUwB+iDui3NcWIkJEHSNHvUzxPdrBv yUWTRTijD9XGOmKJBTuPRNKmaltJ+0cVfcpiU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9O4vayA2XFJge9v6ySy8FCcWCe4sfIKtielmms+HdCM=; b=K11wEPDkfDjT4N8KVlL1Unt0Y9hODGKffNJIvUr7/kWiu9w4qU7soifRF5JeD9Bhum p0SojIk6pZp1pKeXmzuCOpNpBPVp/F4EZHZ3Tdhw+nT+l9XITPdsAlaOxpm06/4V7qAz aHnZpHVDM2uTJ6cg1oXscwZ/lZWmyyxFhFaHrYWbb/vCjt2f2N8s5A4/lO4Xo+CWviwx 8XkBFUKtmZrhY4UgV7CkTVR5sMDcEstpp8kLiuoleaBLPijvO/Nx1Kes7+EoT8c37qOD tT9DUF8zIkMi2pTARK0DYcFMuFH1oRrJPL1aug1xymtxm7UEzKmNNoKyV59KWKyCmDCJ FqQQ== X-Gm-Message-State: ANhLgQ0PC2VtDclryvXguXhGSER6TSVOCvvMlapmYel1Ed+EvEYLwfrN R+G51EjKVSHLwn9WZh1AccPUxaE9wI8= X-Google-Smtp-Source: ADFU+vtvCwVuLdvn7Q6HnMKU6Xbgf5np5t/oJIYTqKIBRTbWBG/d57XfI2taLG0QaD7vekSJxMljEQ== X-Received: by 2002:a05:6214:294:: with SMTP id l20mr9926511qvv.46.1585535621011; Sun, 29 Mar 2020 19:33:41 -0700 (PDT) Received: from joelaf.cam.corp.google.com ([2620:15c:6:12:9c46:e0da:efbf:69cc]) by smtp.gmail.com with ESMTPSA id q15sm10030625qtj.83.2020.03.29.19.33.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 29 Mar 2020 19:33:40 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Uladzislau Rezki (Sony)" , Joel Fernandes , Andrew Morton , Ingo Molnar , Josh Triplett , Lai Jiangshan , linux-mm@kvack.org, Mathieu Desnoyers , "Paul E. McKenney" , "Rafael J. Wysocki" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs Date: Sun, 29 Mar 2020 22:32:40 -0400 Message-Id: <20200330023248.164994-11-joel@joelfernandes.org> X-Mailer: git-send-email 2.26.0.rc2.310.g2932bb562d-goog In-Reply-To: <20200330023248.164994-1-joel@joelfernandes.org> References: <20200330023248.164994-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" To do so we use an array of common kvfree_rcu_bulk_data structure. It consists of two elements, index number 0 corresponds to SLAB ptrs., whereas vmalloc pointers can be accessed by using index number 1. The reason of not mixing pointers is to have an easy way to to distinguish them. It is also the preparation patch for head-less objects support. When an object is head-less we can not queue it into any list, instead a pointer is placed directly into an array. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Joel Fernandes (Google) Reported-by: kbuild test robot Reported-by: kbuild test robot --- kernel/rcu/tree.c | 179 ++++++++++++++++++++++++++++------------------ 1 file changed, 108 insertions(+), 71 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cfe456e68c644..8fbc8450284db 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2808,38 +2808,36 @@ EXPORT_SYMBOL_GPL(call_rcu); #define KFREE_N_BATCHES 2 /** - * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers + * struct kvfree_rcu_bulk_data - single block to store kvfree() pointers * @nr_records: Number of active pointers in the array - * @records: Array of the kfree_rcu() pointers * @next: Next bulk object in the block chain - * @head_free_debug: For debug, when CONFIG_DEBUG_OBJECTS_RCU_HEAD is set + * @records: Array of the SLAB pointers */ -struct kfree_rcu_bulk_data { +struct kvfree_rcu_bulk_data { unsigned long nr_records; - void *records[KFREE_BULK_MAX_ENTR]; - struct kfree_rcu_bulk_data *next; + struct kvfree_rcu_bulk_data *next; + void *records[]; }; /* * This macro defines how many entries the "records" array * will contain. It is based on the fact that the size of - * kfree_rcu_bulk_data structure becomes exactly one page. + * kvfree_rcu_bulk_data become exactly one page. */ -#define KFREE_BULK_MAX_ENTR \ - ((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *)) +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) /** * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period * @head_free: List of kfree_rcu() objects waiting for a grace period - * @bhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period + * @bkvhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period * @krcp: Pointer to @kfree_rcu_cpu structure */ - struct kfree_rcu_cpu_work { struct rcu_work rcu_work; struct rcu_head *head_free; - struct kfree_rcu_bulk_data *bhead_free; + struct kvfree_rcu_bulk_data *bkvhead_free[2]; struct kfree_rcu_cpu *krcp; }; @@ -2861,8 +2859,9 @@ struct kfree_rcu_cpu_work { */ struct kfree_rcu_cpu { struct rcu_head *head; - struct kfree_rcu_bulk_data *bhead; - struct kfree_rcu_bulk_data *bcached; + struct kvfree_rcu_bulk_data *bkvhead[2]; + struct kvfree_rcu_bulk_data *bkvcache[2]; + struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES]; spinlock_t lock; struct delayed_work monitor_work; @@ -2875,7 +2874,7 @@ struct kfree_rcu_cpu { static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc); static __always_inline void -debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) +debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead) { #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD for (int i = 0; i < bhead->nr_records; i++) @@ -2890,45 +2889,77 @@ debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead) static void kfree_rcu_work(struct work_struct *work) { unsigned long flags; + struct kvfree_rcu_bulk_data *bkhead, *bknext; + struct kvfree_rcu_bulk_data *bvhead, *bvnext; struct rcu_head *head, *next; - struct kfree_rcu_bulk_data *bhead, *bnext; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; + int i; krwp = container_of(to_rcu_work(work), - struct kfree_rcu_cpu_work, rcu_work); + struct kfree_rcu_cpu_work, rcu_work); + krcp = krwp->krcp; spin_lock_irqsave(&krcp->lock, flags); + /* Channel 1. */ + bkhead = krwp->bkvhead_free[0]; + krwp->bkvhead_free[0] = NULL; + + /* Channel 2. */ + bvhead = krwp->bkvhead_free[1]; + krwp->bkvhead_free[1] = NULL; + + /* Channel 3. */ head = krwp->head_free; krwp->head_free = NULL; - bhead = krwp->bhead_free; - krwp->bhead_free = NULL; spin_unlock_irqrestore(&krcp->lock, flags); - /* "bhead" is now private, so traverse locklessly. */ - for (; bhead; bhead = bnext) { - bnext = bhead->next; + /* kmalloc()/kfree() channel. */ + for (; bkhead; bkhead = bknext) { + bknext = bkhead->next; - debug_rcu_bhead_unqueue(bhead); + debug_rcu_bhead_unqueue(bkhead); rcu_lock_acquire(&rcu_callback_map); trace_rcu_invoke_kfree_bulk_callback(rcu_state.name, - bhead->nr_records, bhead->records); + bkhead->nr_records, bkhead->records); + + kfree_bulk(bkhead->nr_records, bkhead->records); + rcu_lock_release(&rcu_callback_map); + + if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead)) + free_page((unsigned long) bkhead); + + cond_resched_tasks_rcu_qs(); + } + + /* vmalloc()/vfree() channel. */ + for (; bvhead; bvhead = bvnext) { + bvnext = bvhead->next; + + debug_rcu_bhead_unqueue(bvhead); - kfree_bulk(bhead->nr_records, bhead->records); + rcu_lock_acquire(&rcu_callback_map); + for (i = 0; i < bvhead->nr_records; i++) { + trace_rcu_invoke_kvfree_callback(rcu_state.name, + (struct rcu_head *) bvhead->records[i], 0); + vfree(bvhead->records[i]); + } rcu_lock_release(&rcu_callback_map); - if (cmpxchg(&krcp->bcached, NULL, bhead)) - free_page((unsigned long) bhead); + if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead)) + free_page((unsigned long) bvhead); cond_resched_tasks_rcu_qs(); } /* - * We can end up here either with 1) vmalloc() pointers or 2) were low - * on memory and could not allocate a bulk array. It can happen under - * low memory condition when an allocation gets failed, so the "bulk" - * path can not be temporarly used. + * This path covers emergency case only due to high + * memory pressure also means low memory condition, + * when we could not allocate a bulk array. + * + * Under that condition an object is queued to the + * list instead. */ for (; head; head = next) { unsigned long offset = (unsigned long)head->func; @@ -2965,21 +2996,34 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) krwp = &(krcp->krw_arr[i]); /* - * Try to detach bhead or head and attach it over any + * Try to detach bkvhead or head and attach it over any * available corresponding free channel. It can be that * a previous RCU batch is in progress, it means that * immediately to queue another one is not possible so * return false to tell caller to retry. */ - if ((krcp->bhead && !krwp->bhead_free) || + if ((krcp->bkvhead[0] && !krwp->bkvhead_free[0]) || + (krcp->bkvhead[1] && !krwp->bkvhead_free[1]) || (krcp->head && !krwp->head_free)) { - /* Channel 1. */ - if (!krwp->bhead_free) { - krwp->bhead_free = krcp->bhead; - krcp->bhead = NULL; + /* + * Channel 1 corresponds to SLAB ptrs. + */ + if (!krwp->bkvhead_free[0]) { + krwp->bkvhead_free[0] = krcp->bkvhead[0]; + krcp->bkvhead[0] = NULL; + } + + /* + * Channel 2 corresponds to vmalloc ptrs. + */ + if (!krwp->bkvhead_free[1]) { + krwp->bkvhead_free[1] = krcp->bkvhead[1]; + krcp->bkvhead[1] = NULL; } - /* Channel 2. */ + /* + * Channel 3 corresponds to emergency path. + */ if (!krwp->head_free) { krwp->head_free = krcp->head; krcp->head = NULL; @@ -2988,10 +3032,11 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp) WRITE_ONCE(krcp->count, 0); /* - * One work is per one batch, so there are two "free channels", - * "bhead_free" and "head_free" the batch can handle. It can be - * that the work is in the pending state when two channels have - * been detached following each other, one by one. + * One work is per one batch, so there are three + * "free channels", the batch can handle. It can + * be that the work is in the pending state when + * channels have been detached following by each + * other. */ queue_rcu_work(system_wq, &krwp->rcu_work); queued = true; @@ -3036,26 +3081,25 @@ static void kfree_rcu_monitor(struct work_struct *work) } static inline bool -kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, - struct rcu_head *head, rcu_callback_t func) +kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr) { - struct kfree_rcu_bulk_data *bnode; + struct kvfree_rcu_bulk_data *bnode; + int idx; if (unlikely(!krcp->initialized)) return false; lockdep_assert_held(&krcp->lock); + idx = !is_vmalloc_addr(ptr) ? 0:1; /* Check if a new block is required. */ - if (!krcp->bhead || - krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) { - bnode = xchg(&krcp->bcached, NULL); - if (!bnode) { - WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE); - - bnode = (struct kfree_rcu_bulk_data *) + if (!krcp->bkvhead[idx] || + krcp->bkvhead[idx]->nr_records == + KVFREE_BULK_MAX_ENTR) { + bnode = xchg(&krcp->bkvcache[idx], NULL); + if (!bnode) + bnode = (struct kvfree_rcu_bulk_data *) __get_free_page(GFP_NOWAIT | __GFP_NOWARN); - } /* Switch to emergency path. */ if (unlikely(!bnode)) @@ -3063,30 +3107,30 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, /* Initialize the new block. */ bnode->nr_records = 0; - bnode->next = krcp->bhead; + bnode->next = krcp->bkvhead[idx]; /* Attach it to the head. */ - krcp->bhead = bnode; + krcp->bkvhead[idx] = bnode; } /* Finally insert. */ - krcp->bhead->records[krcp->bhead->nr_records++] = - (void *) head - (unsigned long) func; + krcp->bkvhead[idx]->records + [krcp->bkvhead[idx]->nr_records++] = ptr; return true; } /* - * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace - * period. Please note there are two paths are maintained, one is the main one - * that uses kfree_bulk() interface and second one is emergency one, that is - * used only when the main path can not be maintained temporary, due to memory - * pressure. + * Queue a request for lazy invocation of appropriate free routine after a + * grace period. Please note there are three paths are maintained, two are the + * main ones that use array of pointers interface and third one is emergency + * one, that is used only when the main path can not be maintained temporary, + * due to memory pressure. * * Each kvfree_call_rcu() request is added to a batch. The batch will be drained * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will * be free'd in workqueue context. This allows us to: batch requests together to - * reduce the number of grace periods during heavy kfree_rcu() load. + * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load. */ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { @@ -3110,17 +3154,10 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) } /* - * We do not queue vmalloc pointers into array, - * instead they are just queued to the list. We - * do it because of: - * a) to distinguish kmalloc()/vmalloc() ptrs; - * b) there is no vmalloc_bulk() interface. - * * Under high memory pressure GFP_NOWAIT can fail, * in that case the emergency path is maintained. */ - if (is_vmalloc_addr(ptr) || - !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) { + if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) { head->func = func; head->next = krcp->head; krcp->head = head;