From patchwork Tue Apr 28 20:59:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 11515597 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18B0B112C for ; Tue, 28 Apr 2020 21:00:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CE38E2072A for ; Tue, 28 Apr 2020 21:00:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CWL6Mhxs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CE38E2072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2889C8E001B; Tue, 28 Apr 2020 17:00:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 210EC8E0007; Tue, 28 Apr 2020 17:00:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D9348E001B; Tue, 28 Apr 2020 17:00:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0009.hostedemail.com [216.40.44.9]) by kanga.kvack.org (Postfix) with ESMTP id DE90B8E0007 for ; Tue, 28 Apr 2020 17:00:00 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 9FC8D180AD802 for ; Tue, 28 Apr 2020 21:00:00 +0000 (UTC) X-FDA: 76758480960.22.stone87_5759df62e0e36 X-Spam-Summary: 2,0,0,5837024122872888,d41d8cd98f00b204,urezki@gmail.com,,RULES_HIT:2:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2559:2562:2907:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6119:6261:6653:7514:7875:7903:8660:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13148:13211:13229:13230:13894:14096:14394:14687:21080:21324:21433:21444:21451:21627:21666:21740:21789:21939:21987:21990:30045:30054:30070,0,RBL:209.85.208.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:16,LUA_SUMMARY:none X-HE-Tag: stone87_5759df62e0e36 X-Filterd-Recvd-Size: 8648 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Apr 2020 21:00:00 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id l19so199594lje.10 for ; Tue, 28 Apr 2020 14:00:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kEunEa6ntH1HytcdQSdsY2CsdbEZEQ/S23NHaGYEDh0=; b=CWL6Mhxss7Hv7VG0ZlIx12jk8sHFvdJzDPhUwW40jXJ/R4y20EdTQ5Nqk1C7iyTu1A /NnaS20p/fcIuDjH62b+awTfCcP5fdc5Zo1ZuxyntzXrA0F7TkzLFSEPnyQQzk2x6bxU w/I7BKkaKWNxFO06OxepNfWb5QfIp+Sf8gJXVuVilwidIM+OorINFbKbFH5kz871t/6p FMX6JxEr+RNc9VK/aYO6sW9mmQQPTx2smO4BDGaqPkYIbtlYMAsOH9kNQoDbxhgNUB49 snoL2XjyusdH6OFmnR/L0Bo1kn5nTu1xdtc4R2+siuvSwuj/zYa49CuhQK+YFIhdclY8 jSog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kEunEa6ntH1HytcdQSdsY2CsdbEZEQ/S23NHaGYEDh0=; b=VDktAOTVuBAhXKIrf4gGuL8cNHlXuXS6J55GkLDNsNNXJPUlS3hbukfIu9cqE9Kz4c 3Rn9b+dEUeXOXAV0r7cuWNUb6oFvG/gGfneHc8KsQ2QJ9XIjOFs7JXujK2y56Wk66+Ug OImzfLXRlTXNtWuLjYCyf0dgWtipfgDkDvg8KU14cmIgXLSvspInih7gSyK1lhbFrgBd Ggl3+YDWxn6Avhzo7Uy3DwQDyelm5Bm9ExBjieA65Td/5j3lrizDpryHHXgCOuyZjOzl MmCMw9iragGowzgrcyTVT4ieCX63pv3bdZaaNS0+zz1bDJFP1HwZum2gpHR7UMgEy3EU qqgg== X-Gm-Message-State: AGi0PuYNgcWEyAHzQbo1j588A6A+u+gFQFywqXE07ctDwRGJ8DFX/ZoL C1SXOlCyMwqweneM1kLakVQ= X-Google-Smtp-Source: APiQypIuEVHqAxoNEWEpH8q58I4tuk4mhXqNnaVsvdWrBoPEevlMRUF3qOaGJoFkVwIyEeRklqxn/g== X-Received: by 2002:ac2:4426:: with SMTP id w6mr20163280lfl.8.1588107598830; Tue, 28 Apr 2020 13:59:58 -0700 (PDT) Received: from pc638.lan (h5ef52e31.seluork.dyn.perspektivbredband.net. [94.245.46.49]) by smtp.gmail.com with ESMTPSA id z21sm295483ljh.42.2020.04.28.13.59.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Apr 2020 13:59:58 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: LKML , linux-mm@kvack.org Cc: Andrew Morton , "Paul E . McKenney" , "Theodore Y . Ts'o" , Matthew Wilcox , Joel Fernandes , RCU , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 22/24] rcu/tiny: support reclaim for head-less object Date: Tue, 28 Apr 2020 22:59:01 +0200 Message-Id: <20200428205903.61704-23-urezki@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200428205903.61704-1-urezki@gmail.com> References: <20200428205903.61704-1-urezki@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make a kvfree_call_rcu() function to support head-less freeing. Same as for tree-RCU, for such purpose we store pointers in array. SLAB and vmalloc ptrs. are mixed and coexist together. Under high memory pressure it can be that maintaining of arrays becomes impossible. Objects with an rcu_head are released via call_rcu(). When it comes to the head-less variant, the kvfree() call is directly inlined, i.e. we do the same as for tree-RCU: a) wait until a grace period has elapsed; b) direct inlining of the kvfree() call. Thus the current context has to follow might_sleep() annotation. Also please note that for tiny-RCU any call of synchronize_rcu() is actually a quiescent state, therefore (a) does nothing. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki (Sony) --- kernel/rcu/tiny.c | 157 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 156 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index 508c82faa45c..b1c31a935db9 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -40,6 +40,29 @@ static struct rcu_ctrlblk rcu_ctrlblk = { .curtail = &rcu_ctrlblk.rcucblist, }; +/* Can be common with tree-RCU. */ +#define KVFREE_DRAIN_JIFFIES (HZ / 50) + +/* Can be common with tree-RCU. */ +struct kvfree_rcu_bulk_data { + unsigned long nr_records; + struct kvfree_rcu_bulk_data *next; + void *records[]; +}; + +/* Can be common with tree-RCU. */ +#define KVFREE_BULK_MAX_ENTR \ + ((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *)) + +static struct kvfree_rcu_bulk_data *kvhead; +static struct kvfree_rcu_bulk_data *kvhead_free; +static struct kvfree_rcu_bulk_data *kvcache; + +static DEFINE_STATIC_KEY_FALSE(rcu_init_done); +static struct delayed_work monitor_work; +static struct rcu_work rcu_work; +static bool monitor_todo; + void rcu_barrier(void) { wait_rcu_gp(call_rcu); @@ -177,9 +200,137 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } EXPORT_SYMBOL_GPL(call_rcu); +static inline bool +kvfree_call_rcu_add_ptr_to_bulk(void *ptr) +{ + struct kvfree_rcu_bulk_data *bnode; + + if (!kvhead || kvhead->nr_records == KVFREE_BULK_MAX_ENTR) { + bnode = xchg(&kvcache, NULL); + if (!bnode) + bnode = (struct kvfree_rcu_bulk_data *) + __get_free_page(GFP_NOWAIT | __GFP_NOWARN); + + if (unlikely(!bnode)) + return false; + + /* Initialize the new block. */ + bnode->nr_records = 0; + bnode->next = kvhead; + + /* Attach it to the bvhead. */ + kvhead = bnode; + } + + /* Done. */ + kvhead->records[kvhead->nr_records++] = ptr; + return true; +} + +static void +kvfree_rcu_work(struct work_struct *work) +{ + struct kvfree_rcu_bulk_data *kvhead_tofree, *next; + unsigned long flags; + int i; + + local_irq_save(flags); + kvhead_tofree = kvhead_free; + kvhead_free = NULL; + local_irq_restore(flags); + + /* Reclaim process. */ + for (; kvhead_tofree; kvhead_tofree = next) { + next = kvhead_tofree->next; + + for (i = 0; i < kvhead_tofree->nr_records; i++) { + debug_rcu_head_unqueue((struct rcu_head *) + kvhead_tofree->records[i]); + kvfree(kvhead_tofree->records[i]); + } + + if (cmpxchg(&kvcache, NULL, kvhead_tofree)) + free_page((unsigned long) kvhead_tofree); + } +} + +static inline bool +queue_kvfree_rcu_work(void) +{ + /* Check if the free channel is available. */ + if (kvhead_free) + return false; + + kvhead_free = kvhead; + kvhead = NULL; + + /* + * Queue the job for memory reclaim after GP. + */ + queue_rcu_work(system_wq, &rcu_work); + return true; +} + +static void kvfree_rcu_monitor(struct work_struct *work) +{ + unsigned long flags; + bool queued; + + local_irq_save(flags); + queued = queue_kvfree_rcu_work(); + if (queued) + /* Success. */ + monitor_todo = false; + local_irq_restore(flags); + + /* + * If previous RCU reclaim process is still in progress, + * schedule the work one more time to try again later. + */ + if (monitor_todo) + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); +} + void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) { - call_rcu(head, func); + unsigned long flags; + bool success; + void *ptr; + + if (head) { + ptr = (void *) head - (unsigned long) func; + } else { + might_sleep(); + ptr = (void *) func; + } + + if (debug_rcu_head_queue(ptr)) { + /* Probable double free, just leak. */ + WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n", + __func__, head); + return; + } + + local_irq_save(flags); + success = kvfree_call_rcu_add_ptr_to_bulk(ptr); + if (static_branch_likely(&rcu_init_done)) { + if (success && !monitor_todo) { + monitor_todo = true; + schedule_delayed_work(&monitor_work, + KVFREE_DRAIN_JIFFIES); + } + } + local_irq_restore(flags); + + if (!success) { + if (!head) { + synchronize_rcu(); + kvfree(ptr); + } else { + call_rcu(head, func); + } + } } EXPORT_SYMBOL_GPL(kvfree_call_rcu); @@ -188,4 +339,8 @@ void __init rcu_init(void) open_softirq(RCU_SOFTIRQ, rcu_process_callbacks); rcu_early_boot_tests(); srcu_init(); + + INIT_DELAYED_WORK(&monitor_work, kvfree_rcu_monitor); + INIT_RCU_WORK(&rcu_work, kvfree_rcu_work); + static_branch_enable(&rcu_init_done); }