From patchwork Thu Jun 27 20:39:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11020483 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 075F21575 for ; Thu, 27 Jun 2019 20:40:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ECAFA285BD for ; Thu, 27 Jun 2019 20:40:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DC75028706; Thu, 27 Jun 2019 20:40:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B5ACD285BD for ; Thu, 27 Jun 2019 20:40:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726726AbfF0UkA (ORCPT ); Thu, 27 Jun 2019 16:40:00 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:37682 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726463AbfF0Uj7 (ORCPT ); Thu, 27 Jun 2019 16:39:59 -0400 Received: by mail-qt1-f193.google.com with SMTP id y57so3995770qtk.4; Thu, 27 Jun 2019 13:39:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=AMywy7ALEe2t0mNJ1TRlsbiFePyakRxLczRqN/PYLqo=; b=R0VDCHQe0+ZVtPpHXez9PeF4IKo1fN/BqfuQtE78nnlRkmudxsQrwwOWjfkHoL5sKY N0Ur2BTQ62Cs6+R/42uHdn0Pso0IGX3tHyknB/AXOI69hG7Dia8Zt6MDTUsuAefjmMua Amli6efEBxs/T/55RsLg9AfCjqq/ZxPXniVAYKgayjMnyJWIPmEPOqwVIn/F0vmIt12g ItguiZP9RbnGqEPMz7RF7EpZ1xZh69kPWN3TBaaQTg+oclUpMByWLcny5yyFLHYakIRo /sz5Yggzahgf2fmL6TWYauIEGFlgYFtrTFbuRyUQZ2J5l9pb22W62vF4K1JIPvhGCC8N dV3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=AMywy7ALEe2t0mNJ1TRlsbiFePyakRxLczRqN/PYLqo=; b=Yv55e6iB+CMpiWQ4ku/2bCtuZhBT4z3fllqbDfTRJCvNxRT9h6Qi6Qbf/rYnXt3cu0 SLwU9fYMCgOJMM//z5w1bgZrD/GZzjmbA5bncowuyqtpr1td8AX411L3R63LVHFVOaEy qQlrvdCbWKFIxHMIf3XHdUANnLRGVZKVzw+ynBgmQ2fVFQ4gq5tDWKBIPEXNxJ19rxkL r3UDfWzunG7JdagFxvl+7oiNrzZehGTouWPgx/UzPumMNrOKkh/mYd4+AOtU/sRbsqkp 1HWXLgixibQSrx9aLogIgSlFhML0tK4xuEV5ePjpAjLR6NilocgFRSqyQCFson5voFSJ Ow1g== X-Gm-Message-State: APjAAAVuFrYJVCDtMhFK+GLvwtqrBFNdon8eKgANelfXL77iqH3onePf NrMshqCzB52rgI//dUApy0c= X-Google-Smtp-Source: APXvYqweuAgf29mRrCJ2Yfz4Yw3CYiuLfScCA+pqmcAtQjba1z/7eF/+dYrC95C0D1Syo5Ho5LZfgA== X-Received: by 2002:a0c:af18:: with SMTP id i24mr4742816qvc.115.1561667998558; Thu, 27 Jun 2019 13:39:58 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5a51]) by smtp.gmail.com with ESMTPSA id k55sm101322qtf.68.2019.06.27.13.39.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jun 2019 13:39:58 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: jack@suse.cz, josef@toxicpanda.com, clm@fb.com, dsterba@suse.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, kernel-team@fb.com, Tejun Heo Subject: [PATCH 1/5] cgroup, blkcg: Prepare some symbols for module and !CONFIG_CGROUP usages Date: Thu, 27 Jun 2019 13:39:48 -0700 Message-Id: <20190627203952.386785-2-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190627203952.386785-1-tj@kernel.org> References: <20190627203952.386785-1-tj@kernel.org> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP btrfs is going to use css_put() and wbc helpers to improve cgroup writeback support. Add dummy css_get() definition and export wbc helpers to prepare for module and !CONFIG_CGROUP builds. Signed-off-by: Tejun Heo Reported-by: kbuild test robot Reviewed-by: Jan Kara --- block/blk-cgroup.c | 1 + fs/fs-writeback.c | 3 +++ include/linux/cgroup.h | 1 + 3 files changed, 5 insertions(+) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 53b7bd4c7000..3319ab4ff262 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -47,6 +47,7 @@ struct blkcg blkcg_root; EXPORT_SYMBOL_GPL(blkcg_root); struct cgroup_subsys_state * const blkcg_root_css = &blkcg_root.css; +EXPORT_SYMBOL_GPL(blkcg_root_css); static struct blkcg_policy *blkcg_policy[BLKCG_MAX_POLS]; diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 9ebfb1b28430..a8a40bc26c2f 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -270,6 +270,7 @@ void __inode_attach_wb(struct inode *inode, struct page *page) if (unlikely(cmpxchg(&inode->i_wb, NULL, wb))) wb_put(wb); } +EXPORT_SYMBOL_GPL(__inode_attach_wb); /** * locked_inode_to_wb_and_lock_list - determine a locked inode's wb and lock it @@ -582,6 +583,7 @@ void wbc_attach_and_unlock_inode(struct writeback_control *wbc, if (unlikely(wb_dying(wbc->wb))) inode_switch_wbs(inode, wbc->wb_id); } +EXPORT_SYMBOL_GPL(wbc_attach_and_unlock_inode); /** * wbc_detach_inode - disassociate wbc from inode and perform foreign detection @@ -701,6 +703,7 @@ void wbc_detach_inode(struct writeback_control *wbc) wb_put(wbc->wb); wbc->wb = NULL; } +EXPORT_SYMBOL_GPL(wbc_detach_inode); /** * wbc_account_io - account IO issued during writeback diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index c0077adeea83..da9d2afbcf0c 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -687,6 +687,7 @@ void cgroup_path_from_kernfs_id(const union kernfs_node_id *id, struct cgroup_subsys_state; struct cgroup; +static inline void css_get(struct cgroup_subsys_state *css) {} static inline void css_put(struct cgroup_subsys_state *css) {} static inline int cgroup_attach_task_all(struct task_struct *from, struct task_struct *t) { return 0; } From patchwork Thu Jun 27 20:39:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11020497 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F5CA14BB for ; Thu, 27 Jun 2019 20:40:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15EF2285BD for ; Thu, 27 Jun 2019 20:40:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0A49528706; Thu, 27 Jun 2019 20:40:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E096285BD for ; Thu, 27 Jun 2019 20:40:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726766AbfF0UkE (ORCPT ); Thu, 27 Jun 2019 16:40:04 -0400 Received: from mail-qk1-f194.google.com ([209.85.222.194]:36304 "EHLO mail-qk1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726463AbfF0UkD (ORCPT ); Thu, 27 Jun 2019 16:40:03 -0400 Received: by mail-qk1-f194.google.com with SMTP id g18so2980084qkl.3; Thu, 27 Jun 2019 13:40:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=Z9P+uDMVPtBfoUAIKs3Fmsc6yAaDVnxFpK1kJ0YpSOA=; b=ANDxW4Dbui2Z3jjmv+vdQpYyZbSpd+mGmM6YnuYtYMfJ/smE9j4QuHgSfPsXmm4DRS PxLeC3UIGKpSw6M0BPXhSVRtjXBc+IhgRR/2RRiGbqGJMyu69u+5JuEhCwxmSfMIy/h+ LzfGho08ZfEGzKTsRQMhF1RjsDv/6hMHs3nURck1OY2rpCgSUN1jXJ+uyvnG7Z5qtnbc ie0m5KZpBzNzps1JDwhLkzxCh4QYUVGYeoO7v5NFi2lAUqyClZRflNMmBLgyWmayDNt/ EMrE02F3y+LX8qakM1RImLDdAFFo4GAQ5NuqdqZgbLhY0BknwZCgxLmItqK+vdGqhESY Ms0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=Z9P+uDMVPtBfoUAIKs3Fmsc6yAaDVnxFpK1kJ0YpSOA=; b=mRiak0l/GXXkZRGbPFZTAlIClPpsyxLRWYtzHYZXOMBdhhWJw1Vb4qFm+zn0RhcPJ3 H21qEhVxfasC8i1zVziQCSGjSSDYOIlgF+wJ2ttHxenUwre6Xa80bA1d5NhrONiga+gr ddlgln4NiDvmOJ1pqJZQgJugPW71HKgSqTevHzq5xSl8rU3MH4OFu5gaqhwF1CfT/53k tdDrsVL2zTWR0m8pXlyyMVqH8UaqDco990Y08J/8nPQk69KzClQA/LkNoQAr5C6oHejW 98jDGK7zhLaCjf4YdQGOkvkwE7rB7MnwKNTMuBGKD3Jc0kKLNSu703xqdktv6bQNFIGP SfWA== X-Gm-Message-State: APjAAAU+Gkrx8ztAfUd0IqjU40fYMDcGUB60bA+nWHGvDr7Tsu6dSHH5 iSSAL2M91tLyLwJAW8DkinY= X-Google-Smtp-Source: APXvYqw4fIESsqhpVDnxupIdT9Tu57hFl4bvw0OD6fCF3vrvisYnJs8hzZ6XmMDowQKAc4to4kAEpw== X-Received: by 2002:a37:815:: with SMTP id 21mr5282354qki.257.1561668001841; Thu, 27 Jun 2019 13:40:01 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5a51]) by smtp.gmail.com with ESMTPSA id i48sm99535qte.93.2019.06.27.13.40.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jun 2019 13:40:01 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: jack@suse.cz, josef@toxicpanda.com, clm@fb.com, dsterba@suse.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, kernel-team@fb.com, Tejun Heo Subject: [PATCH 2/5] blkcg, writeback: Rename wbc_account_io() to wbc_account_cgroup_owner() Date: Thu, 27 Jun 2019 13:39:49 -0700 Message-Id: <20190627203952.386785-3-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190627203952.386785-1-tj@kernel.org> References: <20190627203952.386785-1-tj@kernel.org> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP wbc_account_io() does a very specific job - try to see which cgroup is actually dirtying an inode and transfer its ownership to the majority dirtier if needed. The name is too generic and confusing. Let's rename it to something more specific. Signed-off-by: Tejun Heo Cc: Jan Kara Reviewed-by: Jan Kara --- Documentation/admin-guide/cgroup-v2.rst | 2 +- fs/btrfs/extent_io.c | 4 ++-- fs/buffer.c | 2 +- fs/ext4/page-io.c | 2 +- fs/f2fs/data.c | 4 ++-- fs/fs-writeback.c | 8 ++++---- fs/mpage.c | 2 +- include/linux/writeback.h | 8 ++++---- 8 files changed, 16 insertions(+), 16 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index cf88c1f98270..356a7a3dcb2f 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2108,7 +2108,7 @@ following two functions. a queue (device) has been associated with the bio and before submission. - wbc_account_io(@wbc, @page, @bytes) + wbc_account_cgroup_owner(@wbc, @page, @bytes) Should be called for each data segment being written out. While this function doesn't care exactly when it's called during the writeback session, it's the easiest and most diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index db337e53aab3..5106008f5e28 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2911,7 +2911,7 @@ static int submit_extent_page(unsigned int opf, struct extent_io_tree *tree, bio = NULL; } else { if (wbc) - wbc_account_io(wbc, page, page_size); + wbc_account_cgroup_owner(wbc, page, page_size); return 0; } } @@ -2924,7 +2924,7 @@ static int submit_extent_page(unsigned int opf, struct extent_io_tree *tree, bio->bi_opf = opf; if (wbc) { wbc_init_bio(wbc, bio); - wbc_account_io(wbc, page, page_size); + wbc_account_cgroup_owner(wbc, page, page_size); } *bio_ret = bio; diff --git a/fs/buffer.c b/fs/buffer.c index e450c55f6434..40547bbbea94 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -3093,7 +3093,7 @@ static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, if (wbc) { wbc_init_bio(wbc, bio); - wbc_account_io(wbc, bh->b_page, bh->b_size); + wbc_account_cgroup_owner(wbc, bh->b_page, bh->b_size); } submit_bio(bio); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index 4690618a92e9..56e287f5ee50 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -404,7 +404,7 @@ static int io_submit_add_bh(struct ext4_io_submit *io, ret = bio_add_page(io->io_bio, page, bh->b_size, bh_offset(bh)); if (ret != bh->b_size) goto submit_and_retry; - wbc_account_io(io->io_wbc, page, bh->b_size); + wbc_account_cgroup_owner(io->io_wbc, page, bh->b_size); io->io_next_block++; return 0; } diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index eda4181d2092..e1cab1717ac7 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -470,7 +470,7 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio) } if (fio->io_wbc && !is_read_io(fio->op)) - wbc_account_io(fio->io_wbc, page, PAGE_SIZE); + wbc_account_cgroup_owner(fio->io_wbc, page, PAGE_SIZE); bio_set_op_attrs(bio, fio->op, fio->op_flags); @@ -537,7 +537,7 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio) } if (fio->io_wbc) - wbc_account_io(fio->io_wbc, bio_page, PAGE_SIZE); + wbc_account_cgroup_owner(fio->io_wbc, bio_page, PAGE_SIZE); io->last_block_in_bio = fio->new_blkaddr; f2fs_trace_ios(fio, 0); diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index a8a40bc26c2f..0aef79e934bb 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -706,7 +706,7 @@ void wbc_detach_inode(struct writeback_control *wbc) EXPORT_SYMBOL_GPL(wbc_detach_inode); /** - * wbc_account_io - account IO issued during writeback + * wbc_account_cgroup_owner - account writeback to update inode cgroup ownership * @wbc: writeback_control of the writeback in progress * @page: page being written out * @bytes: number of bytes being written out @@ -715,8 +715,8 @@ EXPORT_SYMBOL_GPL(wbc_detach_inode); * controlled by @wbc. Keep the book for foreign inode detection. See * wbc_detach_inode(). */ -void wbc_account_io(struct writeback_control *wbc, struct page *page, - size_t bytes) +void wbc_account_cgroup_owner(struct writeback_control *wbc, struct page *page, + size_t bytes) { struct cgroup_subsys_state *css; int id; @@ -753,7 +753,7 @@ void wbc_account_io(struct writeback_control *wbc, struct page *page, else wbc->wb_tcand_bytes -= min(bytes, wbc->wb_tcand_bytes); } -EXPORT_SYMBOL_GPL(wbc_account_io); +EXPORT_SYMBOL_GPL(wbc_account_cgroup_owner); /** * inode_congested - test whether an inode is congested diff --git a/fs/mpage.c b/fs/mpage.c index 436a85260394..a63620cdb73a 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -647,7 +647,7 @@ static int __mpage_writepage(struct page *page, struct writeback_control *wbc, * the confused fail path above (OOM) will be very confused when * it finds all bh marked clean (i.e. it will not write anything) */ - wbc_account_io(wbc, page, PAGE_SIZE); + wbc_account_cgroup_owner(wbc, page, PAGE_SIZE); length = first_unmapped << blkbits; if (bio_add_page(bio, page, length, 0) < length) { bio = mpage_bio_submit(REQ_OP_WRITE, op_flags, bio); diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 738a0c24874f..dda5cf228172 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -188,8 +188,8 @@ void wbc_attach_and_unlock_inode(struct writeback_control *wbc, struct inode *inode) __releases(&inode->i_lock); void wbc_detach_inode(struct writeback_control *wbc); -void wbc_account_io(struct writeback_control *wbc, struct page *page, - size_t bytes); +void wbc_account_cgroup_owner(struct writeback_control *wbc, struct page *page, + size_t bytes); void cgroup_writeback_umount(void); /** @@ -291,8 +291,8 @@ static inline void wbc_init_bio(struct writeback_control *wbc, struct bio *bio) { } -static inline void wbc_account_io(struct writeback_control *wbc, - struct page *page, size_t bytes) +static inline void wbc_account_cgroup_owner(struct writeback_control *wbc, + struct page *page, size_t bytes) { } From patchwork Thu Jun 27 20:39:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11020485 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7404314BB for ; Thu, 27 Jun 2019 20:40:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6AE71285BD for ; Thu, 27 Jun 2019 20:40:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E99C28706; Thu, 27 Jun 2019 20:40:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05DB6285BD for ; Thu, 27 Jun 2019 20:40:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726795AbfF0UkH (ORCPT ); Thu, 27 Jun 2019 16:40:07 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:41061 "EHLO mail-qk1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726785AbfF0UkG (ORCPT ); Thu, 27 Jun 2019 16:40:06 -0400 Received: by mail-qk1-f195.google.com with SMTP id c11so2956762qkk.8; Thu, 27 Jun 2019 13:40:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=rSl5Em3E7t4PowLidlG5XJQX39ZMKusUKIIGePH/ZaI=; b=M2xcfW5/a5KT7xxFW2u20pf83/TLWWue8R/PvJp98eItgcG0KR5HAkEJaKay6cgAYz hcZWI7mnqLjIWP7yuA3zpAvSyE39gFO2SBOaZXUtW9nET16EnhritPjGtxlVjF4jCc0M bBjy+JCy4waWrXCPeLXqoBPXdK20yC4G56Ca37Xf3Q1GTqnsrJC3PeLPI/5QhCX2KiLy 5omBq5FrgEfiDtcrRSYyLidLrVzFQUyYd9eFS84BGqNptzBrVhhxePPfx5px9+x0jSqr iNMnontB+5QRT6QqVwfwE0ctQozPwpmpB60iEIFxgVjGrUkYQBerscXbZURfolLPEcKK /D5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=rSl5Em3E7t4PowLidlG5XJQX39ZMKusUKIIGePH/ZaI=; b=AchmqMKmQU4rJe1Yz/WcUcOQL4Aq4pXSYDMHPyvDmMOdBiyHNcJgQ07Fq6m5V3udD4 WEM+5+wp/IKEoDwxwMWL8EKouHib15onB7KAP1FYV9zZqnxHMxZkiB9zJJbbpPkMh1Rs Z7qVMcDZTZN+M2aGdr/snZTOvPMBe7D03kEhLazEkf4qjk140jLspx289Qw0MuwozPff bZQbEomzcyr1U55l6uZciYayujPp408CMBcPRNqs4trvZ9Oj8BI7qYyAmqPa6g02rgjb 7/bSEplMHZN/sePOlzCP4sp9EC3yW/8B9z2WoUATntmixryVlZFJeMp8vYDVeNPHHuNV 8m5A== X-Gm-Message-State: APjAAAUQbEgFyhM6hYtb2+ZDTYM8bNInfWBBCabkSYIvJqVtiW84KIaQ if/pOtSnEjyG29uNorWW698= X-Google-Smtp-Source: APXvYqx2JI0dl6XBW4xAWJ1qXPBVZ1WFgQyiXRixoGKykxjclQPXHWLff2XckRTwxx1XnTwBdnN18g== X-Received: by 2002:a37:a86:: with SMTP id 128mr5373463qkk.169.1561668004948; Thu, 27 Jun 2019 13:40:04 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5a51]) by smtp.gmail.com with ESMTPSA id o38sm120603qte.5.2019.06.27.13.40.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jun 2019 13:40:04 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: jack@suse.cz, josef@toxicpanda.com, clm@fb.com, dsterba@suse.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, kernel-team@fb.com, Tejun Heo Subject: [PATCH 3/5] blkcg, writeback: Add wbc->no_cgroup_owner Date: Thu, 27 Jun 2019 13:39:50 -0700 Message-Id: <20190627203952.386785-4-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190627203952.386785-1-tj@kernel.org> References: <20190627203952.386785-1-tj@kernel.org> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When writeback IOs are bounced through async layers, the IOs should only be accounted against the wbc from the original bdi writeback to avoid confusing cgroup inode ownership arbitration. Add wbc->no_cgroup_owner to allow disabling wbc cgroup owner accounting. This will be used make btrfs compression work well with cgroup IO control. v2: Renamed from no_wbc_acct to no_cgroup_owner and added comment as per Jan. Signed-off-by: Tejun Heo Reviewed-by: Josef Bacik Reviewed-by: Jan Kara --- fs/fs-writeback.c | 2 +- include/linux/writeback.h | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 0aef79e934bb..542b02d170f8 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -727,7 +727,7 @@ void wbc_account_cgroup_owner(struct writeback_control *wbc, struct page *page, * behind a slow cgroup. Ultimately, we want pageout() to kick off * regular writeback instead of writing things out itself. */ - if (!wbc->wb) + if (!wbc->wb || wbc->no_cgroup_owner) return; css = mem_cgroup_css_from_page(page); diff --git a/include/linux/writeback.h b/include/linux/writeback.h index dda5cf228172..33a50fa09fac 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -68,6 +68,15 @@ struct writeback_control { unsigned for_reclaim:1; /* Invoked from the page allocator */ unsigned range_cyclic:1; /* range_start is cyclic */ unsigned for_sync:1; /* sync(2) WB_SYNC_ALL writeback */ + + /* + * When writeback IOs are bounced through async layers, only the + * initial synchronous phase should be accounted towards inode + * cgroup ownership arbitration to avoid confusion. Later stages + * can set the following flag to disable the accounting. + */ + unsigned no_cgroup_owner:1; + #ifdef CONFIG_CGROUP_WRITEBACK struct bdi_writeback *wb; /* wb this writeback is issued under */ struct inode *inode; /* inode being written out */ From patchwork Thu Jun 27 20:39:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11020487 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74DF414BB for ; Thu, 27 Jun 2019 20:40:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6CE35285BD for ; Thu, 27 Jun 2019 20:40:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6108C28706; Thu, 27 Jun 2019 20:40:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 06F04285BD for ; Thu, 27 Jun 2019 20:40:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726820AbfF0UkK (ORCPT ); Thu, 27 Jun 2019 16:40:10 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:39467 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726805AbfF0UkI (ORCPT ); Thu, 27 Jun 2019 16:40:08 -0400 Received: by mail-qt1-f196.google.com with SMTP id i34so3979192qta.6; Thu, 27 Jun 2019 13:40:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=UB496nZNnRPiORNl4LCpW+PnNNrOz863nyL0U+bIG7o=; b=PJdskUZFeAVMJf1A7tBLKCASPDuqGxwbiVjGtUHLv01PnRojQUgWtVdCV3XUxiM9Dv uet7xMu3Kc1dl6zI43zJy/4LYs0iEAoVsi5ZOxi40g2XknDD4u/jS+IKx9gXXCuWSwUH ypdwm0b8zE6NMFAURX0Prj/n7eQRPAV0mBSUGCNPAftALQTTyu7DfvqqySdWw5Ju+mc7 q5o3/mz9PDs7qhb57ofgxxqb39UdKm6Gw+psPKYslnASuNoPFPP22XLAIEx0AIqrk8Hn bPDz7bRJAn3kVd9VMULMoyLdSjXv5pFbjBteoXehvw3BT17ZT9hFDTP6+Elyx+meCbb4 criw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=UB496nZNnRPiORNl4LCpW+PnNNrOz863nyL0U+bIG7o=; b=ntCSKUhelEZw60q2yhIr6OgF9dwpDTQZVbl1aZ/UFsSzT+M33CK65huvysZMfwpRv+ 53kyjbNsJXcaXNi8YxtugiJExPnLu/+CkQTbrzEdc9Q4BG7ykOk3mgFVM0EaLpgGnpJl Omzitu1mcD+wefQs0HrP29SxTBv6WLdAbcUAtcxmYQhC7d78NMtndRPIE6ZuoLHE+ClS o7FxhZWrhTbzL3YMvkpO9OdJ/KsJDP6LbOrKwcvQE/aPRAUo97sS1R2NsEXUbKvgioUU VzbnCLkiPIs1FLHjkD4dUZeTYR9ozgNy9gUMqB7Na9+lz3cd8C6sazN1isIJN1Mq5Xvr /oew== X-Gm-Message-State: APjAAAVXA/Wm1FQYJhERIunwP4/eNreNJTUA6PDO6D0JMXhG5t7efelo zN9ae2ONFGZGlhvEe464C90= X-Google-Smtp-Source: APXvYqyALzQqBghugGroUSJDHL5rxgItlFbAzVRgCrSAfe7FnvM0oRj/34UbQyYI0+xlAUaWDpFigQ== X-Received: by 2002:ac8:2b62:: with SMTP id 31mr5058917qtv.140.1561668007628; Thu, 27 Jun 2019 13:40:07 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5a51]) by smtp.gmail.com with ESMTPSA id v126sm97388qkd.27.2019.06.27.13.40.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jun 2019 13:40:07 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: jack@suse.cz, josef@toxicpanda.com, clm@fb.com, dsterba@suse.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, kernel-team@fb.com, Tejun Heo Subject: [PATCH 4/5] blkcg, writeback: Implement wbc_blkcg_css() Date: Thu, 27 Jun 2019 13:39:51 -0700 Message-Id: <20190627203952.386785-5-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190627203952.386785-1-tj@kernel.org> References: <20190627203952.386785-1-tj@kernel.org> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add a helper to determine the target blkcg from wbc. Signed-off-by: Tejun Heo Reviewed-by: Josef Bacik Reviewed-by: Jan Kara --- include/linux/writeback.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/writeback.h b/include/linux/writeback.h index 33a50fa09fac..e056a22075cf 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -11,6 +11,7 @@ #include #include #include +#include struct bio; @@ -101,6 +102,16 @@ static inline int wbc_to_write_flags(struct writeback_control *wbc) return 0; } +static inline struct cgroup_subsys_state * +wbc_blkcg_css(struct writeback_control *wbc) +{ +#ifdef CONFIG_CGROUP_WRITEBACK + if (wbc->wb) + return wbc->wb->blkcg_css; +#endif + return blkcg_root_css; +} + /* * A wb_domain represents a domain that wb's (bdi_writeback's) belong to * and are measured against each other in. There always is one global From patchwork Thu Jun 27 20:39:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejun Heo X-Patchwork-Id: 11020491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C1D01575 for ; Thu, 27 Jun 2019 20:40:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0471F285BD for ; Thu, 27 Jun 2019 20:40:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EC14F28706; Thu, 27 Jun 2019 20:40:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43D9D285BD for ; Thu, 27 Jun 2019 20:40:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726805AbfF0UkM (ORCPT ); Thu, 27 Jun 2019 16:40:12 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:39475 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726830AbfF0UkM (ORCPT ); Thu, 27 Jun 2019 16:40:12 -0400 Received: by mail-qt1-f193.google.com with SMTP id i34so3979365qta.6; Thu, 27 Jun 2019 13:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=asaU2u3+qE2V/tXvNuPV0itByGZXwrnNGauZ3AE0x8Q=; b=mWIjFi6ksn5M294UtD6OzjBoCcGAe+stAI7ZEU++C0SLPFQDR1HBVYjwQJCYLPaUlL wgPZe4orHMigt/L29ffRXdcLiUyd+1zty+hko8XcOLFP85nKpVkRPaPWPYiDj5H7iqgx pC5XDK6I0MHokMJ5E9gVGffJBBFLXPtPkpPIgs3jRXt/bDf2pgkCQzNKfnoO9D+ag2Ph jybYItkKi6WJHM/vxAeDgtZ3bh0bYcGRZ1anuBwPj2iQLxLyDoXSINEpVF/8d5j1QYSq cU7qp5tB9tLDjlvEdqwP04/TIf4lVNRgweY1tu98OiJaJnAyqchLjGCv4bqCEnCvy01F 838A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=asaU2u3+qE2V/tXvNuPV0itByGZXwrnNGauZ3AE0x8Q=; b=j7IFaXETdf2cwn7kxQM9KmimTzNeWAc7LvlM9Vb2VFRIsVN/pVzMPaUrU7Px2ah0f2 3KgHkbXIgyI3BbDPOYD2hzyQixjjj15BTzuAEmABLLDCbJVN9ZYjcmfol9Zn3W0P329E OM0IjqWAAvN2qyv6HZt8YuLgTl+sjTwhA5NxE9X4mTvGFD03gJpMoCGesrnA2hxYOD8D v6f4YTh47YLb2KLf9ufG414nks/4z+46fO8HOqeLgmdNqk2lRHOmmf1OKO0Rdnm56Yhe RkslnWIhlSgRLxVe5YXVCKZ6poHlUWJYrQ8N47Dur09o7cN0F5rtwwRO0I5H8gxo0xu/ N1xw== X-Gm-Message-State: APjAAAUQi4I4wF8bygQwmwed4ldiLJ3vxJQcEl+MCm9jFx4KhHaqHrnC KQ5cL2v3KyRbJlsTgHFoznk= X-Google-Smtp-Source: APXvYqxEisVOKnA5w+z3iVn+487JJc2VnADsPPw/D/sgNAIpFIMI0h6sXlh2IuEY4rPXtLOql2AbEA== X-Received: by 2002:a0c:93a3:: with SMTP id f32mr5198677qvf.14.1561668010735; Thu, 27 Jun 2019 13:40:10 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::5a51]) by smtp.gmail.com with ESMTPSA id h18sm60802qkj.134.2019.06.27.13.40.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jun 2019 13:40:10 -0700 (PDT) From: Tejun Heo To: axboe@kernel.dk Cc: jack@suse.cz, josef@toxicpanda.com, clm@fb.com, dsterba@suse.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-btrfs@vger.kernel.org, kernel-team@fb.com, Tejun Heo Subject: [PATCH 5/5] blkcg: implement REQ_CGROUP_PUNT Date: Thu, 27 Jun 2019 13:39:52 -0700 Message-Id: <20190627203952.386785-6-tj@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190627203952.386785-1-tj@kernel.org> References: <20190627203952.386785-1-tj@kernel.org> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When a shared kthread needs to issue a bio for a cgroup, doing so synchronously can lead to priority inversions as the kthread can be trapped waiting for that cgroup. This patch implements REQ_CGROUP_PUNT flag which makes submit_bio() punt the actual issuing to a dedicated per-blkcg work item to avoid such priority inversions. This will be used to fix priority inversions in btrfs compression and should be generally useful as we grow filesystem support for comprehensive IO control. Signed-off-by: Tejun Heo Reviewed-by: Josef Bacik Reviewed-by: Jan Kara Cc: Chris Mason --- block/blk-cgroup.c | 53 +++++++++++++++++++++++++++++++++++++ block/blk-core.c | 3 +++ include/linux/backing-dev.h | 1 + include/linux/blk-cgroup.h | 16 ++++++++++- include/linux/blk_types.h | 10 +++++++ include/linux/writeback.h | 13 ++++++--- 6 files changed, 92 insertions(+), 4 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 3319ab4ff262..921a3ef329aa 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -54,6 +54,7 @@ static struct blkcg_policy *blkcg_policy[BLKCG_MAX_POLS]; static LIST_HEAD(all_blkcgs); /* protected by blkcg_pol_mutex */ static bool blkcg_debug_stats = false; +static struct workqueue_struct *blkcg_punt_bio_wq; static bool blkcg_policy_enabled(struct request_queue *q, const struct blkcg_policy *pol) @@ -88,6 +89,8 @@ static void __blkg_release(struct rcu_head *rcu) { struct blkcg_gq *blkg = container_of(rcu, struct blkcg_gq, rcu_head); + WARN_ON(!bio_list_empty(&blkg->async_bios)); + /* release the blkcg and parent blkg refs this blkg has been holding */ css_put(&blkg->blkcg->css); if (blkg->parent) @@ -113,6 +116,23 @@ static void blkg_release(struct percpu_ref *ref) call_rcu(&blkg->rcu_head, __blkg_release); } +static void blkg_async_bio_workfn(struct work_struct *work) +{ + struct blkcg_gq *blkg = container_of(work, struct blkcg_gq, + async_bio_work); + struct bio_list bios = BIO_EMPTY_LIST; + struct bio *bio; + + /* as long as there are pending bios, @blkg can't go away */ + spin_lock_bh(&blkg->async_bio_lock); + bio_list_merge(&bios, &blkg->async_bios); + bio_list_init(&blkg->async_bios); + spin_unlock_bh(&blkg->async_bio_lock); + + while ((bio = bio_list_pop(&bios))) + submit_bio(bio); +} + /** * blkg_alloc - allocate a blkg * @blkcg: block cgroup the new blkg is associated with @@ -141,6 +161,9 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct request_queue *q, blkg->q = q; INIT_LIST_HEAD(&blkg->q_node); + spin_lock_init(&blkg->async_bio_lock); + bio_list_init(&blkg->async_bios); + INIT_WORK(&blkg->async_bio_work, blkg_async_bio_workfn); blkg->blkcg = blkcg; for (i = 0; i < BLKCG_MAX_POLS; i++) { @@ -1527,6 +1550,25 @@ void blkcg_policy_unregister(struct blkcg_policy *pol) } EXPORT_SYMBOL_GPL(blkcg_policy_unregister); +bool __blkcg_punt_bio_submit(struct bio *bio) +{ + struct blkcg_gq *blkg = bio->bi_blkg; + + /* consume the flag first */ + bio->bi_opf &= ~REQ_CGROUP_PUNT; + + /* never bounce for the root cgroup */ + if (!blkg->parent) + return false; + + spin_lock_bh(&blkg->async_bio_lock); + bio_list_add(&blkg->async_bios, bio); + spin_unlock_bh(&blkg->async_bio_lock); + + queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work); + return true; +} + /* * Scale the accumulated delay based on how long it has been since we updated * the delay. We only call this when we are adding delay, in case it's been a @@ -1727,5 +1769,16 @@ void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta) atomic64_add(delta, &blkg->delay_nsec); } +static int __init blkcg_init(void) +{ + blkcg_punt_bio_wq = alloc_workqueue("blkcg_punt_bio", + WQ_MEM_RECLAIM | WQ_FREEZABLE | + WQ_UNBOUND | WQ_SYSFS, 0); + if (!blkcg_punt_bio_wq) + return -ENOMEM; + return 0; +} +subsys_initcall(blkcg_init); + module_param(blkcg_debug_stats, bool, 0644); MODULE_PARM_DESC(blkcg_debug_stats, "True if you want debug stats, false if not"); diff --git a/block/blk-core.c b/block/blk-core.c index 5d1fc8e17dd1..812052c835fc 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1127,6 +1127,9 @@ EXPORT_SYMBOL_GPL(direct_make_request); */ blk_qc_t submit_bio(struct bio *bio) { + if (blkcg_punt_bio_submit(bio)) + return BLK_QC_T_NONE; + /* * If it's a regular read/write or a barrier with data attached, * go through the normal accounting stuff before submission. diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h index f9b029180241..35b31d176f74 100644 --- a/include/linux/backing-dev.h +++ b/include/linux/backing-dev.h @@ -48,6 +48,7 @@ extern spinlock_t bdi_lock; extern struct list_head bdi_list; extern struct workqueue_struct *bdi_wq; +extern struct workqueue_struct *bdi_async_bio_wq; static inline bool wb_has_dirty_io(struct bdi_writeback *wb) { diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 33f23a858438..689a58231288 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -132,13 +132,17 @@ struct blkcg_gq { struct blkg_policy_data *pd[BLKCG_MAX_POLS]; - struct rcu_head rcu_head; + spinlock_t async_bio_lock; + struct bio_list async_bios; + struct work_struct async_bio_work; atomic_t use_delay; atomic64_t delay_nsec; atomic64_t delay_start; u64 last_delay; int last_use; + + struct rcu_head rcu_head; }; typedef struct blkcg_policy_data *(blkcg_pol_alloc_cpd_fn)(gfp_t gfp); @@ -701,6 +705,15 @@ static inline bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg struct bio *bio) { return false; } #endif +bool __blkcg_punt_bio_submit(struct bio *bio); + +static inline bool blkcg_punt_bio_submit(struct bio *bio) +{ + if (bio->bi_opf & REQ_CGROUP_PUNT) + return __blkcg_punt_bio_submit(bio); + else + return false; +} static inline void blkcg_bio_issue_init(struct bio *bio) { @@ -848,6 +861,7 @@ static inline char *blkg_path(struct blkcg_gq *blkg) { return NULL; } static inline void blkg_get(struct blkcg_gq *blkg) { } static inline void blkg_put(struct blkcg_gq *blkg) { } +static inline bool blkcg_punt_bio_submit(struct bio *bio) { return false; } static inline void blkcg_bio_issue_init(struct bio *bio) { } static inline bool blkcg_bio_issue_check(struct request_queue *q, struct bio *bio) { return true; } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 6a53799c3fe2..feff3fe4467e 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -311,6 +311,14 @@ enum req_flag_bits { __REQ_RAHEAD, /* read ahead, can fail anytime */ __REQ_BACKGROUND, /* background IO */ __REQ_NOWAIT, /* Don't wait if request will block */ + /* + * When a shared kthread needs to issue a bio for a cgroup, doing + * so synchronously can lead to priority inversions as the kthread + * can be trapped waiting for that cgroup. CGROUP_PUNT flag makes + * submit_bio() punt the actual issuing to a dedicated per-blkcg + * work item to avoid such priority inversions. + */ + __REQ_CGROUP_PUNT, /* command specific flags for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ @@ -337,6 +345,8 @@ enum req_flag_bits { #define REQ_RAHEAD (1ULL << __REQ_RAHEAD) #define REQ_BACKGROUND (1ULL << __REQ_BACKGROUND) #define REQ_NOWAIT (1ULL << __REQ_NOWAIT) +#define REQ_CGROUP_PUNT (1ULL << __REQ_CGROUP_PUNT) + #define REQ_NOUNMAP (1ULL << __REQ_NOUNMAP) #define REQ_HIPRI (1ULL << __REQ_HIPRI) diff --git a/include/linux/writeback.h b/include/linux/writeback.h index e056a22075cf..8945aac31392 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -78,6 +78,8 @@ struct writeback_control { */ unsigned no_cgroup_owner:1; + unsigned punt_to_cgroup:1; /* cgrp punting, see __REQ_CGROUP_PUNT */ + #ifdef CONFIG_CGROUP_WRITEBACK struct bdi_writeback *wb; /* wb this writeback is issued under */ struct inode *inode; /* inode being written out */ @@ -94,12 +96,17 @@ struct writeback_control { static inline int wbc_to_write_flags(struct writeback_control *wbc) { + int flags = 0; + + if (wbc->punt_to_cgroup) + flags = REQ_CGROUP_PUNT; + if (wbc->sync_mode == WB_SYNC_ALL) - return REQ_SYNC; + flags |= REQ_SYNC; else if (wbc->for_kupdate || wbc->for_background) - return REQ_BACKGROUND; + flags |= REQ_BACKGROUND; - return 0; + return flags; } static inline struct cgroup_subsys_state *