From patchwork Thu Aug 4 01:42:20 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wanpeng Li X-Patchwork-Id: 9262511 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 749456048F for ; Thu, 4 Aug 2016 01:42:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 628AC27FA1 for ; Thu, 4 Aug 2016 01:42:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 50F642810E; Thu, 4 Aug 2016 01:42:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,FREEMAIL_FROM,RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB6A227FA1 for ; Thu, 4 Aug 2016 01:42:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754800AbcHDBm3 (ORCPT ); Wed, 3 Aug 2016 21:42:29 -0400 Received: from mail-pf0-f195.google.com ([209.85.192.195]:36050 "EHLO mail-pf0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751193AbcHDBm2 (ORCPT ); Wed, 3 Aug 2016 21:42:28 -0400 Received: by mail-pf0-f195.google.com with SMTP id y134so15822287pfg.3; Wed, 03 Aug 2016 18:42:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id; bh=VviIpRNQ3Tc2AkfEPP7xvh8FnjSiNcqbU08bmM6NYBY=; b=bRIuNUexutyWOkwugS9ur+3JUAwwRNmHAziaJaUtMgM6yBuhE9cT3pg7R7QXsjgvNd RK2+McQUFOgJqDguBi8cbHppTMkhSEaPRCPudaVj/1m2SgpY9l6b462YznnjXBdZjuOc W5Se2KNBM10jKrnlSqdWdMMYY/H0MrbGJkFI7HlMmUahKinkmn9BgewNWa+Zeuo4E4FT IFye6Tth/GqqjtgqDCNOHDxlVYOA4kvXzuHb4JWVC1+MSIKKzxs9j1AudbpWI7bT2YkP cJpg1/QlSr3kG1iT76rRxybzvyE9Mjqt7e933qeLl5V9+S1IE7yPPPr0HT83Oash4znT xB5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=VviIpRNQ3Tc2AkfEPP7xvh8FnjSiNcqbU08bmM6NYBY=; b=lZWz1Xi5WJDG64cMsFufnh8I1nVGSeu7kzdFuoFTJjxku8LlFPcfH5XW1r1vi5nDmn vmkhSaoqgw2884wpeEg9Kt4LHsbWxsd5Cgaf+PCmn70IhTCdKkNt4/uXKd3VoEDeRcTM beLhxk63lEJd+nJIS1s4Fnw0urMRyAsJni3vEKhaHI9//n4NmsDlF1QFBGjmvFIjMCgO rJyPVh3wOdoOm2VXXgzJbmX9p/WPusFQ1ycz+HDqwI4h/qN+s8LGJDd4l3if4tZtx4NE UrnA3eGB0SVQCzRu4X7WL8ThloEA7oV7abx6Jb1eDyS4m2US9U2Rx/cQZtwtTD4nko0p YD2A== X-Gm-Message-State: AEkoouv+HNEemciJFohlbsywY4y8ect0EIHdVWRc2oXSdkiph8gODEa+le23noiSTQZxVg== X-Received: by 10.98.17.152 with SMTP id 24mr120966772pfr.13.1470274947489; Wed, 03 Aug 2016 18:42:27 -0700 (PDT) Received: from kernel.kingsoft.cn ([114.255.44.132]) by smtp.gmail.com with ESMTPSA id f3sm15420994pff.26.2016.08.03.18.42.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Aug 2016 18:42:26 -0700 (PDT) From: Wanpeng Li X-Google-Original-From: Wanpeng Li To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Wanpeng Li , Ingo Molnar , Peter Zijlstra , Juri Lelli , Luca Abeni Subject: [PATCH] sched/deadline: Fix lock pinning warning during cpu hotplug Date: Thu, 4 Aug 2016 09:42:20 +0800 Message-Id: <1470274940-17976-1-git-send-email-wanpeng.li@hotmail.com> X-Mailer: git-send-email 1.9.1 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Wanpeng Li WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:3531 lock_release+0x690/0x6a0 releasing a pinned lock Call Trace: dump_stack+0x99/0xd0 __warn+0xd1/0xf0 ? dl_task_timer+0x1a1/0x2b0 warn_slowpath_fmt+0x4f/0x60 ? sched_clock+0x13/0x20 lock_release+0x690/0x6a0 ? enqueue_pushable_dl_task+0x9b/0xa0 ? enqueue_task_dl+0x1ca/0x480 _raw_spin_unlock+0x1f/0x40 dl_task_timer+0x1a1/0x2b0 ? push_dl_task.part.31+0x190/0x190 WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:3649 lock_unpin_lock+0x181/0x1a0 unpinning an unpinned lock Call Trace: dump_stack+0x99/0xd0 __warn+0xd1/0xf0 warn_slowpath_fmt+0x4f/0x60 lock_unpin_lock+0x181/0x1a0 dl_task_timer+0x127/0x2b0 ? push_dl_task.part.31+0x190/0x190 This can be triggered by hot-unplug the cpu which dl task is running on. DL task will be migrated to a most suitable later deadline rq or fallback to any eligible online CPU after dl timer fires if current rq is offline. We need to hold rq lock of both rqs to confirm there is nothing changed and compare the task deadline and the earliest_dl deadline of the destination cpu in the progress of finding suitable destination cpu. The rq lock of offline cpu will be unlocked in _double_lock_balance and then reacquire both rq locks in a fair way, the rq lock of offline cpu is held and lockdep pin before, however, it will be unlocked in _double_lock_balance directly w/o lockdep unpin, which triggers the "releasing a pinned lock" warning above. Queue the task to the destination cpu might have overloaded rq, so the current implementation will check if we need to kick some away, rq lock will be lockdep unpin since push_dl_task() will release the rq lock and reacquire it. However, if there is cpu hot-unplug and dl task migration, the rq lock is lockdep unpin, so it will trigger the "unpinning an unpinned lock" above. This patch fix it by lockdep unpin the offline cpu before task migration during cpu hotplug handling and lockdep pin destination cpu after this handling since the rq lock is held during this handling. Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Luca Abeni Signed-off-by: Wanpeng Li --- kernel/sched/deadline.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index fcb7f02..1ce8867 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -658,8 +658,11 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer) * * XXX figure out if select_task_rq_dl() deals with offline cpus. */ - if (unlikely(!rq->online)) + if (unlikely(!rq->online)) { + lockdep_unpin_lock(&rq->lock, rf.cookie); rq = dl_task_offline_migration(rq, p); + rf.cookie = lockdep_pin_lock(&rq->lock); + } /* * Queueing this task back might have overloaded rq, check if we need