From patchwork Thu Jan 3 15:01:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Penyaev X-Patchwork-Id: 10747469 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C7746C5 for ; Thu, 3 Jan 2019 15:01:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C248283A5 for ; Thu, 3 Jan 2019 15:01:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1082828415; Thu, 3 Jan 2019 15:01:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D7FE283E2 for ; Thu, 3 Jan 2019 15:01:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732096AbfACPBO (ORCPT ); Thu, 3 Jan 2019 10:01:14 -0500 Received: from mx2.suse.de ([195.135.220.15]:33102 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727066AbfACPBO (ORCPT ); Thu, 3 Jan 2019 10:01:14 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 563A6AC8D; Thu, 3 Jan 2019 15:01:12 +0000 (UTC) From: Roman Penyaev Cc: Roman Penyaev , Davidlohr Bueso , Jason Baron , Al Viro , "Paul E. McKenney" , Linus Torvalds , Andrew Morton , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 0/4] use rwlock in order to reduce ep_poll_callback() contention Date: Thu, 3 Jan 2019 16:01:00 +0100 Message-Id: <20190103150104.17128-1-rpenyaev@suse.de> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 To: unlisted-recipients:; (no To-header on input) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The last patch targets the contention problem in ep_poll_callback(), which can be very well reproduced by generating events (write to pipe or eventfd) from many threads, while consumer thread does polling. The following are some microbenchmark results based on the test [1] which starts threads which generate N events each. The test ends when all events are successfully fetched by the poller thread: spinlock ======== threads events/ms run-time ms 8 6402 12495 16 7045 22709 32 7395 43268 rwlock + xchg ============= threads events/ms run-time ms 8 10038 7969 16 12178 13138 32 13223 24199 According to the results bandwidth of delivered events is significantly increased, thus execution time is reduced. This series is based on linux-next/akpm. v2: o I was wrong saying that ep_poll_callback() can't be called concurrently for the same epi: several wait queues can be attached to the single epoll item, thus several event sources can signal in parallel. To cover this case lockless element addition has to detect that the same @epi is not yet in the list. This is done by extra cmpxchg() operation. o unify awaking of wakeup source calling ep_pm_stay_awake_rcu(epi) in all the cases from ep_poll_callback() path. o more explicit comments [1] https://github.com/rouming/test-tools/blob/master/stress-epoll.c Roman Penyaev (4): epoll: make sure all elements in ready list are in FIFO order epoll: loosen irq safety in ep_poll_callback() epoll: unify awaking of wakeup source on ep_poll_callback() path epoll: use rwlock in order to reduce ep_poll_callback() contention fs/eventpoll.c | 178 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 133 insertions(+), 45 deletions(-) Signed-off-by: Roman Penyaev Cc: Davidlohr Bueso Cc: Jason Baron Cc: Al Viro Cc: "Paul E. McKenney" Cc: Linus Torvalds Cc: Andrew Morton Cc: linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org