From patchwork Fri Aug 23 17:30:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 13775600 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D23ED1917DA for ; Fri, 23 Aug 2024 17:31:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434309; cv=none; b=JQ3p6VW5gWSHaWRHeSMr+3MGBnbG4BB8vV5ykz6lv+JLYCESLiXLpP2PAsBZWip7koDD1GLOgU43zwSozQCLPXxyX2C4g9xTE7TdGUlivk/m1IBTduR+3WcwUtV/biIGPPewkX4wJKr9eqFhuRn183mg03QWpCebyniY5rU4fT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434309; c=relaxed/simple; bh=VNWu2J39++LQzHtTrd4Q6xUHjBc0llUb4IP1EArdOZw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e2bQDwG69oy7TJCphcWz/rw0vKWyjdXPqcHqLk0GiDVAgPznPxAGiP63oCzjdiHNysSNUaeaGbftoLHRXP0ELwUbdSyBwD6pnb3KL7xKDDS8pj2LFpAsGCMSsHVJ6UFOk3xMF3Mha9n+VwVsgprDOqgCygGMYuntBgP+F40/3mU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=b7EgIJHt; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="b7EgIJHt" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7b0c9bbddb4so1583948a12.3 for ; Fri, 23 Aug 2024 10:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724434307; x=1725039107; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7yL0MPvBF8bNsfhufIdh1IITs7ZJ9No4pUlKB9Ss45Q=; b=b7EgIJHtUTkvQqKhxVwAF7xGob00f5q8CfXcd7qGXd8PgMUOj34bO7KPDgzIcX+27F 2O48J7fmJ7tCwTVMxHID7qp6H+A3i4F6CYqIjTdXhA3T3KV8JDtqyq/BpHfA7qUXZjjS yM0hGzoXmLXtgPT4TaENEKnz05Z//SRu/AgxI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724434307; x=1725039107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7yL0MPvBF8bNsfhufIdh1IITs7ZJ9No4pUlKB9Ss45Q=; b=dRuC84gGITx9LTm13d+EGSyvxGyv07gDeIFYLzCFiAg62SoxdvFQmivJLNOyH7Tp9e FwtRIgYMaTYxGbeave64f6pgO3UvdQLUT4CJCID5DBA3I1OBK7N26+LUKsSt26pMrIVp aBoApSnilOggOZrStXErXUg08tWCnYRWb5LwTejMwcfKuCGX31WqZ05UNcrcfvyRx+KF mw+Cc0nFv+pAnGwIXOqkl2SPar0CiQ3+zpIeQRi9I8wqbmQF48YzSoOyNcVGt09IAU6S CGsydOzcByj4IkvGWnoZy7m22b4HwisOq+XFGXOgd/tDihEybx2/3OmQnxQ0hnpLjd1z 8hJA== X-Gm-Message-State: AOJu0Yz0qoYO0Zp70WRmxY9xX0dvqLPLN3SATeOhc0FcgEf5rHvRbQE8 wxx05EzxFaSeEq71jXbK9SQPSheXabC6eltXV6X2ojWipE2AIl1t+T8XbNlKLs7xgCSCABRPCCq FhPzMt8Rg7VsWwQdoNwMYvz6egLBerlLcaj3zMkZw4BEgxev5XHle2K7E5fpYoMB9yg2dSGcuHa ajCKBbjJToIolzNLDLAYbVyDWFrHaJItOW2Sbf/A== X-Google-Smtp-Source: AGHT+IFmojFMICV8FHNtia4R5sPL+9AeNF/vbaHALXfPws8KtlkVXl3M4m9hBDPIXTtBqi5YwD+2Nw== X-Received: by 2002:a05:6a21:a343:b0:1c2:956a:a909 with SMTP id adf61e73a8af0-1cc89dbc287mr3646076637.27.1724434300683; Fri, 23 Aug 2024 10:31:40 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7143430964fsm3279624b3a.150.2024.08.23.10.31.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 10:31:40 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Daniel Borkmann , Breno Leitao , Johannes Berg , Heiner Kallweit , linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next 1/6] net: Add sysfs parameter irq_suspend_timeout Date: Fri, 23 Aug 2024 17:30:52 +0000 Message-Id: <20240823173103.94978-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240823173103.94978-1-jdamato@fastly.com> References: <20240823173103.94978-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Martin Karsten This patch doesn't change any behavior but prepares the code for other changes in the following commits which use irq_suspend_timeout as a timeout for IRQ suspension. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten --- rfc -> v1: - Removed napi.rst documentation from this patch; added to patch 6. include/linux/netdevice.h | 2 ++ net/core/dev.c | 3 ++- net/core/net-sysfs.c | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 0ef3eaa23f4b..31867bb2ff65 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1857,6 +1857,7 @@ enum netdev_reg_state { * @gro_flush_timeout: timeout for GRO layer in NAPI * @napi_defer_hard_irqs: If not zero, provides a counter that would * allow to avoid NIC hard IRQ, on busy queues. + * @irq_suspend_timeout: IRQ suspension timeout * * @rx_handler: handler for received packets * @rx_handler_data: XXX: need comments on this one @@ -2060,6 +2061,7 @@ struct net_device { struct netdev_rx_queue *_rx; unsigned long gro_flush_timeout; int napi_defer_hard_irqs; + unsigned long irq_suspend_timeout; unsigned int gro_max_size; unsigned int gro_ipv4_max_size; rx_handler_func_t __rcu *rx_handler; diff --git a/net/core/dev.c b/net/core/dev.c index e7260889d4cb..3bf325ec25a3 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -11945,6 +11945,7 @@ static void __init net_dev_struct_check(void) CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, _rx); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, gro_flush_timeout); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, napi_defer_hard_irqs); + CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, irq_suspend_timeout); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, gro_max_size); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, gro_ipv4_max_size); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, rx_handler); @@ -11956,7 +11957,7 @@ static void __init net_dev_struct_check(void) #ifdef CONFIG_NET_XGRESS CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, tcx_ingress); #endif - CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_rx, 104); + CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_rx, 112); } /* diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index 0e2084ce7b75..fb6f3327310f 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -440,6 +440,23 @@ static ssize_t napi_defer_hard_irqs_store(struct device *dev, } NETDEVICE_SHOW_RW(napi_defer_hard_irqs, fmt_dec); +static int change_irq_suspend_timeout(struct net_device *dev, unsigned long val) +{ + WRITE_ONCE(dev->irq_suspend_timeout, val); + return 0; +} + +static ssize_t irq_suspend_timeout_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + return netdev_store(dev, attr, buf, len, change_irq_suspend_timeout); +} +NETDEVICE_SHOW_RW(irq_suspend_timeout, fmt_ulong); + static ssize_t ifalias_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { @@ -664,6 +681,7 @@ static struct attribute *net_class_attrs[] __ro_after_init = { &dev_attr_tx_queue_len.attr, &dev_attr_gro_flush_timeout.attr, &dev_attr_napi_defer_hard_irqs.attr, + &dev_attr_irq_suspend_timeout.attr, &dev_attr_phys_port_id.attr, &dev_attr_phys_port_name.attr, &dev_attr_phys_switch_id.attr, From patchwork Fri Aug 23 17:30:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 13775601 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1CE2191F8E for ; Fri, 23 Aug 2024 17:31:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434312; cv=none; b=oWA8Jw5FYmXI8QbWueb3oHV1SNGHhBfVafaGQbjNvF8jQW60rr2bcK09qHzuVRmaP0c51ZPcTzM7jXqtTHjAfHP/1s6MFNIVcUPWBupRRFwW9DfXiWK0+Jo+mv7Xx8Qvv3QOGAGDIknQhbvYBfyVCKp40wrGo1Rdf7kczNKLzJU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434312; c=relaxed/simple; bh=LxeFyxsuYCcPtr/MQv2af2sWojNlmhphYuIJiMvBues=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NJAVBlJevB1i0Tc40ru2lTsUfk+sWDaeBToE+OlMBhE/wkFRTGdMUshgRdJCCJwErs5FgbyM5QZKiNe46GeKASaP9uqiYa3bF2Lt3gVsqP5Yuwqw1FkQaazwsnMCS1g2hB3GmiV8DOo2dDQb94Srvw15icKiNBNzBCjzr2u2etE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=nMfkGg2n; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="nMfkGg2n" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-714262f1bb4so1842606b3a.3 for ; Fri, 23 Aug 2024 10:31:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724434310; x=1725039110; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VENZ7u9r+KFqHJZ3BdWz6q0JyzsBYDvzURd4w22e1CM=; b=nMfkGg2nH8Ecfi9ElHrvrwn+pvkDxnum08m8jDN2+sCJKx0URNWDCJoHn2ArjkGC5J leUXvC7dYpd8tKD9N/YGP7GZLRznRU94kGCzqcq9TCdXl3HHOKrwBE1QdjvRTLF0Zawi 8JA4WZ08FPYWcz11kc0zTUH5dHwKrinagm6b0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724434310; x=1725039110; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VENZ7u9r+KFqHJZ3BdWz6q0JyzsBYDvzURd4w22e1CM=; b=pkmUgqjieF6FCoCuFjIy0t6rmh2603pXbrW0tNHYuVD6Ip/MoSdS3U7idaR7xh0Ian fu9/amI5SjwInEbnzTEqrVj/10wWjEhJFsu8DeuV8Udr9frrmNAKCIFKlZJjIYtVQxiV TFJozFC7DcOLiDx5Sdejc9+oyQF/6PCfpwau3ZJZwlOrINEDxI16MJYjQ+OwtiweflU/ FWUq0CW3BBMqPitbRs94DAfT3nuSauvVqBuWrIakTpSGKHMO1eU4gMkPem6Qs8WjiSlf VKK+tsxTds7F8nQez5D3CfVMP2SXG/GuRzhlmxdWdkqOraAJ4nUQ8Bq6sEasdlQ7i/yT IoTw== X-Gm-Message-State: AOJu0YxEAraHsDCPYekDvpLk8ZRMmSOw3MzG/1StsZIuqXBhzqReDEO+ W4BJb82FoL37BqT79VnEKcTRrV+e6MqWCZEKpkWgmwueEmaxQSyGwSNfGYJABRzQ5bPRt4/MT76 0ZW/ipDxotZcpCpNQ1dl3n5qjXe7VF6xw0qG6jjeCczzRCnIO34/5Oen7kogCGD4wzYPThoah3s 6UmAul1nVYAO+K53v8qMi8i1a0iDXtE7mZQpi4Vg== X-Google-Smtp-Source: AGHT+IF80kFwwD0giG0CoU1G7lJncioYNLPbf5NvDHI5LFjxf5lo8LyJnGR5gbXJE+GOFTJmyqHKtA== X-Received: by 2002:a05:6a00:888:b0:70d:2ba1:2402 with SMTP id d2e1a72fcca58-71445f3cc63mr3053603b3a.29.1724434309626; Fri, 23 Aug 2024 10:31:49 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7143430964fsm3279624b3a.150.2024.08.23.10.31.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 10:31:49 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Daniel Borkmann , linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next 2/6] net: Suspend softirq when prefer_busy_poll is set Date: Fri, 23 Aug 2024 17:30:53 +0000 Message-Id: <20240823173103.94978-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240823173103.94978-1-jdamato@fastly.com> References: <20240823173103.94978-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Martin Karsten When NAPI_F_PREFER_BUSY_POLL is set during busy_poll_stop and the irq_suspend_timeout sysfs is nonzero, this timeout is used to defer softirq scheduling, potentially longer than gro_flush_timeout. This can be used to effectively suspend softirq processing during the time it takes for an application to process data and return to the next busy-poll. The call to napi->poll in busy_poll_stop might lead to an invocation of napi_complete_done, but the prefer-busy flag is still set at that time, so the same logic is used to defer softirq scheduling for irq_suspend_timeout. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten --- net/core/dev.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 3bf325ec25a3..74060ba866d4 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6230,7 +6230,12 @@ bool napi_complete_done(struct napi_struct *n, int work_done) timeout = READ_ONCE(n->dev->gro_flush_timeout); n->defer_hard_irqs_count = READ_ONCE(n->dev->napi_defer_hard_irqs); } - if (n->defer_hard_irqs_count > 0) { + if (napi_prefer_busy_poll(n)) { + timeout = READ_ONCE(n->dev->irq_suspend_timeout); + if (timeout) + ret = false; + } + if (ret && n->defer_hard_irqs_count > 0) { n->defer_hard_irqs_count--; timeout = READ_ONCE(n->dev->gro_flush_timeout); if (timeout) @@ -6366,9 +6371,13 @@ static void busy_poll_stop(struct napi_struct *napi, void *have_poll_lock, bpf_net_ctx = bpf_net_ctx_set(&__bpf_net_ctx); if (flags & NAPI_F_PREFER_BUSY_POLL) { - napi->defer_hard_irqs_count = READ_ONCE(napi->dev->napi_defer_hard_irqs); - timeout = READ_ONCE(napi->dev->gro_flush_timeout); - if (napi->defer_hard_irqs_count && timeout) { + timeout = READ_ONCE(napi->dev->irq_suspend_timeout); + if (!timeout) { + napi->defer_hard_irqs_count = READ_ONCE(napi->dev->napi_defer_hard_irqs); + if (napi->defer_hard_irqs_count) + timeout = READ_ONCE(napi->dev->gro_flush_timeout); + } + if (timeout) { hrtimer_start(&napi->timer, ns_to_ktime(timeout), HRTIMER_MODE_REL_PINNED); skip_schedule = true; } From patchwork Fri Aug 23 17:30:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 13775602 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29894192590 for ; Fri, 23 Aug 2024 17:31:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434315; cv=none; b=lUeMWDpy3MxGGHT0oEm0CkapHpVSAyGoZFNUkYdHX2K4nXDGQSV4ZIRdMbTyBIiy8QhBsTeeJgseRPfV3Ji/N+cguXx5UGhPV+IfsVZs/hlDNS3mvUwPj0Sss8/ESTZxAsOdcPEodVyEeWMZj8hO7qbwkyozGin6W1uIIiPodCM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434315; c=relaxed/simple; bh=uX0QibKij4hrx1iInrriZkBV1gHHT5oaikcEGlUnPwI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kpwhlHKX1HADOxQtuactYTX9c07WucL+gr5K3EqAryfBSd7B24fHHupfuZ3W8QfVWSfvzk7B29POjKnAHaGF/7JwFEpZZLgfGuzCz0HDJX5h+Y4Wl5E7cAfVVgQ5fMylujrbbmiUue5omK5U6hizJCWLNBOqTL15t+u5xqf+9fU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=fmUkKKLa; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="fmUkKKLa" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-7143ae1b560so1227212b3a.1 for ; Fri, 23 Aug 2024 10:31:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724434313; x=1725039113; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jJ3Cgz7lMCJbNt90o7mMTqlTtQDLxW3serLqcE11Oz0=; b=fmUkKKLa76VtyrMJONnBCmzjy//kBey/QIPGFSQtVVxllYqvb/oHS7YjGFFJ7j42/6 sF3daiBLlJ45dwelvD7Z1CZ0DUw8W2MsXzA1/DgcmI6rID1rDlHYwQk4aqt1sELaX1pv HQctLfsGX9Xak7VN0hbLOwa3jL2lqyfhfahpU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724434313; x=1725039113; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jJ3Cgz7lMCJbNt90o7mMTqlTtQDLxW3serLqcE11Oz0=; b=c2LIgVkyBYIgXBR/Gm30RBRQiw4miOBkw+cfHZVtW3ZViKmVVLr+hero/rJi36jlck pNY2jE44N5OdONVJ7WWK33ID9N4qyXrYUKNJ3rmxtIFAT9/10sdxS7es41vqfTxD2/mD eYDmaTf77z0YOg3xci7plU4CYQQcGZssctnMlllk928jyOwPFmyoXsG5nZybg4vx6N0a RsB5Mk1xgU76y0lwZIQVTWJIGxukX6s9sKL3/Te2jV8yZBr6KuhymNJL0uPDvr2LnZ4Y Bp0mfb2glxmn2RpB7jtPhrq54vy5xdXCP7nNXhon9WRw4tKFf6MXfGB6tlmAsii4hU53 nQqQ== X-Gm-Message-State: AOJu0YxPo1LxZqDJKy+2vnkfAD8qScogEC5ZwAMits4aiSs+pV1eBorv APR7QKrNEWKE9l2mp1b5EBn4xyhqYzyvevjcW5GGxj+Q/u36J1EF55dmdgV0nJvkeCQ+C/Aj8/0 XRh+bg9pmxW1GKUAvUorRkI5amWp4BS49stdpMvALhcpLsh/Cz6OPlQLSEkD9Hidd6t9qXnAN6p sWp75MpyU0D50NNLzSy3tAhqp75pMyc7DAyNH07g== X-Google-Smtp-Source: AGHT+IFlAQEc9KoqTiJ+uenwue8nngt9FosFsk1pBy96LJQu6gg7G2mzoKPb7tyMW3FTcjllOcQq/w== X-Received: by 2002:a05:6a00:9443:b0:706:6962:4b65 with SMTP id d2e1a72fcca58-714457d3788mr3219030b3a.14.1724434312772; Fri, 23 Aug 2024 10:31:52 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7143430964fsm3279624b3a.150.2024.08.23.10.31.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 10:31:52 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Daniel Borkmann , linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next 3/6] net: Add control functions for irq suspension Date: Fri, 23 Aug 2024 17:30:54 +0000 Message-Id: <20240823173103.94978-4-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240823173103.94978-1-jdamato@fastly.com> References: <20240823173103.94978-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Martin Karsten The napi_suspend_irqs routine bootstraps irq suspension by elongating the defer timeout to irq_suspend_timeout. The napi_resume_irqs routine effectly cancels irq suspension by forcing the napi to be scheduled immediately. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten --- include/net/busy_poll.h | 3 +++ net/core/dev.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/net/busy_poll.h b/include/net/busy_poll.h index 9b09acac538e..f095b2bdeee1 100644 --- a/include/net/busy_poll.h +++ b/include/net/busy_poll.h @@ -52,6 +52,9 @@ void napi_busy_loop_rcu(unsigned int napi_id, bool (*loop_end)(void *, unsigned long), void *loop_end_arg, bool prefer_busy_poll, u16 budget); +void napi_suspend_irqs(unsigned int napi_id); +void napi_resume_irqs(unsigned int napi_id); + #else /* CONFIG_NET_RX_BUSY_POLL */ static inline unsigned long net_busy_loop_on(void) { diff --git a/net/core/dev.c b/net/core/dev.c index 74060ba866d4..4de0dfc86e21 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6507,6 +6507,39 @@ void napi_busy_loop(unsigned int napi_id, } EXPORT_SYMBOL(napi_busy_loop); +void napi_suspend_irqs(unsigned int napi_id) +{ + struct napi_struct *napi; + + rcu_read_lock(); + napi = napi_by_id(napi_id); + if (napi) { + unsigned long timeout = READ_ONCE(napi->dev->irq_suspend_timeout); + + if (timeout) + hrtimer_start(&napi->timer, ns_to_ktime(timeout), HRTIMER_MODE_REL_PINNED); + } + rcu_read_unlock(); +} +EXPORT_SYMBOL(napi_suspend_irqs); + +void napi_resume_irqs(unsigned int napi_id) +{ + struct napi_struct *napi; + + rcu_read_lock(); + napi = napi_by_id(napi_id); + if (napi) { + if (READ_ONCE(napi->dev->irq_suspend_timeout)) { + local_bh_disable(); + napi_schedule(napi); + local_bh_enable(); + } + } + rcu_read_unlock(); +} +EXPORT_SYMBOL(napi_resume_irqs); + #endif /* CONFIG_NET_RX_BUSY_POLL */ static void napi_hash_add(struct napi_struct *napi) From patchwork Fri Aug 23 17:30:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 13775603 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4D091925AF for ; Fri, 23 Aug 2024 17:31:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434317; cv=none; b=X/+RqSSNR3QHoQGU25rnlFd5ocyAzJqURIkygBN8LD9TnzMQuUu8/sSrldCggWZbbBPOSE8tHXxc+D+Qv7nKtXRfI9T34w8RmSBKnVfT8vcNnxDNppEa3vSOnD3OdWl3m2g2icK/df18CK9G8yMN87nyWgZR7T65a4jLNe0lkQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434317; c=relaxed/simple; bh=uGO5dEJrzAGpkDGXLrC8ztAywM3dRMCy6F4elADygWk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=B6zcR87KA84oVbuN4pKbhsjx3ivAqMI8Vk5Ruy1lffOgLdg9kGqWJkY9n9bshdpZ+ZeYJwVy2+ik5setEEcXVA5i4JV1alznwWUGJ0EXNUINGWluEb0wOopOLVad92/1w0eAiKDZMlCWHTRue7sW3GmcrjwjbBKWuqICeNsdxSU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=twgo/R6y; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="twgo/R6y" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7142e002aceso1956986b3a.2 for ; Fri, 23 Aug 2024 10:31:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724434315; x=1725039115; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=58S4qbuyYKoKQjyHpzs28ekwRvFYt2yreo1eLG7u1fM=; b=twgo/R6yBWs7WuZvoSdye6GCNXPvb0JViHYT4zcpQCEdfcSfBy9yJycfC/01CrerAK 9I+R4bRipMWIkMu3GKP4zVmpOGxR0x71MkkOcYk+R6e2CfJwm6D2OJBlUFih08NCNNbS P4esa7u4fAZ6UfK+wpk6ek8BwEDw7BKiE6XGM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724434315; x=1725039115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=58S4qbuyYKoKQjyHpzs28ekwRvFYt2yreo1eLG7u1fM=; b=BclVrb5hrJKak7Pei2i4jQ+RNUsothKlIGRdvOJBu4bvnkXOHZ+ZWhQVcnn7mfv0ax HPIIgGEgtG3gpn930qr/u3pflNUvqUZG3rM6Nf2i767L71Fka4q3hzaZ9JnJ3TM23HU8 K5yTH7mbV746Binc2SY3ZjBEC5A+7J56p3stG+0f+vFUTo0T1RvO5zLf+1ZX2ybX8NfB RghOpLZLwiQ5LjiEKeJrhc8DA6KUxdI8/KaObp9OleMEsz9DO4HpAIWVJvlkVWA3BGgB C7emw05h7CKoEoJpEkiON/hTxi+31NfIudytuPV6yJU9S5ROqeOkiwppcpeXAZmrydcg vcZQ== X-Gm-Message-State: AOJu0YyPoFWnWHbxIIu3X3OgH0IeYJBsvO1P3uSAJktBjpR9XEhqMAos z9lnzJ9mhLi9iMDj/tIZGY2rBIPqwnFj1Ik00OfJ9bOEtUF5c0z3hWE48OrsX/5AacNLNGLrWLh 4DYQh/kJ6H2AYsB3lY2vD9QLtNKH5vPlZaCsi4Pf+8N89GMnt9d3vMJMp7WQBGsrhxa6ROUCupB TdDtEeW39xdfny/Dv99I+DNcOHjFZmruWs+zByFA== X-Google-Smtp-Source: AGHT+IERjh31UHWhIZDePORqNzeJZelgKL0ItV5Or+5rM0DaNHnsupMW/bNRFrWJoppeQW8QJR6Ong== X-Received: by 2002:a05:6a20:9f4a:b0:1c4:c879:b77f with SMTP id adf61e73a8af0-1cc89d7e4b6mr3579864637.27.1724434314568; Fri, 23 Aug 2024 10:31:54 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7143430964fsm3279624b3a.150.2024.08.23.10.31.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 10:31:54 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , Alexander Viro , Christian Brauner , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next 4/6] eventpoll: Trigger napi_busy_loop, if prefer_busy_poll is set Date: Fri, 23 Aug 2024 17:30:55 +0000 Message-Id: <20240823173103.94978-5-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240823173103.94978-1-jdamato@fastly.com> References: <20240823173103.94978-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Martin Karsten Setting prefer_busy_poll now leads to an effectively nonblocking iteration though napi_busy_loop, even when busy_poll_usecs is 0. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten --- fs/eventpoll.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index f53ca4f7fced..cc47f72005ed 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -420,7 +420,9 @@ static bool busy_loop_ep_timeout(unsigned long start_time, static bool ep_busy_loop_on(struct eventpoll *ep) { - return !!ep->busy_poll_usecs || net_busy_loop_on(); + return !!READ_ONCE(ep->busy_poll_usecs) || + READ_ONCE(ep->prefer_busy_poll) || + net_busy_loop_on(); } static bool ep_busy_loop_end(void *p, unsigned long start_time) From patchwork Fri Aug 23 17:30:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 13775604 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0740192B94 for ; Fri, 23 Aug 2024 17:31:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434319; cv=none; b=l3zFDAAOHMRQpy6bssgzyCTE9CpLM/FTgalEP+4nYqkRSHPhKwXfABv0dFUh6hxRFzGG7PJ3OT1n7HxpPIMe7gzg9JqCUtdvQOYX1nS8ie9iE5/JJdAWSAmzqdv7rnfE5xXi0aJNbfq3Dk0/ywF0ZYjEHum6hni1JF+xt+fj0zg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434319; c=relaxed/simple; bh=x08ZrLW3d91mrWZQmC/v9H7iJRa473+4x/SsWYtS3JE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YrGJzkEOu0ci4G1RG2wz30zeHPXSrWQ8l7JSwzhcJ7I1jiZ82coLtlm/4H2oHG2LAZpE/HDQBHVrofcuwlgAFHqMQWymfZUzmC8V7B4ipIMeQ++QQCQKhR3nnTtNHteIEYfHFZxt2t619GV4DoHS/e+XdIq89px2nUx23jsIVQE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=UDZ0sT8u; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="UDZ0sT8u" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-7142e002aceso1957026b3a.2 for ; Fri, 23 Aug 2024 10:31:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724434316; x=1725039116; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IoqQBySxLV4Sefg/kJRVr1Dr3C923zx/iiQfn4tblTE=; b=UDZ0sT8u5MrIFsQ3wnGp9F9y8pQUiZjszYOkWFxo0DmRD7PgI+tWFNlusmmu2kUoI4 7Y51sqGcrLLr9mdv95q3Xg5gPbYTJJwQh280N/zECe56Ppl2iIuY2WwaGeMFHAFrTixS fL8fumpgh/G5j6qj65gnWQugCyPUtzs5Zxseo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724434316; x=1725039116; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IoqQBySxLV4Sefg/kJRVr1Dr3C923zx/iiQfn4tblTE=; b=toSkWHaCmbsB1DzyhHj1Dcl9BjDJWljOPi+QDg5GR07UqnWnfGYaKLfXMUERklcFnz E+bPZj0DtNvFxnxe25GMxInLX5WydQ9rUyDMuF2LBBc7jXXNkJVPpxO5+UNdBHv5MtPq WRhWdRbQcBEvr5ecUOu2bj8sKUWJUndsWjiWt2knYMlpGlndfjTx/42Ax/4PCF+xKdr/ +YHftIGnjtkPf4jsvQQxL1FAAFA+6BlmIWcVTQHeE15Cqzdi0qMStOO7+mm3KgVjLffB emKH/k0IMJAW1EpZFAUyoYqSkQnE5KbMTSPnJeZhvUzq9AKk8QTdzPOkrmCGcrMf4XjT fw1w== X-Gm-Message-State: AOJu0YwitwDSvQtMDptr9a0tj8Z0on/WNoqlJ9OQ1XzMlw5lRlIAXiXO HsLxhawZFiF00JQCWJgRVJ9UaJFJayj+DLs3CUIJHY+guIiTClk1CpX4kzyPU3g4dRagcUgEAgl P8StWh7/dCgwzHPAXAVY1pubfi1ihyWUbuKKBjfmrXyBq3+3rKWlFHVfjSKynP021fUkW5Ruo2K IXy484b8lKE4waEwosdoIFpPyKf9VlPzYjTZ9igg== X-Google-Smtp-Source: AGHT+IHXsgl2NQYO+yMV9X1gIfvGuclsqseoRzkfOcO3p7twoiEGni5X0dIXcScV5If78qjNaxucOA== X-Received: by 2002:a05:6a00:3e2a:b0:710:5605:a986 with SMTP id d2e1a72fcca58-714454098b9mr3730824b3a.0.1724434316322; Fri, 23 Aug 2024 10:31:56 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7143430964fsm3279624b3a.150.2024.08.23.10.31.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 10:31:56 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , Alexander Viro , Christian Brauner , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next 5/6] eventpoll: Control irq suspension for prefer_busy_poll Date: Fri, 23 Aug 2024 17:30:56 +0000 Message-Id: <20240823173103.94978-6-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240823173103.94978-1-jdamato@fastly.com> References: <20240823173103.94978-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Martin Karsten When events are reported to userland and prefer_busy_poll is set, irqs are temporarily suspended using napi_suspend_irqs. If no events are found and ep_poll would go to sleep, irq suspension is cancelled using napi_resume_irqs. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten --- rfc -> v1: - move irq resume code from ep_free to a helper which either resumes IRQs or does nothing if !defined(CONFIG_NET_RX_BUSY_POLL). fs/eventpoll.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index cc47f72005ed..5dbe717c06b4 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -457,6 +457,8 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) * it back in when we have moved a socket with a valid NAPI * ID onto the ready list. */ + if (prefer_busy_poll) + napi_resume_irqs(napi_id); ep->napi_id = 0; return false; } @@ -540,6 +542,22 @@ static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, } } +static void ep_suspend_napi_irqs(struct eventpoll *ep) +{ + unsigned int napi_id = READ_ONCE(ep->napi_id); + + if (napi_id >= MIN_NAPI_ID && READ_ONCE(ep->prefer_busy_poll)) + napi_suspend_irqs(napi_id); +} + +static void ep_resume_napi_irqs(struct eventpoll *ep) +{ + unsigned int napi_id = READ_ONCE(ep->napi_id); + + if (napi_id >= MIN_NAPI_ID && READ_ONCE(ep->prefer_busy_poll)) + napi_resume_irqs(napi_id); +} + #else static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) @@ -557,6 +575,13 @@ static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, return -EOPNOTSUPP; } +static void ep_suspend_napi_irqs(struct eventpoll *ep) +{ +} + +static void ep_resume_napi_irqs(struct eventpoll *ep) +{ +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -788,6 +813,7 @@ static bool ep_refcount_dec_and_test(struct eventpoll *ep) static void ep_free(struct eventpoll *ep) { + ep_resume_napi_irqs(ep); mutex_destroy(&ep->mtx); free_uid(ep->user); wakeup_source_unregister(ep->ws); @@ -2005,8 +2031,10 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * trying again in search of more luck. */ res = ep_send_events(ep, events, maxevents); - if (res) + if (res) { + ep_suspend_napi_irqs(ep); return res; + } } if (timed_out) From patchwork Fri Aug 23 17:30:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 13775596 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C17D190489 for ; Fri, 23 Aug 2024 17:31:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434321; cv=none; b=pz/D2QQgHrYI9IICh+64QCNIMFACvEC4QeEtYeb0Bh7ovfdApsG0Yoy25nsWdxFNQF3COsUyhpdYzUCH0B9sp0C7qJpzsTCD3AA/cooUNFBIV8rOrg2IXqqf0AxZApISqZkO2NFeoXvaTJ4DirQZjtYkfFVEm1JUCC8NjLccSP8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724434321; c=relaxed/simple; bh=A5hbTpFf9NReafBa0UeWIum+icaAKqweWQ059Dzzjcw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JqcSXJZul9fXy3IYntH4uZCYcs43fm2R+Ot4hvT4ccIQg6sxS+sSk2oe2OypYDpILsYfSvjrbqIFtz0rm9rknu1UywuUtWQ5H4Oxt2QyqO2uah6k0EP8AhNomfB59ScG2zs3cibdtmzn4/5hEip+iQ1r6CcHjg4akphUaD79YxE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=QGL9lB6o; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="QGL9lB6o" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-71433cba1b7so1811141b3a.0 for ; Fri, 23 Aug 2024 10:31:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724434318; x=1725039118; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dRJG7gtnGEjX3zB/r9nH89EwAMVuBrmJ1/QaNGiIfQ4=; b=QGL9lB6ogoNYeGHIWooh8MVe/ffaB0F0a9lCxfSy4N18rwBI9mHQC1wPx5MNuIb02V XmbwF8I2dirzc4relE65P4bIEZrmwP9QKJbeGc6829Ub1+xCDFsNfZ29Tusi8JHuaXop w79qa/CawICQYQZHlQcetEnbf3aLyIZM/RYeY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724434318; x=1725039118; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dRJG7gtnGEjX3zB/r9nH89EwAMVuBrmJ1/QaNGiIfQ4=; b=UuJbFU2gXf4EEXSXFHvP7+KQ4clM9aQH2G6CFXyFERNyzKqlYQxj+U3n5BgUAxf/Md R5YiPY6sDI/j0ppD+g4g3sM2NHQEPoChY4u1Yt4d7Q4UP/RYDl3EMgn+VzWMz6NgKjH5 whUlyHt6u46mibBPSIn8hAg7hdlr0bWnZS76zaMvq+3Cu2uSB7dZSqorz5bTf8jPsTBJ sj+p+tRuhUvrIYGGGllvwhTPFckTCQxlmG1qhg1qbXbRlVbGToxx+WH6spMAW1mu3iVk Ldsc8OyilWKlZYVmapz7qsU6CPrIeAeHHeLIy/mkw52CEVgQoo/7vCu3OkzPSp1Jwj7k ucyQ== X-Forwarded-Encrypted: i=1; AJvYcCWvCUz/WTs9+zLECrVe9+3gzoh6k1uR1i5FG1DUh2J+hB65i3zuT1auiDPHC2G1nwOyy0Y=@vger.kernel.org X-Gm-Message-State: AOJu0YznAkY4uW/usuKDtpFVD5fwHrcek0u0Zx/Fm88+/MIMgQMR742o zNGTzHw4Ga+jMUG+0ITr4s1O7eftnL0ftN7+W0H8ROfg6J70JyBh99dkK3JMlVg= X-Google-Smtp-Source: AGHT+IE/GsHsdnuK6rqXk7/a6x4NQp6xoPAAhkZXujnf6d3K6GfaKiO9uswOkRtrYN+I8cZ8TbZK+g== X-Received: by 2002:a05:6a21:2d86:b0:1c4:dfa7:d3ce with SMTP id adf61e73a8af0-1cc89d4cd1fmr3514127637.17.1724434318226; Fri, 23 Aug 2024 10:31:58 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7143430964fsm3279624b3a.150.2024.08.23.10.31.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 10:31:57 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Joe Damato , Martin Karsten , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jonathan Corbet , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [MISC]:Keyword:(?:\b|_)bpf(?:\b|_)) Subject: [PATCH net-next 6/6] docs: networking: Describe irq suspension Date: Fri, 23 Aug 2024 17:30:57 +0000 Message-Id: <20240823173103.94978-7-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240823173103.94978-1-jdamato@fastly.com> References: <20240823173103.94978-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Describe irq suspension, the epoll ioctls, and the tradeoffs of using different gro_flush_timeout values. Signed-off-by: Joe Damato Co-developed-by: Martin Karsten Signed-off-by: Martin Karsten Tested-by: Joe Damato Tested-by: Martin Karsten --- Documentation/networking/napi.rst | 112 +++++++++++++++++++++++++++++- 1 file changed, 110 insertions(+), 2 deletions(-) diff --git a/Documentation/networking/napi.rst b/Documentation/networking/napi.rst index 7bf7b95c4f7a..04e838835b50 100644 --- a/Documentation/networking/napi.rst +++ b/Documentation/networking/napi.rst @@ -192,6 +192,9 @@ The ``gro_flush_timeout`` sysfs configuration of the netdevice is reused to control the delay of the timer, while ``napi_defer_hard_irqs`` controls the number of consecutive empty polls before NAPI gives up and goes back to using hardware IRQs. +``irq_suspend_timeout`` is used to determine how long an application can +completely suspend IRQs. It is used in combination with SO_PREFER_BUSY_POLL, +which can be set on a per-epoll context basis with ``EPIOCSPARAMS`` ioctl. .. _poll: @@ -208,6 +211,46 @@ selected sockets or using the global ``net.core.busy_poll`` and ``net.core.busy_read`` sysctls. An io_uring API for NAPI busy polling also exists. +epoll-based busy polling +------------------------ + +It is possible to trigger packet processing directly from calls to +``epoll_wait``. In order to use this feature, a user application must ensure +all file descriptors which are added to an epoll context have the same NAPI ID. + +If the application uses a dedicated acceptor thread, the application can obtain +the NAPI ID of the incoming connection using SO_INCOMING_NAPI_ID and then +distribute that file descriptor to a worker thread. The worker thread would add +the file descriptor to its epoll context. This would ensure each worker thread +has an epoll context with FDs that have the same NAPI ID. + +Alternatively, if the application uses SO_REUSEPORT, a bpf or ebpf program be +inserted to distribute incoming connections to threads such that each thread is +only given incoming connections with the same NAPI ID. Care must be taken to +carefully handle cases where a system may have multiple NICs. + +In order to enable busy polling, there are two choices: + +1. ``/proc/sys/net/core/busy_poll`` can be set with a time in useconds to busy + loop waiting for events. This is a system-wide setting and will cause all + epoll-based applications to busy poll when they call epoll_wait. This may + not be desireable as many applications may not have the need to busy poll. + +2. Applications using recent kernels can issue an ioctl on the epoll context + file descriptor to set (``EPIOCSPARAMS``) or get (``EPIOCGPARAMS``) ``struct + epoll_params``:, which user programs can define as follows: + +.. code-block:: c + + struct epoll_params { + uint32_t busy_poll_usecs; + uint16_t busy_poll_budget; + uint8_t prefer_busy_poll; + + /* pad the struct to a multiple of 64bits */ + uint8_t __pad; + }; + IRQ mitigation --------------- @@ -223,12 +266,77 @@ Such applications can pledge to the kernel that they will perform a busy polling operation periodically, and the driver should keep the device IRQs permanently masked. This mode is enabled by using the ``SO_PREFER_BUSY_POLL`` socket option. To avoid system misbehavior the pledge is revoked -if ``gro_flush_timeout`` passes without any busy poll call. +if ``gro_flush_timeout`` passes without any busy poll call. For epoll-based +busy polling applications, the ``prefer_busy_poll`` field of ``struct +epoll_params`` can be set to 1 and the ``EPIOCSPARAMS`` ioctl can be issued to +enable this mode. See the above section for more details. The NAPI budget for busy polling is lower than the default (which makes sense given the low latency intention of normal busy polling). This is not the case with IRQ mitigation, however, so the budget can be adjusted -with the ``SO_BUSY_POLL_BUDGET`` socket option. +with the ``SO_BUSY_POLL_BUDGET`` socket option. For epoll-based busy polling +applications, the ``busy_poll_budget`` field can be adjusted to the desired value +in ``struct epoll_params`` and set on a specific epoll context using the ``EPIOCSPARAMS`` +ioctl. See the above section for more details. + +It is important to note that choosing a large value for ``gro_flush_timeout`` +will defer IRQs to allow for better batch processing, but will induce latency +when the system is not fully loaded. Choosing a small value for +``gro_flush_timeout`` can cause interference of the user application which is +attempting to busy poll by device IRQs and softirq processing. This value +should be chosen carefully with these tradeoffs in mind. epoll-based busy +polling applications may be able to mitigate how much user processing happens +by choosing an appropriate value for ``maxevents``. + +Users may want to consider an alternate approach, IRQ suspension, to help deal +with these tradeoffs. + +IRQ suspension +-------------- + +IRQ suspension is a mechanism wherein device IRQs are masked while epoll +triggers NAPI packet processing. + +While application calls to epoll_wait successfully retrieve events, the kernel will +defer the IRQ suspension timer. If the kernel does not retrieve any events +while busy polling (for example, because network traffic levels subsided), IRQ +suspension is disabled and the IRQ mitigation strategies described above are +engaged. + +This allows users to balance CPU consumption with network processing +efficiency. + +To use this mechanism: + + 1. The sysfs parameter ``irq_suspend_timeout`` should be set to the maximum + time (in nanoseconds) the application can have its IRQs suspended. This + timeout serves as a safety mechanism to restart IRQ driver interrupt + processing if the application has stalled. This value should be chosen so + that it covers the amount of time the user application needs to process + data from its call to epoll_wait, noting that applications can control how + much data they retrieve by setting ``max_events`` when calling epoll_wait. + + 2. The sysfs parameter ``gro_flush_timeout`` and ``napi_defer_hard_irqs`` can + be set to low values. They will be used to defer IRQs after busy poll has + found no data. + + 3. The ``prefer_busy_poll`` flag must be set to true. This can be done using + the ``EPIOCSPARAMS`` ioctl as described above. + + 4. The application uses epoll as described above to trigger NAPI packet + processing. + +As mentioned above, as long as subsequent calls to epoll_wait return events to +userland, the ``irq_suspend_timeout`` is deferred and IRQs are disabled. This +allows the application to process data without interference. + +Once a call to epoll_wait results in no events being found, IRQ suspension is +automatically disabled and the ``gro_flush_timeout`` and +``napi_defer_hard_irqs`` mitigation mechanisms take over. + +It is expected that ``irq_suspend_timeout`` will be set to a value much larger +than ``gro_flush_timeout`` as ``irq_suspend_timeout`` should suspend IRQs for +the duration of one userland processing cycle. .. _threaded: