From patchwork Wed Mar 13 16:25:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhai X-Patchwork-Id: 13591569 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C7A54AEEA for ; Wed, 13 Mar 2024 16:25:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710347155; cv=none; b=DNJawFXBT/uVli6QY/lYQQD2n9TY8QF3vKVNm4fwykO61n8yRlZrwtKTPL5kKtd4aecckJUMwETh9WLHGP6C3K7m8YGvCHgBqbyqfpItxaolvhISGBaYH8hkkzpbdZqnuSa3f3QzFMZPfxpMCSQyd1Yq9wJtZ6i/To7P2YO1tlE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710347155; c=relaxed/simple; bh=waophCxTcodcZBiaZRBq4Vcy/9u+Db/aHwd0TOyESlA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hjNGCYBUrU7qpuaboIxw366T7rXYAQzsx+JCFnGi99sUyRyQlda7S4tY/XIKJRRqaKgVx9uDjAy6Z/htkfoR2Q2b/pUgjr7v29b+64R6vJ8yDHQOMWr8WkTQk5M7VLlcM06uKXSeLteguG7uGkjQHkNe4C9XUYL+ubuPl7DTYyw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com; spf=pass smtp.mailfrom=cloudflare.com; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.b=RdSU3EY0; arc=none smtp.client-ip=209.85.222.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.b="RdSU3EY0" Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-788412a4b2eso250617585a.1 for ; Wed, 13 Mar 2024 09:25:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google09082023; t=1710347152; x=1710951952; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=hz52VDhsyBUm+rkXRTpJXcM4FSy0vt7HjYW4U4Q0pQw=; b=RdSU3EY000ormZCobNVsi8X8ZcmHO2hUrQybNRFpiTKEhAbnstxVmtInPHXSt2aKxb PS3A5Z9vFgtBWo3OPKELMRGKXRFnYFn8oEm0OKdRZqBg8XqlpdTDHmYxlpIMgtpWIHro jzi7Vw1k9/zp0D+bp8lbe8/J0BOahc1XAJakEumFQ0uR9j3An0CbhrWPdBJMZc6eUzy1 zYo0kk+DpgXL9eMpD62O/sjKMw4puhi8quHZVoOpGurS7WHZkzM4GR2S6jxq55FJZFoc jAX2H1BZSlt7eZcJQ/IwKdhWWSjdo/fakbvzyJD7sBBG28olu8Ns0gmSrQ9khAJMSaXx P/Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710347152; x=1710951952; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=hz52VDhsyBUm+rkXRTpJXcM4FSy0vt7HjYW4U4Q0pQw=; b=qGCc+RdD49X9YowJ1yyIv6BxuYLDDQpaLuAIHmVKfNO8eYBtl0b1t4YASIgkcdHPyQ LPlFiL6GmM18zFp7A2Iap/x/xw7A4cehNEr7jzggfZH8Dj0RkkEqxpNU9Vs5uEdiILC5 SKKcDSHhfp8qwuX3K90Y5UpMgFZ24/EBt/SIXsNUZpEWL57AdaPG3MypL+hsAT5u00Xj WEcKCOg7jJ9KlGUdyFIq5qqObVcUc+qmsQjnZM79w7Q1+WH5kSQ//fjopdHS2c5MnmMT ek0G/XGZsgBjSXxVoOG7D7/v2nqX7FWp3wrVcCX5bsyHnpkysHa73Ap30RYSjhTtnAdm pLew== X-Forwarded-Encrypted: i=1; AJvYcCVwhonfx33a+VNj3i2lE+bG9s+uZoeB4W3PBFhKeBCyGo+tWU/nG87Ve7SW+8LN0zOaLicpYhJkfj0zuc/lKydQTD/K X-Gm-Message-State: AOJu0YwOSudCpveTelcWJE2ByFZQDOQMHh1xhqm24tEl3cjnIUvmfMJM qafGDZA+MktZTJTYVpdcyPFpme+uZcdIx3VwKSdGfraYdEu8e7zvi8UggwfppEQ= X-Google-Smtp-Source: AGHT+IHG2Wo7WJzKa6z0HU+as2odTaqY/F5UrV7eoi67pzTk35M3ZippREVklQyBwoPl9T2v05w+XQ== X-Received: by 2002:a05:620a:28d1:b0:788:2dd5:6f12 with SMTP id l17-20020a05620a28d100b007882dd56f12mr467010qkp.67.1710347152457; Wed, 13 Mar 2024 09:25:52 -0700 (PDT) Received: from debian.debian ([2a09:bac5:7a49:f91::18d:37]) by smtp.gmail.com with ESMTPSA id az39-20020a05620a172700b00789ca2c923esm818990qkb.114.2024.03.13.09.25.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Mar 2024 09:25:51 -0700 (PDT) Date: Wed, 13 Mar 2024 09:25:49 -0700 From: Yan Zhai To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jiri Pirko , Simon Horman , Daniel Borkmann , Lorenzo Bianconi , Coco Li , Wei Wang , Alexander Duyck , Hannes Frederic Sowa , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, bpf@vger.kernel.org, kernel-team@cloudflare.com, Joel Fernandes , "Paul E. McKenney" , Toke =?utf-8?q?H=C3=B8iland-J?= =?utf-8?q?=C3=B8rgensen?= , Alexei Starovoitov , Steven Rostedt , mark.rutland@arm.com, Jesper Dangaard Brouer Subject: [PATCH v3 net 1/3] rcu: add a helper to report consolidated flavor QS Message-ID: References: Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: There are several scenario in network processing that can run extensively under heavy traffic. In such situation, RCU synchronization might not observe desired quiescent states for indefinitely long period. Create a helper to safely raise the desired RCU quiescent states for such scenario. Reviewed-by: Jesper Dangaard Brouer Signed-off-by: Yan Zhai --- include/linux/rcupdate.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 0746b1b0b663..e91ae38c33e3 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -247,6 +247,29 @@ do { \ cond_resched(); \ } while (0) +/** + * rcu_softirq_qs_periodic - Periodically report consolidated quiescent states + * + * This helper is for network processing in non-RT kernels, where there could + * be busy polling threads that block RCU synchronization indefinitely. In + * such context, simply calling cond_resched is insufficient, so give it a + * stronger push to eliminate potential blockage of all RCU types. + * + * NOTE: unless absolutely sure, this helper should in general be called + * outside of bh lock section to avoid reporting a surprising QS to updaters, + * who could be expecting RCU read critical section to end at local_bh_enable(). + */ +#define rcu_softirq_qs_periodic(old_ts) \ +do { \ + if (!IS_ENABLED(CONFIG_PREEMPT_RT) && \ + time_after(jiffies, (old_ts) + HZ / 10)) { \ + preempt_disable(); \ + rcu_softirq_qs(); \ + preempt_enable(); \ + (old_ts) = jiffies; \ + } \ +} while (0) + /* * Infrastructure to implement the synchronize_() primitives in * TREE_RCU and rcu_barrier_() primitives in TINY_RCU. From patchwork Wed Mar 13 16:25:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhai X-Patchwork-Id: 13591570 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADA834DA13 for ; Wed, 13 Mar 2024 16:25:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710347158; cv=none; b=Jyw3s+Psw1r6grntfzE5fiDcjGfwSBC+fREVYiygXZ1tye69jTNpHhNl28HYHVoruvW7/lZOiXNTYvsjodeWu66HkWPR3DES9k2BWXiB8+bWNkguOf9xK7cotCkQnj/jrV6gwT6fYe7xHupE2/zNuvvrXmQGIHP22HD2/1QghWg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710347158; c=relaxed/simple; bh=1+v+hX17pdDJJ6b6jYIIjDKh50fBpD8puF0YHUfUemQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=QMn8nPX8IwNqkQ2OwaCmXijaKn0ISo1NvGUwtLUBE4JxuRIqPD8syvq25hZzEXADBOp8EG+7Xu+rZIKfjkq7sDM0pVP/EiGaZpzu1Qrs7hChJ4R8NyNjoYyhFgdr/wz7oacpivne+voFRoPbIrtS2DZcZ5M9j1ADdSl06Ge83m8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com; spf=pass smtp.mailfrom=cloudflare.com; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.b=OgTGHzNv; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.b="OgTGHzNv" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-42f2ee83fccso26770111cf.2 for ; Wed, 13 Mar 2024 09:25:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google09082023; t=1710347155; x=1710951955; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=zuZkoM/pNXkg3fwRgcPCCwXFIp95GZp6NoxUGsneVz4=; b=OgTGHzNv8DL7sPU+pYpZcazNi/0ETXxuLOa5V1z097C80bO+gxIi4ImuaAne7AgEpQ nB4wzP19UO0aWqwcHTrcpMWHYDVlQGzXbkKIIizx5Jq+Gb12ZZXplh+Z2efe0DDkFF8S 9y5YSC5TB1e1PjQoTbMJtbTv7x3nzriddVec0+NWiojZ2Ivdlm4L43V19Rn3C90NHJpw P9D4utQNZM4ZSpJB27cafYC4MingLA1wHKTcVQa3hL/b+D0GorfVwRXMj5ajVo7KL0rb I2pxn7VQIZWUlia3hln/UUZDTHacIy+mi891u4BcLdfMqcfgI0kB+6We8lJjkCxXF474 cOGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710347155; x=1710951955; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=zuZkoM/pNXkg3fwRgcPCCwXFIp95GZp6NoxUGsneVz4=; b=Q1iHmCx43bn6npLxhvIMHVRqspu+Gdid4OSEJabQgIFaJ9/FtcLU4ZFyFQkBtY6v1j qp/0hNLGHag8txavA8bcDJwT1xDyMc4lyB8hSD5FGnmSYMJKc+h8gyL0D99ZGwc+GPwA 864EJIH0xJQGPLI9kzEgvyxztHRDsErCJjKrKy2tDP5Ie5I4cIsPsuYblWqSPWjp87K+ FIlVD4tbeZrDHYKh/cQj2JDgB/Na8J9zzcEDHis41VCrbIIY0hzOh32CCmK1Ud0ixBTi x3Up1Y4VFej6/dqNDU6/jeKm9mZ/4O6jzVX159D8humdv4USwMybJraBPcyzYfX3m5qo xSEA== X-Forwarded-Encrypted: i=1; AJvYcCXV6yW00FjGu7bf9A21aFJov4mFXTnFLpBDYQGpwYBisSlog5ilp6TYzwoOnccN/6psl36jJeJodgf8o9jMoDVIwydZ X-Gm-Message-State: AOJu0Yy16w9/tMUk1qgQ8ifb8/bpvflE0xfbEaB4peaFPf+HMThaNs9z HiX80RagUgLKgwXuPGGyp46xx1ho6clAXLV/hkqdR3kn6ec56ifFIMIZCyzbO58= X-Google-Smtp-Source: AGHT+IE4RSELlhIjyVIl9mdoB2c6/y2ghhFHfvK2xWIoar8kwAstJE8efv21r1w2Khjmt3u/eHzNNw== X-Received: by 2002:ac8:7d55:0:b0:42e:7a9a:f13b with SMTP id h21-20020ac87d55000000b0042e7a9af13bmr17444916qtb.58.1710347155602; Wed, 13 Mar 2024 09:25:55 -0700 (PDT) Received: from debian.debian ([2a09:bac5:7a49:f91::18d:37]) by smtp.gmail.com with ESMTPSA id k10-20020ac8474a000000b0042f1c348853sm4948440qtp.21.2024.03.13.09.25.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Mar 2024 09:25:55 -0700 (PDT) Date: Wed, 13 Mar 2024 09:25:52 -0700 From: Yan Zhai To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jiri Pirko , Simon Horman , Daniel Borkmann , Lorenzo Bianconi , Coco Li , Wei Wang , Alexander Duyck , Hannes Frederic Sowa , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, bpf@vger.kernel.org, kernel-team@cloudflare.com, Joel Fernandes , "Paul E. McKenney" , Toke =?utf-8?q?H=C3=B8iland-J?= =?utf-8?q?=C3=B8rgensen?= , Alexei Starovoitov , Steven Rostedt , mark.rutland@arm.com, Jesper Dangaard Brouer Subject: [PATCH v3 net 2/3] net: report RCU QS on threaded NAPI repolling Message-ID: References: Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: NAPI threads can keep polling packets under load. Currently it is only calling cond_resched() before repolling, but it is not sufficient to clear out the holdout of RCU tasks, which prevent BPF tracing programs from detaching for long period. This can be reproduced easily with following set up: ip netns add test1 ip netns add test2 ip -n test1 link add veth1 type veth peer name veth2 netns test2 ip -n test1 link set veth1 up ip -n test1 link set lo up ip -n test2 link set veth2 up ip -n test2 link set lo up ip -n test1 addr add 192.168.1.2/31 dev veth1 ip -n test1 addr add 1.1.1.1/32 dev lo ip -n test2 addr add 192.168.1.3/31 dev veth2 ip -n test2 addr add 2.2.2.2/31 dev lo ip -n test1 route add default via 192.168.1.3 ip -n test2 route add default via 192.168.1.2 for i in `seq 10 210`; do for j in `seq 10 210`; do ip netns exec test2 iptables -I INPUT -s 3.3.$i.$j -p udp --dport 5201 done done ip netns exec test2 ethtool -K veth2 gro on ip netns exec test2 bash -c 'echo 1 > /sys/class/net/veth2/threaded' ip netns exec test1 ethtool -K veth1 tso off Then run an iperf3 client/server and a bpftrace script can trigger it: ip netns exec test2 iperf3 -s -B 2.2.2.2 >/dev/null& ip netns exec test1 iperf3 -c 2.2.2.2 -B 1.1.1.1 -u -l 1500 -b 3g -t 100 >/dev/null& bpftrace -e 'kfunc:__napi_poll{@=count();} interval:s:1{exit();}' Report RCU quiescent states periodically will resolve the issue. Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support") Suggested-by: Paul E. McKenney Reviewed-by: Jesper Dangaard Brouer Signed-off-by: Yan Zhai --- v2->v3: abstracted the work into a RCU helper v1->v2: moved rcu_softirq_qs out from bh critical section, and only raise it after a second of repolling. Added some brief perf test result. v2: https://lore.kernel.org/bpf/ZeFPz4D121TgvCje@debian.debian/ v1: https://lore.kernel.org/lkml/Zd4DXTyCf17lcTfq@debian.debian/#t --- net/core/dev.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/net/core/dev.c b/net/core/dev.c index 76e6438f4858..6b7fc42d4b3e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6708,6 +6708,8 @@ static int napi_threaded_poll(void *data) void *have; while (!napi_thread_wait(napi)) { + unsigned long last_qs = jiffies; + for (;;) { bool repoll = false; @@ -6732,6 +6734,7 @@ static int napi_threaded_poll(void *data) if (!repoll) break; + rcu_softirq_qs_periodic(last_qs); cond_resched(); } } From patchwork Wed Mar 13 16:25:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yan Zhai X-Patchwork-Id: 13591571 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9092E4D9FC for ; Wed, 13 Mar 2024 16:25:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710347161; cv=none; b=MtDkqFFfrGaoHM9Nk1j/CrQObiYDaQswGe3PDAQy+wXq8aL96SbQZ8lfHS7g7oIwSjrE9cD9tMDzg2aNDlZSMSIFCNazOQ+NVhgVgpAz6zsUmS2XGGXqHea+mfnQ3IqU/tQuLeZ2B/MzMETSvtERylAFNeOcPM4Vw9m7vG+G9iM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710347161; c=relaxed/simple; bh=WZXQ3n6OAHQk56baal6CzZ+snXfrxRm2xjx8a0VoWYc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=qPG1kspZrNAtE/EMy1nLByDJTrTAKDK8MuJHacf7Z+UNl6ZlZ/yPModfNka12GZgBU0G7ke+zcr8i949H/gAUjUJxVXYPaOOO951KiB0ciBNDa9zo3FJ/e9/QU9mKZe3QbimGlsAQK3q2uVTGDTS4g2ibJga1b0Dq7MdwxgheeY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com; spf=pass smtp.mailfrom=cloudflare.com; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.b=YveUiEzC; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cloudflare.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cloudflare.com header.i=@cloudflare.com header.b="YveUiEzC" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-42f37657e91so26800341cf.2 for ; Wed, 13 Mar 2024 09:25:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google09082023; t=1710347158; x=1710951958; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=JuTpc8vOHdn72vm+nW+iFZ+lBgZnpQ42Xv3ej2YV4rQ=; b=YveUiEzC/32H1ytCovwTz9zaPUHN1E0CPx783n6uHVYrg7GjxAUME3P0UhV1DTd6Ah 3zMTCZED8Fjuvqmq57tZma7oeF+mofWikhPkKytUA+a4s5Og6RErjr2LieFYUovauFWf AKoyQ3gh7xQ13w7nKgU/BOnF9UNTvzgh58A45IPS7AIKCE6zCtSq5askW1IljZw/xnKK 8yRqkdSl5Noc9xw8a4zyK1+jySvQuVsJV6jJuJUALn1aalzUIfP4aBeat0wXg4y2PVWh CevkxTP3UDfbhulwPsQTwX7a/cSOZnQvBmpF1A+5qIQE+C+MtNxsFh8zJnvYeszxXIQB KplA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710347158; x=1710951958; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=JuTpc8vOHdn72vm+nW+iFZ+lBgZnpQ42Xv3ej2YV4rQ=; b=QJ9yboUSpi2aM8usQkAvn3ib6MqId8yF7az3WpehKHOLwxzqjobMiaP64pTi6FaybT hTMCFZ3zYYZE9t5ep8jMr8YgV+gffoD3mVAIfCkvFhJZPBC92DjM0/gjUUkJbQyHZrO4 jdNkTR6PCBt6zIjpUAJYWFf7YWdnJWpZoNfNNV+DJx8RC/lZwSw9KRtihdRGkY9HznPP zwnsXNw6Jasas8xg5jXIdLrwrpujbBXrPh/kIEADbU8JLln2ehKh6zF88NlwOG4ZQNoI WvNlbatuVMASvJx7E2VvUyv+PkoMtKonVAfjKYf9Wd6FAvOVQ47WnsRbSDyF/Ssf4oC0 kZ1Q== X-Forwarded-Encrypted: i=1; AJvYcCXeWK8j54k2CFgAnajcTg2ixMD8kxencVDRdMjTc8pfsVKYP4Ov+QRSgWA1l9VlJlpyHjD5pxO8CuAnNiH17QNHR8BS X-Gm-Message-State: AOJu0YzxAQS+lIZ7lOzQYEXMyQY+4wcuSy/5Hme5HfrQr0AWPhnnV5aX 5N/8atde94FGJUkrEj7Yv2XEedZBygjhJcQzfRF9HDgzbUYyeVH41gW6hMBLkes= X-Google-Smtp-Source: AGHT+IEI5lC3i4ZzAG9M2/sNgNAyGuzNjHl2dssxPjvKQlI6GB/Cg43iYqdnjnkGtZ41amsgTrXgsg== X-Received: by 2002:a05:622a:11d5:b0:430:9773:b083 with SMTP id n21-20020a05622a11d500b004309773b083mr823078qtk.19.1710347158423; Wed, 13 Mar 2024 09:25:58 -0700 (PDT) Received: from debian.debian ([2a09:bac5:7a49:f91::18d:37]) by smtp.gmail.com with ESMTPSA id c25-20020ac853d9000000b0042f2130cd0csm4975824qtq.34.2024.03.13.09.25.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Mar 2024 09:25:57 -0700 (PDT) Date: Wed, 13 Mar 2024 09:25:55 -0700 From: Yan Zhai To: netdev@vger.kernel.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jiri Pirko , Simon Horman , Daniel Borkmann , Lorenzo Bianconi , Coco Li , Wei Wang , Alexander Duyck , Hannes Frederic Sowa , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, bpf@vger.kernel.org, kernel-team@cloudflare.com, Joel Fernandes , "Paul E. McKenney" , Toke =?utf-8?q?H=C3=B8iland-J?= =?utf-8?q?=C3=B8rgensen?= , Alexei Starovoitov , Steven Rostedt , mark.rutland@arm.com, Jesper Dangaard Brouer Subject: [PATCH v3 net 3/3] bpf: report RCU QS in cpumap kthread Message-ID: <3112a13efb21893b6cf285b3757877dc466c5f58.1710346410.git.yan@cloudflare.com> References: Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When there are heavy load, cpumap kernel threads can be busy polling packets from redirect queues and block out RCU tasks from reaching quiescent states. It is insufficient to just call cond_resched() in such context. Periodically raise a consolidated RCU QS before cond_resched fixes the problem. Fixes: 6710e1126934 ("bpf: introduce new bpf cpu map type BPF_MAP_TYPE_CPUMAP") Reviewed-by: Jesper Dangaard Brouer Signed-off-by: Yan Zhai --- kernel/bpf/cpumap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index ef82ffc90cbe..8f1d390bcbde 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -262,6 +262,7 @@ static int cpu_map_bpf_prog_run(struct bpf_cpu_map_entry *rcpu, void **frames, static int cpu_map_kthread_run(void *data) { struct bpf_cpu_map_entry *rcpu = data; + unsigned long last_qs = jiffies; complete(&rcpu->kthread_running); set_current_state(TASK_INTERRUPTIBLE); @@ -287,10 +288,12 @@ static int cpu_map_kthread_run(void *data) if (__ptr_ring_empty(rcpu->queue)) { schedule(); sched = 1; + last_qs = jiffies; } else { __set_current_state(TASK_RUNNING); } } else { + rcu_softirq_qs_periodic(last_qs); sched = cond_resched(); }