From patchwork Wed May 15 12:53:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13665197 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3578E82860; Wed, 15 May 2024 12:54:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.177.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777663; cv=none; b=IK7nyROKa1bvMmRA/YI0AUwa4EBuD8dSCpyIgwQiLC34kxrTVkQWokRBLhY7ZQb2OXxKTNHpmT3Xp/cpoR7IUbbDHZ+1ftF2j62B5NZaNFRHA6WPvvKWm52Yf2ux/QiikbTDbKc+8O5dOqiGQP3M+gHLQ74TXgwZ3fA1nYovEtA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777663; c=relaxed/simple; bh=MIxN3WKFAODN7v/HXRR/ds7aPZze5HQV8ktUpU9r0F0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=FkE7+y7gnISfAPLmROeZ8pTjDUCFpb1E2nRKLs4k6tj2tw35e6LjNl3e/Fn8xpbb9MuYEXqrSzHWsSE6aFxlK/dDCJzOm5fD2SwelgkNrlJ3VQrbhqoxiXkvHTcmJUNpRIQtwiiT53kOETKTpT+iS2K3vOgRFHJme2JlRp68bEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=P7mc4APK; arc=none smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="P7mc4APK" Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44F7nsbN023517; Wed, 15 May 2024 12:54:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=vu11EeMFysr0s0X80qX6ZZNyhjciJQ++IXjrRGklOts=; b=P7mc4APK1I6HcyPO6LgBCnhV+8uWWO8hALvsD6ANMfvT2zceJJ09V0sWu04q9gHgAXkz uiGIUdK3CbHlCTEK2v7P2EKG2kmmkQt9OtoejRs4gmMNXoZHwCywINDyLnVa9C0BHjV6 2MChULgUIOabFDkccXM1Ei1V9zelAzPZ+nNAdgmT+8xVT2qz0OT1HQMb9qFBdN6a6+IU KcJRo8KdLVX2BC9TxBT3PMUqefUxxsawHCdgzn4T7JlMkFXT8t0mluhpkbCi1z4qvgZO HyWTSKf+324KFOSGF67F/Ctl3/gZZiPzLQdoirSDTeGSbXKIQIrYpabqohHr4MDEgPpL 0A== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3y3tx34rwu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:00 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44FCZeHJ038274; Wed, 15 May 2024 12:53:53 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3y24pxgujk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:53:53 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 44FCmlra038458; Wed, 15 May 2024 12:53:52 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3y24pxgud9-2; Wed, 15 May 2024 12:53:52 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, rds-devel@oss.oracle.com Cc: Jason Gunthorpe , Leon Romanovsky , Saeed Mahameed , Tariq Toukan , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Tejun Heo , Lai Jiangshan , Allison Henderson , Manjunath Patil , Mark Zhang , =?utf-8?q?H=C3=A5kon_Bugge?= , Chuck Lever , Shiraz Saleem , Yang Li Subject: [PATCH v2 1/6] workqueue: Inherit NOIO and NOFS alloc flags Date: Wed, 15 May 2024 14:53:37 +0200 Message-Id: <20240515125342.1069999-2-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240515125342.1069999-1-haakon.bugge@oracle.com> References: <20240515125342.1069999-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-05-15_06,2024-05-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405150090 X-Proofpoint-GUID: MZWHxKg_o9YJ2gNopnjKutRdnNNRyYBR X-Proofpoint-ORIG-GUID: MZWHxKg_o9YJ2gNopnjKutRdnNNRyYBR For drivers/modules running inside a memalloc_{noio,nofs}_{save,restore} region, if a work-queue is created, we make sure work executed on the work-queue inherits the same flag(s). This in order to conditionally enable drivers to work aligned with block I/O devices. This commit makes sure that any work queued later on work-queues created during module initialization, when current's flags has PF_MEMALLOC_{NOIO,NOFS} set, will inherit the same flags. We do this in order to enable drivers to be used as a network block I/O device. This in order to support XFS or other file-systems on top of a raw block device which uses said drivers as the network transport layer. Under intense memory pressure, we get memory reclaims. Assume the file-system reclaims memory, goes to the raw block device, which calls into said drivers. Now, if regular GFP_KERNEL allocations in the drivers require reclaims to be fulfilled, we end up in a circular dependency. We break this circular dependency by: 1. Force all allocations in the drivers to use GFP_NOIO, by means of a parenthetic use of memalloc_noio_{save,restore} on all relevant entry points. 2. Make sure work-queues inherits current->flags wrt. PF_MEMALLOC_{NOIO,NOFS}, such that work executed on the work-queue inherits the same flag(s). That is what this commit contributes with. Signed-off-by: Håkon Bugge --- v1 -> v2: * Added missing hunk in alloc_workqueue() --- include/linux/workqueue.h | 2 ++ kernel/workqueue.c | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 158784dd189ab..09ecc692ffcae 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -398,6 +398,8 @@ enum wq_flags { __WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */ __WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */ __WQ_LEGACY = 1 << 18, /* internal: create*_workqueue() */ + __WQ_NOIO = 1 << 19, /* internal: execute work with NOIO */ + __WQ_NOFS = 1 << 20, /* internal: execute work with NOFS */ /* BH wq only allows the following flags */ __WQ_BH_ALLOWS = WQ_BH | WQ_HIGHPRI, diff --git a/kernel/workqueue.c b/kernel/workqueue.c index d2dbe099286b9..8eb7562372ce2 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -51,6 +51,7 @@ #include #include #include +#include #include #include #include @@ -3172,6 +3173,10 @@ __acquires(&pool->lock) unsigned long work_data; int lockdep_start_depth, rcu_start_depth; bool bh_draining = pool->flags & POOL_BH_DRAINING; + bool use_noio_allocs = pwq->wq->flags & __WQ_NOIO; + bool use_nofs_allocs = pwq->wq->flags & __WQ_NOFS; + unsigned long noio_flags; + unsigned long nofs_flags; #ifdef CONFIG_LOCKDEP /* * It is permissible to free the struct work_struct from @@ -3184,6 +3189,12 @@ __acquires(&pool->lock) lockdep_copy_map(&lockdep_map, &work->lockdep_map); #endif + /* Set inherited alloc flags */ + if (use_noio_allocs) + noio_flags = memalloc_noio_save(); + if (use_nofs_allocs) + nofs_flags = memalloc_nofs_save(); + /* ensure we're on the correct CPU */ WARN_ON_ONCE(!(pool->flags & POOL_DISASSOCIATED) && raw_smp_processor_id() != pool->cpu); @@ -3320,6 +3331,12 @@ __acquires(&pool->lock) /* must be the last step, see the function comment */ pwq_dec_nr_in_flight(pwq, work_data); + + /* Restore alloc flags */ + if (use_nofs_allocs) + memalloc_nofs_restore(nofs_flags); + if (use_noio_allocs) + memalloc_noio_restore(noio_flags); } /** @@ -5583,6 +5600,10 @@ struct workqueue_struct *alloc_workqueue(const char *fmt, /* init wq */ wq->flags = flags; + if (current->flags & PF_MEMALLOC_NOIO) + wq->flags |= __WQ_NOIO; + if (current->flags & PF_MEMALLOC_NOFS) + wq->flags |= __WQ_NOFS; wq->max_active = max_active; wq->min_active = min(max_active, WQ_DFL_MIN_ACTIVE); wq->saved_max_active = wq->max_active; From patchwork Wed May 15 12:53:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13665196 X-Patchwork-Delegate: kuba@kernel.org Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C5C784A37; Wed, 15 May 2024 12:54:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.177.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777660; cv=none; b=OFfefZQEXhJN4J5LLHzEok8yxK2LAckIiMgG148/mM2YSorvT4UgrBy2H8bjamyv2U5APWR1l1ugH29ZmZOoylxabNLEy32tGbWHv0ZhyHB0GbilVsgjXitjWJOcEc13kGzdqvkVMO8E/mSAUnJzR+DBaLWrEr3O92vi+KOQdtw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777660; c=relaxed/simple; bh=EcHU8N3DwLiEKSuSv/X62OvipkQbysqSYGuBdlNY1pg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=mVkGHTQf2y/R6lUw+ZlcCsftpPqenVgoakwlijo/WuVhIEdAEqO9eTSZUoaNzzLaZdb+NCasvJHKJKRcBFCDRwUefxArjPs3ebYnQ2k4tYKIGk9lhmJ2ppBgfrpNYBbvSUiqCSlatK0/c/DqIMHwm/NFR9Ca9MKnq/GWVbMrGMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=nx5RbzDa; arc=none smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="nx5RbzDa" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44F7n2n1008508; Wed, 15 May 2024 12:53:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=u17tuwlq4ab0H7PsxNcpg5nduawD3jBBMb6XH3A+JLs=; b=nx5RbzDaz1Dq245+AGp/S95PwVqDgTt0TNKgvJNaBCDcQkfk3TwNfS/ECLwnuALPXUby O9DTkVLIt9Wae1H5H6UmjEhMW2zFyLmtAs6Z4GyKEIXaXBOQUntTYoZadrEl8l1yhy7x GuPeCRCjIBOnp+qTy7hCeiE4YxPUpL0l6XPDycyqT6ZB7j7OT+wSNO9v/QqUg7sxETA+ Xp289kaa+L3d28BK2VpXZvAjLoQJ0h1BFNKTXTBgY1gs8O5tN4DKvqHnyw5qt+hbZRNT zP1ZK7VWDfjLhBSWgtYiaaTFvtNg8kFF9/C3eeIOT7UtZ+y8ohSOcp4F98SXX5vZh7cq ZQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3y3t4fcxjn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:53:59 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44FCZeHO038274; Wed, 15 May 2024 12:53:58 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3y24pxgun9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:53:58 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 44FCmlrd038458; Wed, 15 May 2024 12:53:57 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3y24pxgud9-3; Wed, 15 May 2024 12:53:56 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, rds-devel@oss.oracle.com Cc: Jason Gunthorpe , Leon Romanovsky , Saeed Mahameed , Tariq Toukan , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Tejun Heo , Lai Jiangshan , Allison Henderson , Manjunath Patil , Mark Zhang , =?utf-8?q?H=C3=A5kon_Bugge?= , Chuck Lever , Shiraz Saleem , Yang Li Subject: [PATCH v2 2/6] rds: Brute force GFP_NOIO Date: Wed, 15 May 2024 14:53:38 +0200 Message-Id: <20240515125342.1069999-3-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240515125342.1069999-1-haakon.bugge@oracle.com> References: <20240515125342.1069999-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-05-15_06,2024-05-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405150090 X-Proofpoint-ORIG-GUID: nSfqDXeJWV05AJdzTXEiPMEz7r85g8Pl X-Proofpoint-GUID: nSfqDXeJWV05AJdzTXEiPMEz7r85g8Pl X-Patchwork-Delegate: kuba@kernel.org For most entry points to RDS, we call memalloc_noio_{save,restore} in a parenthetic fashion when enabled by the module parameter force_noio. We skip the calls to memalloc_noio_{save,restore} in rds_ioctl(), as no memory allocations are executed in this function or its callees. The reason we execute memalloc_noio_{save,restore} in rds_poll(), is due to the following call chain: rds_poll() poll_wait() __pollwait() poll_get_entry() __get_free_page(GFP_KERNEL) The function rds_setsockopt() allocates memory in its callee's rds_get_mr() and rds_get_mr_for_dest(). Hence, we need memalloc_noio_{save,restore} in rds_setsockopt(). In rds_getsockopt(), we have rds_info_getsockopt() that allocates memory. Hence, we need memalloc_noio_{save,restore} in rds_getsockopt(). All the above, in order to conditionally enable RDS to become a block I/O device. Signed-off-by: Håkon Bugge --- v1 -> v2: * s/EXPORT_SYMBOL/static/ for the rds_force_noio variable as pin-pointed by Simon * Straightened the reverse xmas tree two places * Fixed C/P error in rds_cancel_sent_to() where I had two _save()s and no _restore() as reported by Simon --- net/rds/af_rds.c | 59 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-) diff --git a/net/rds/af_rds.c b/net/rds/af_rds.c index 8435a20968ef5..846ad20b3783a 100644 --- a/net/rds/af_rds.c +++ b/net/rds/af_rds.c @@ -37,10 +37,15 @@ #include #include #include +#include #include #include "rds.h" +static bool rds_force_noio; +module_param_named(force_noio, rds_force_noio, bool, 0444); +MODULE_PARM_DESC(force_noio, "Force the use of GFP_NOIO (Y/N)"); + /* this is just used for stats gathering :/ */ static DEFINE_SPINLOCK(rds_sock_lock); static unsigned long rds_sock_count; @@ -59,8 +64,12 @@ DECLARE_WAIT_QUEUE_HEAD(rds_poll_waitq); static int rds_release(struct socket *sock) { struct sock *sk = sock->sk; + unsigned int noio_flags; struct rds_sock *rs; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + if (!sk) goto out; @@ -90,6 +99,8 @@ static int rds_release(struct socket *sock) sock->sk = NULL; sock_put(sk); out: + if (rds_force_noio) + memalloc_noio_restore(noio_flags); return 0; } @@ -214,9 +225,13 @@ static __poll_t rds_poll(struct file *file, struct socket *sock, { struct sock *sk = sock->sk; struct rds_sock *rs = rds_sk_to_rs(sk); + unsigned int noio_flags; __poll_t mask = 0; unsigned long flags; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + poll_wait(file, sk_sleep(sk), wait); if (rs->rs_seen_congestion) @@ -249,6 +264,8 @@ static __poll_t rds_poll(struct file *file, struct socket *sock, if (mask) rs->rs_seen_congestion = 0; + if (rds_force_noio) + memalloc_noio_restore(noio_flags); return mask; } @@ -293,9 +310,13 @@ static int rds_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) static int rds_cancel_sent_to(struct rds_sock *rs, sockptr_t optval, int len) { struct sockaddr_in6 sin6; + unsigned int noio_flags; struct sockaddr_in sin; int ret = 0; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + /* racing with another thread binding seems ok here */ if (ipv6_addr_any(&rs->rs_bound_addr)) { ret = -ENOTCONN; /* XXX not a great errno */ @@ -324,6 +345,8 @@ static int rds_cancel_sent_to(struct rds_sock *rs, sockptr_t optval, int len) rds_send_drop_to(rs, &sin6); out: + if (rds_force_noio) + memalloc_noio_restore(noio_flags); return ret; } @@ -485,8 +508,12 @@ static int rds_getsockopt(struct socket *sock, int level, int optname, { struct rds_sock *rs = rds_sk_to_rs(sock->sk); int ret = -ENOPROTOOPT, len; + unsigned int noio_flags; int trans; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + if (level != SOL_RDS) goto out; @@ -529,6 +556,8 @@ static int rds_getsockopt(struct socket *sock, int level, int optname, } out: + if (rds_force_noio) + memalloc_noio_restore(noio_flags); return ret; } @@ -538,12 +567,16 @@ static int rds_connect(struct socket *sock, struct sockaddr *uaddr, { struct sock *sk = sock->sk; struct sockaddr_in *sin; + unsigned int noio_flags; struct rds_sock *rs = rds_sk_to_rs(sk); int ret = 0; if (addr_len < offsetofend(struct sockaddr, sa_family)) return -EINVAL; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + lock_sock(sk); switch (uaddr->sa_family) { @@ -626,6 +659,8 @@ static int rds_connect(struct socket *sock, struct sockaddr *uaddr, } release_sock(sk); + if (rds_force_noio) + memalloc_noio_restore(noio_flags); return ret; } @@ -697,16 +732,28 @@ static int __rds_create(struct socket *sock, struct sock *sk, int protocol) static int rds_create(struct net *net, struct socket *sock, int protocol, int kern) { + unsigned int noio_flags; struct sock *sk; + int ret; if (sock->type != SOCK_SEQPACKET || protocol) return -ESOCKTNOSUPPORT; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + sk = sk_alloc(net, AF_RDS, GFP_KERNEL, &rds_proto, kern); - if (!sk) - return -ENOMEM; + if (!sk) { + ret = -ENOMEM; + goto out; + } - return __rds_create(sock, sk, protocol); + ret = __rds_create(sock, sk, protocol); +out: + if (rds_force_noio) + memalloc_noio_restore(noio_flags); + + return ret; } void rds_sock_addref(struct rds_sock *rs) @@ -895,8 +942,12 @@ u32 rds_gen_num; static int __init rds_init(void) { + unsigned int noio_flags; int ret; + if (rds_force_noio) + noio_flags = memalloc_noio_save(); + net_get_random_once(&rds_gen_num, sizeof(rds_gen_num)); ret = rds_bind_lock_init(); @@ -947,6 +998,8 @@ static int __init rds_init(void) out_bind: rds_bind_lock_destroy(); out: + if (rds_force_noio) + memalloc_noio_restore(noio_flags); return ret; } module_init(rds_init); From patchwork Wed May 15 12:53:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13665198 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBCE6128382; Wed, 15 May 2024 12:54:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.177.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777666; cv=none; b=dfX4V+rBl93i6NvpzTzY2DK8Or8wI7Kj1Zz/VQ5H+D+/y98tNVnA9FM3VQsbr9gi+r/0HFFYw6Da2RIPUm3pjrzEzLvFR9mnml7/bPanhTT7mBtFSmoljITPaDdj9IU+/+N1v3lcGryf7sd/dppZcPcGMKFhfrkUgdmUK/LOJ28= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777666; c=relaxed/simple; bh=bC51ekxs+x55vMw5lPjDrzXR33HcuoruRI1JEsJzfAQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=nsoFQqTm0CIwfy5yuhbYfiLswqL/vCf7Uw6ua+5JMe1n5uuQD63n5KcdxLT5p45BcSWj/txuCfYzkBgZHRMBb5TPuYPe49nd+TIyXQwF6Y2VpXge5GfJx2sbgAz3pXPN7nkAsrjpm62d4TCrpVfmEhzusH1M2bxLmzhD+KXMMsA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=jwZU8C3x; arc=none smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="jwZU8C3x" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44F7n2n4008508; Wed, 15 May 2024 12:54:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=lwFCNObJX0XFJa8hB9AVQPbhaHJVkhAUATWysKIQAgU=; b=jwZU8C3x/otvDKVXjLu5jM6iBLWrzNPMqLm5OumM7uLmuVPYNzI/YWNJkaI3BVcJ6d93 9049Uz8Mc8W7z6BARMbO/iM/Q1vtmwzbCb9KisAQcr9gK/WmSktePESkZKMCpm9CbOsB dQBJznWtdS/sys7Q3VqnqLF0ZfT0gzzX5ujb4N2x/InM3YAZ2fh/L3zxAoTbKY8CEtJr 2A5AHO1rnzpsdFcHegQjHOkCFHNx7lFQST/SV5U+vMcXxsX/oC0f3OPwwL+Hg+mOC92g CbXP9pGbwGqB8mBlH+acFqjVz1jru3N+i2LXVfiJdyWKsWT7+uJyV0jVuMC4kZfwJT4f 1A== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3y3t4fcxkc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:03 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44FB3H0j038357; Wed, 15 May 2024 12:54:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3y24pxgus7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:02 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 44FCmlrg038458; Wed, 15 May 2024 12:54:02 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3y24pxgud9-4; Wed, 15 May 2024 12:54:01 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, rds-devel@oss.oracle.com Cc: Jason Gunthorpe , Leon Romanovsky , Saeed Mahameed , Tariq Toukan , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Tejun Heo , Lai Jiangshan , Allison Henderson , Manjunath Patil , Mark Zhang , =?utf-8?q?H=C3=A5kon_Bugge?= , Chuck Lever , Shiraz Saleem , Yang Li Subject: [PATCH v2 3/6] RDMA/cma: Brute force GFP_NOIO Date: Wed, 15 May 2024 14:53:39 +0200 Message-Id: <20240515125342.1069999-4-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240515125342.1069999-1-haakon.bugge@oracle.com> References: <20240515125342.1069999-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-05-15_06,2024-05-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405150090 X-Proofpoint-ORIG-GUID: oFChewOnUQFX9hbMFeg7JCRgZXWct7Wm X-Proofpoint-GUID: oFChewOnUQFX9hbMFeg7JCRgZXWct7Wm In cma_init(), we call memalloc_noio_{save,restore} in a parenthetic fashion when enabled by the module parameter force_noio. This in order to conditionally enable rdma_cm to work aligned with block I/O devices. Any work queued later on work-queues created during module initialization will inherit the PF_MEMALLOC_{NOIO,NOFS} flag(s), due to commit ("workqueue: Inherit NOIO and NOFS alloc flags"). We do this in order to enable ULPs using the RDMA stack to be used as a network block I/O device. This to support a filesystem on top of a raw block device which uses said ULP(s) and the RDMA stack as the network transport layer. Under intense memory pressure, we get memory reclaims. Assume the filesystem reclaims memory, goes to the raw block device, which calls into the ULP in question, which calls the RDMA stack. Now, if regular GFP_KERNEL allocations in the ULP or the RDMA stack require reclaims to be fulfilled, we end up in a circular dependency. We break this circular dependency by: 1. Force all allocations in the ULP and the relevant RDMA stack to use GFP_NOIO, by means of a parenthetic use of memalloc_noio_{save,restore} on all relevant entry points. 2. Make sure work-queues inherits current->flags wrt. PF_MEMALLOC_{NOIO,NOFS}, such that work executed on the work-queue inherits the same flag(s). Signed-off-by: Håkon Bugge --- drivers/infiniband/core/cma.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 1e2cd7c8716e8..23a50cc3e81cb 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -50,6 +50,10 @@ MODULE_LICENSE("Dual BSD/GPL"); #define CMA_IBOE_PACKET_LIFETIME 16 #define CMA_PREFERRED_ROCE_GID_TYPE IB_GID_TYPE_ROCE_UDP_ENCAP +static bool cma_force_noio; +module_param_named(force_noio, cma_force_noio, bool, 0444); +MODULE_PARM_DESC(force_noio, "Force the use of GFP_NOIO (Y/N)"); + static const char * const cma_events[] = { [RDMA_CM_EVENT_ADDR_RESOLVED] = "address resolved", [RDMA_CM_EVENT_ADDR_ERROR] = "address error", @@ -5424,6 +5428,10 @@ static struct pernet_operations cma_pernet_operations = { static int __init cma_init(void) { int ret; + unsigned int noio_flags; + + if (cma_force_noio) + noio_flags = memalloc_noio_save(); /* * There is a rare lock ordering dependency in cma_netdev_callback() @@ -5439,8 +5447,10 @@ static int __init cma_init(void) } cma_wq = alloc_ordered_workqueue("rdma_cm", WQ_MEM_RECLAIM); - if (!cma_wq) - return -ENOMEM; + if (!cma_wq) { + ret = -ENOMEM; + goto out; + } ret = register_pernet_subsys(&cma_pernet_operations); if (ret) @@ -5458,7 +5468,8 @@ static int __init cma_init(void) if (ret) goto err_ib; - return 0; + ret = 0; + goto out; err_ib: ib_unregister_client(&cma_client); @@ -5469,6 +5480,9 @@ static int __init cma_init(void) unregister_pernet_subsys(&cma_pernet_operations); err_wq: destroy_workqueue(cma_wq); +out: + if (cma_force_noio) + memalloc_noio_restore(noio_flags); return ret; } From patchwork Wed May 15 12:53:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13665201 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE854101C8; Wed, 15 May 2024 12:54:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.177.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777698; cv=none; b=qhTleSnS7voAYGKp18OWs0rF7SZfLBu99ZeEmH/lZ+C1a3ORs+qzraRMr+Y0AEw0OTXG+eaHpuCcMsYSI7PXkB7xNQUEyx1Z80Kao3msQV7oGA60ImgehkdWEZUw3L0NYeVAouyBlv7jW7T7I/b23RcvMQfyQCvW0TTB+a6amDg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777698; c=relaxed/simple; bh=wUZJNekknAtY0xAkk6kJRnfr+7rn2D06aTHGKn1Bv90=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=PD77YI6FXUn1ZckuTvN+3gWrrJvfULeYM6v+nlYjUskfV+9ZuVDwsdnc/+e3dZeLdmSD7wTt5MlWDJNKtNyWY5TR3Wcphy9Kukb4RoDZ4voR3YBZr0dR/sfrVzSBF7oMHkEPdA/CEc2d3QxluY3oFqV5PrrjtY0VywM5hL6ZQaA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=hx28RVeN; arc=none smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="hx28RVeN" Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44F7n23v008505; Wed, 15 May 2024 12:54:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=y5Ep9p+cIwPUrjX1dgCqftpNDUoKQDrgqAGsTqnOQGI=; b=hx28RVeNq/YVOuf09oBVYT2LGDZ5RQmlcJ5exnHEZEmvKviuAcO4xu2pE/SY7tWd+KhW 0NRNlenWkE2BUNbAhvA+pas1BbmelpDIak9pZ9ffNfBAHNBUQQBBGaiem9cNSlpPWgWd yms+HZo/6ueYLrBIcQwOq8w4qhTF0Xpj6oyFTeS4YGk8w2iChwoX7DeCoDVMl5BFOoss EDzRaJd9dSqKjbCXRU8Gf7kSFZ75/I0yE68XnyjTxRABN1LVG5cz0UYKtvE9lBsavfhl +ztZPU4XgKP7PP+OYoRjepJthWuAb4bFl0HNuspCz9HISsQok8RCAC6OkLHrrl+CjMce 8g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3y3t4fcxkj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:08 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44FBLBLx038451; Wed, 15 May 2024 12:54:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3y24pxguvp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:07 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 44FCmlrj038458; Wed, 15 May 2024 12:54:06 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3y24pxgud9-5; Wed, 15 May 2024 12:54:06 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, rds-devel@oss.oracle.com Cc: Jason Gunthorpe , Leon Romanovsky , Saeed Mahameed , Tariq Toukan , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Tejun Heo , Lai Jiangshan , Allison Henderson , Manjunath Patil , Mark Zhang , =?utf-8?q?H=C3=A5kon_Bugge?= , Chuck Lever , Shiraz Saleem , Yang Li Subject: [PATCH v2 4/6] RDMA/cm: Brute force GFP_NOIO Date: Wed, 15 May 2024 14:53:40 +0200 Message-Id: <20240515125342.1069999-5-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240515125342.1069999-1-haakon.bugge@oracle.com> References: <20240515125342.1069999-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-05-15_06,2024-05-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405150090 X-Proofpoint-ORIG-GUID: KnmR6otdKS4p2wIQ7YWGWGOG1z_IZBy2 X-Proofpoint-GUID: KnmR6otdKS4p2wIQ7YWGWGOG1z_IZBy2 In ib_cm_init(), we call memalloc_noio_{save,restore} in a parenthetic fashion when enabled by the module parameter force_noio. This in order to conditionally enable ib_cm to work aligned with block I/O devices. Any work queued later on work-queues created during module initialization will inherit the PF_MEMALLOC_{NOIO,NOFS} flag(s), due to commit ("workqueue: Inherit NOIO and NOFS alloc flags"). We do this in order to enable ULPs using the RDMA stack to be used as a network block I/O device. This to support a filesystem on top of a raw block device which uses said ULP(s) and the RDMA stack as the network transport layer. Under intense memory pressure, we get memory reclaims. Assume the filesystem reclaims memory, goes to the raw block device, which calls into the ULP in question, which calls the RDMA stack. Now, if regular GFP_KERNEL allocations in ULP or the RDMA stack require reclaims to be fulfilled, we end up in a circular dependency. We break this circular dependency by: 1. Force all allocations in the ULP and the relevant RDMA stack to use GFP_NOIO, by means of a parenthetic use of memalloc_noio_{save,restore} on all relevant entry points. 2. Make sure work-queues inherits current->flags wrt. PF_MEMALLOC_{NOIO,NOFS}, such that work executed on the work-queue inherits the same flag(s). Signed-off-by: Håkon Bugge --- drivers/infiniband/core/cm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c index 07fb8d3c037f0..767eec38eb57d 100644 --- a/drivers/infiniband/core/cm.c +++ b/drivers/infiniband/core/cm.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include @@ -35,6 +36,11 @@ MODULE_DESCRIPTION("InfiniBand CM"); MODULE_LICENSE("Dual BSD/GPL"); #define CM_DESTROY_ID_WAIT_TIMEOUT 10000 /* msecs */ + +static bool cm_force_noio; +module_param_named(force_noio, cm_force_noio, bool, 0444); +MODULE_PARM_DESC(force_noio, "Force the use of GFP_NOIO (Y/N)"); + static const char * const ibcm_rej_reason_strs[] = { [IB_CM_REJ_NO_QP] = "no QP", [IB_CM_REJ_NO_EEC] = "no EEC", @@ -4504,6 +4510,10 @@ static void cm_remove_one(struct ib_device *ib_device, void *client_data) static int __init ib_cm_init(void) { int ret; + unsigned int noio_flags; + + if (cm_force_noio) + noio_flags = memalloc_noio_save(); INIT_LIST_HEAD(&cm.device_list); rwlock_init(&cm.device_lock); @@ -4527,10 +4537,13 @@ static int __init ib_cm_init(void) if (ret) goto error3; - return 0; + goto error2; error3: destroy_workqueue(cm.wq); error2: + if (cm_force_noio) + memalloc_noio_restore(noio_flags); + return ret; } From patchwork Wed May 15 12:53:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13665199 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0C2612D1EB; Wed, 15 May 2024 12:54:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.165.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777670; cv=none; b=lEbgAakJs7jUrJ1NJ6r98N0AyH1iFMquqm4LWNDHB0rohz/jMRA6VHN+xImAEAPq20Ye0td838tyRyyU89NGby6XKYkwgiArSwQoieaanvJI3Jd0qMrBZb8/AY6MLAOPdkhw5YhuBcn69bG8Twkj3Ydfq3rP8NJZCWZhIylsHPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777670; c=relaxed/simple; bh=WDczq20XbWRkd6erlw7nKOvVwrFt2POHObD/O5KkuIM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=sJnMUNALiIaJnEaTdqX/I10PiFAM5dd9BbbF7o28ll5JriT5faSVJ4y/SHRiKGT7BILXy8+sq+i940lf2fuBTgBhpb3x5/rwoB1AFUagg6XZpmAbXHfz9xO0aZi9TNKgSDgPL13exincqgNsvkH1tBkWFhs+evtmJXY8B4nKeQg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=Stl2WT2D; arc=none smtp.client-ip=205.220.165.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="Stl2WT2D" Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44F7nmhU007983; Wed, 15 May 2024 12:54:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=Oul7kiaHhrfZm4+kHLzEtR9tfTF72t6p8eV981BEwMs=; b=Stl2WT2D3dUP3PdsN7iX9vIfmcdXi13jHmxVEn1gCFzFXFsMHDbVezfv3ahd3GzWYpxH Orcvt54dz4gkROZSox0YWAKms0v6sxJZvUoW66MLUpnaza6KzRxbnNRVTsGKaoKlO4Rl PG04pkr6+zKkOgNm3XR8+7/4Myop1IpYG46uc5F8XnQ3ImYuwx9l8YMxOHifNWnkzj1J ot/uGeB2Pp3d9dSXxq0yCJXh02jx5mnjW8Ab8ldwZowvArPxiLk3Q4jNUpYOLWo23ve0 PDKeBaoiYJUhlQ7SU91D3N9ihXVkLUFzqoUfDw3RGBtfuazaqOhtjR7y1JqG/+aeSt8E qw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3y3txc2xy4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:12 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44FCoUj4038301; Wed, 15 May 2024 12:54:12 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3y24pxguyk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:12 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 44FCmlrm038458; Wed, 15 May 2024 12:54:11 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3y24pxgud9-6; Wed, 15 May 2024 12:54:11 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, rds-devel@oss.oracle.com Cc: Jason Gunthorpe , Leon Romanovsky , Saeed Mahameed , Tariq Toukan , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Tejun Heo , Lai Jiangshan , Allison Henderson , Manjunath Patil , Mark Zhang , =?utf-8?q?H=C3=A5kon_Bugge?= , Chuck Lever , Shiraz Saleem , Yang Li Subject: [PATCH v2 5/6] RDMA/mlx5: Brute force GFP_NOIO Date: Wed, 15 May 2024 14:53:41 +0200 Message-Id: <20240515125342.1069999-6-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240515125342.1069999-1-haakon.bugge@oracle.com> References: <20240515125342.1069999-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-05-15_06,2024-05-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405150090 X-Proofpoint-GUID: uqAvpFv5BDNlq6wgEzrgXWF0NFwsdt3m X-Proofpoint-ORIG-GUID: uqAvpFv5BDNlq6wgEzrgXWF0NFwsdt3m In mlx5_ib_init(), we call memalloc_noio_{save,restore} in a parenthetic fashion when enabled by the module parameter force_noio. This in order to conditionally enable mlx5_ib to work aligned with I/O devices. Any work queued later on work-queues created during module initialization will inherit the PF_MEMALLOC_{NOIO,NOFS} flag(s), due to commit ("workqueue: Inherit NOIO and NOFS alloc flags"). We do this in order to enable ULPs using the RDMA stack and the mlx5_ib driver to be used as a network block I/O device. This to support a filesystem on top of a raw block device which uses said ULP(s) and the RDMA stack as the network transport layer. Under intense memory pressure, we get memory reclaims. Assume the filesystem reclaims memory, goes to the raw block device, which calls into the ULP in question, which calls the RDMA stack. Now, if regular GFP_KERNEL allocations in ULP or the RDMA stack require reclaims to be fulfilled, we end up in a circular dependency. We break this circular dependency by: 1. Force all allocations in the ULP and the relevant RDMA stack to use GFP_NOIO, by means of a parenthetic use of memalloc_noio_{save,restore} on all relevant entry points. 2. Make sure work-queues inherits current->flags wrt. PF_MEMALLOC_{NOIO,NOFS}, such that work executed on the work-queue inherits the same flag(s). Signed-off-by: Håkon Bugge --- drivers/infiniband/hw/mlx5/main.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index c2b557e642906..a424d518538ed 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -56,6 +56,10 @@ MODULE_AUTHOR("Eli Cohen "); MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) IB driver"); MODULE_LICENSE("Dual BSD/GPL"); +static bool mlx5_ib_force_noio; +module_param_named(force_noio, mlx5_ib_force_noio, bool, 0444); +MODULE_PARM_DESC(force_noio, "Force the use of GFP_NOIO (Y/N)"); + struct mlx5_ib_event_work { struct work_struct work; union { @@ -4489,16 +4493,23 @@ static struct auxiliary_driver mlx5r_driver = { static int __init mlx5_ib_init(void) { + unsigned int noio_flags; int ret; + if (mlx5_ib_force_noio) + noio_flags = memalloc_noio_save(); + xlt_emergency_page = (void *)__get_free_page(GFP_KERNEL); - if (!xlt_emergency_page) - return -ENOMEM; + if (!xlt_emergency_page) { + ret = -ENOMEM; + goto out; + } mlx5_ib_event_wq = alloc_ordered_workqueue("mlx5_ib_event_wq", 0); if (!mlx5_ib_event_wq) { free_page((unsigned long)xlt_emergency_page); - return -ENOMEM; + ret = -ENOMEM; + goto out; } ret = mlx5_ib_qp_event_init(); @@ -4515,7 +4526,7 @@ static int __init mlx5_ib_init(void) ret = auxiliary_driver_register(&mlx5r_driver); if (ret) goto drv_err; - return 0; + goto out; drv_err: auxiliary_driver_unregister(&mlx5r_mp_driver); @@ -4526,6 +4537,9 @@ static int __init mlx5_ib_init(void) qp_event_err: destroy_workqueue(mlx5_ib_event_wq); free_page((unsigned long)xlt_emergency_page); +out: + if (mlx5_ib_force_noio) + memalloc_noio_restore(noio_flags); return ret; } From patchwork Wed May 15 12:53:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13665200 X-Patchwork-Delegate: kuba@kernel.org Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 156C412A14C; Wed, 15 May 2024 12:54:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.177.32 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777688; cv=none; b=ZPQs56phd+qaMw+rKkVlPQFs1GCisu04RiQTtWOeTpJM7mMNAxp+1pUKKRIuMh8kOtHFq/rclWBlgDfEt5Ebm/d/It3Rq+ojFnZ/miEIjHT+e0l0AbdvilwOP7/E5z7LA2vn/dTSDkat/aaMteoys+en5ENirdbtZND1GU4a/7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715777688; c=relaxed/simple; bh=BwJptKrYjAbKhxsaZQSlX4GaCMh473E6P/Oc3sa5jBg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=eGukuESUSM/IQRhg6II/IENxMyn5nAajc7BCe2NIlC32Mz0KbgUofnm12zCnLW2kDqoPyszBHD6kElmA3r3HWl40xEcZXW6OX8ITc2YrOBb84qfVuMGAORlWXH5ZMS8F3r4MYw32w5LgUNs+rlLJRf+kED9cVdi4j+igO7xJFSA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=abbjIaF0; arc=none smtp.client-ip=205.220.177.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=oracle.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="abbjIaF0" Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 44F7nSqc020198; Wed, 15 May 2024 12:54:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=s0o7+CZvTIXe6mctHjZROAylxKEWHdEvfBwqKM9aQ9o=; b=abbjIaF0AwNIVU8KK/hiL64wblA/l9agjjV6C+0tS4HXrSL4j9ZCQ9KNzp0CkxGnStWY 4ze8kyDLHTdTGBJoS4omt00OUHuH6EP/D3H7dzTpM3EdkJmj4JTul1DLpiBCzusrMfDX Vlk7CTCOBk11ZJN5E0Q+GXgGnUi6C6Gy0wGCqtTT56EfwSuGz3W4Fho1nZWe1OGurpBE TgSMh8GZ/umtuGjljh4PPDFVyKRsludETde96qiHo0woiJeZ7wf9Q8HgG2H7+/KH0a/w Y+Ei6JIwpj0KSz32mBq/+qGebYmzfxc403btMpNWZMuG+mxUHJS9G/rfQWS+E7oU26T5 /g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3y3rh7d68a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:18 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 44FCoUj7038301; Wed, 15 May 2024 12:54:17 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3y24pxgv26-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 May 2024 12:54:17 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 44FCmlrp038458; Wed, 15 May 2024 12:54:16 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3y24pxgud9-7; Wed, 15 May 2024 12:54:15 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, rds-devel@oss.oracle.com Cc: Jason Gunthorpe , Leon Romanovsky , Saeed Mahameed , Tariq Toukan , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Tejun Heo , Lai Jiangshan , Allison Henderson , Manjunath Patil , Mark Zhang , =?utf-8?q?H=C3=A5kon_Bugge?= , Chuck Lever , Shiraz Saleem , Yang Li Subject: [PATCH v2 6/6] net/mlx5: Brute force GFP_NOIO Date: Wed, 15 May 2024 14:53:42 +0200 Message-Id: <20240515125342.1069999-7-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240515125342.1069999-1-haakon.bugge@oracle.com> References: <20240515125342.1069999-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-05-15_06,2024-05-15_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000 definitions=main-2405150090 X-Proofpoint-ORIG-GUID: N-dzc11NI8Kv6kMPLKg5cydwUWlPdkj4 X-Proofpoint-GUID: N-dzc11NI8Kv6kMPLKg5cydwUWlPdkj4 X-Patchwork-Delegate: kuba@kernel.org In mlx5_core_init(), we call memalloc_noio_{save,restore} in a parenthetic fashion when enabled by the module parameter force_noio. This in order to conditionally enable mlx5_core to work aligned with I/O devices. Any work queued later on work-queues created during module initialization will inherit the PF_MEMALLOC_{NOIO,NOFS} flag(s), due to commit ("workqueue: Inherit NOIO and NOFS alloc flags"). We do this in order to enable ULPs using the RDMA stack and the mlx5_core driver to be used as a network block I/O device. This to support a filesystem on top of a raw block device which uses said ULP(s) and the RDMA stack as the network transport layer. Under intense memory pressure, we get memory reclaims. Assume the filesystem reclaims memory, goes to the raw block device, which calls into the ULP in question, which calls the RDMA stack. Now, if regular GFP_KERNEL allocations in ULP or the RDMA stack require reclaims to be fulfilled, we end up in a circular dependency. We break this circular dependency by: 1. Force all allocations in the ULP and the relevant RDMA stack to use GFP_NOIO, by means of a parenthetic use of memalloc_noio_{save,restore} on all relevant entry points. 2. Make sure work-queues inherits current->flags wrt. PF_MEMALLOC_{NOIO,NOFS}, such that work executed on the work-queue inherits the same flag(s). Signed-off-by: Håkon Bugge --- drivers/net/ethernet/mellanox/mlx5/core/main.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 331ce47f51a17..aa1bf8bb5d15c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "mlx5_core.h" #include "lib/eq.h" #include "fs_core.h" @@ -87,6 +88,10 @@ static unsigned int prof_sel = MLX5_DEFAULT_PROF; module_param_named(prof_sel, prof_sel, uint, 0444); MODULE_PARM_DESC(prof_sel, "profile selector. Valid range 0 - 2"); +static bool mlx5_core_force_noio; +module_param_named(force_noio, mlx5_core_force_noio, bool, 0444); +MODULE_PARM_DESC(force_noio, "Force the use of GFP_NOIO (Y/N)"); + static u32 sw_owner_id[4]; #define MAX_SW_VHCA_ID (BIT(__mlx5_bit_sz(cmd_hca_cap_2, sw_vhca_id)) - 1) static DEFINE_IDA(sw_vhca_ida); @@ -2312,8 +2317,12 @@ static void mlx5_core_verify_params(void) static int __init mlx5_init(void) { + unsigned int noio_flags; int err; + if (mlx5_core_force_noio) + noio_flags = memalloc_noio_save(); + WARN_ONCE(strcmp(MLX5_ADEV_NAME, KBUILD_MODNAME), "mlx5_core name not in sync with kernel module name"); @@ -2334,7 +2343,7 @@ static int __init mlx5_init(void) if (err) goto err_pci; - return 0; + goto out; err_pci: mlx5_sf_driver_unregister(); @@ -2342,6 +2351,9 @@ static int __init mlx5_init(void) mlx5e_cleanup(); err_debug: mlx5_unregister_debugfs(); +out: + if (mlx5_core_force_noio) + memalloc_noio_restore(noio_flags); return err; }