From patchwork Fri Oct 14 21:37:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0EEAC4332F for ; Fri, 14 Oct 2022 21:38:23 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0B60hZ5z1y60; Fri, 14 Oct 2022 14:38:22 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B00BrWz1xJR for ; Fri, 14 Oct 2022 14:38:16 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id DF97D1006F1A; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id D7AA7DD53C; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:52 -0400 Message-Id: <1665783491-13827-2-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 01/20] lustre: ptlrpc: protect rq_repmsg in ptlrpc_req_drop_rs() X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lei Feng , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Lei Feng There is a race condition that: on server side, one thread sent reply message and is deleting the reply message, another is searching for existing request and print some debug information in _debug_req() if there is a duplicated request. They both operate on req->rq_repmsg but it is not protected in ptlrpc_req_drop_rs(). So we protected it with req->rq_early_free_lock. WC-bug-id: https://jira.whamcloud.com/browse/LU-15986 Lustre-commit: aaef545cff2dd9584 ("LU-15986 ptlrpc: protect rq_repmsg in ptlrpc_req_drop_rs()") Signed-off-by: Lei Feng Reviewed-on: https://review.whamcloud.com/47839 Reviewed-by: Andreas Dilger Reviewed-by: Li Xi Reviewed-by: Qian Yingjin Reviewed-by: Andrew Perepechko Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_net.h | 9 ++++++++- fs/lustre/ptlrpc/service.c | 1 + 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/fs/lustre/include/lustre_net.h b/fs/lustre/include/lustre_net.h index f8d28619a6d7..1605fcc64fc4 100644 --- a/fs/lustre/include/lustre_net.h +++ b/fs/lustre/include/lustre_net.h @@ -2234,9 +2234,16 @@ static inline void ptlrpc_req_drop_rs(struct ptlrpc_request *req) { if (!req->rq_reply_state) return; /* shouldn't occur */ + + /* req_repmsg equals rq_reply_state->rs_msg, + * so set it to NULL before rq_reply_state is possibly freed + */ + spin_lock(&req->rq_early_free_lock); + req->rq_repmsg = NULL; + spin_unlock(&req->rq_early_free_lock); + ptlrpc_rs_decref(req->rq_reply_state); req->rq_reply_state = NULL; - req->rq_repmsg = NULL; } static inline u32 lustre_request_magic(struct ptlrpc_request *req) diff --git a/fs/lustre/ptlrpc/service.c b/fs/lustre/ptlrpc/service.c index 277fbdbc590a..59fe1f4aa18f 100644 --- a/fs/lustre/ptlrpc/service.c +++ b/fs/lustre/ptlrpc/service.c @@ -1136,6 +1136,7 @@ static int ptlrpc_at_send_early_reply(struct ptlrpc_request *req) } *reqcopy = *req; + spin_lock_init(&reqcopy->rq_early_free_lock); reqcopy->rq_reply_state = NULL; reqcopy->rq_rep_swab_mask = 0; reqcopy->rq_pack_bulk = 0; From patchwork Fri Oct 14 21:37:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A92FEC433FE for ; Fri, 14 Oct 2022 21:38:24 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0B674QNz1y2J; Fri, 14 Oct 2022 14:38:22 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B05Fz2z1xJR for ; Fri, 14 Oct 2022 14:38:16 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id E3EBF1006F20; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id D89B4DD53E; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:53 -0400 Message-Id: <1665783491-13827-3-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 02/20] lustre: obdclass: set OBD_MD_FLGROUP for ladvise RPC X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Li Dongyang , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Li Dongyang ladvise RPC doesn't have OBD_MD_FLGROUP set, when RPC reaches server, tgt_validate_obdo() will corrupt the FID if it's seq is in FID_SEQ_NORMAL range. Do not mess with seq in obdo_to_ioobj() and tgt_validate_obdo(), since 2.0 all RPCs should have OBD_MD_FLGROUP set. Add OBD_MD_FLGROUP for ladvise RPC to fix new client talking to old servers. WC-bug-id: https://jira.whamcloud.com/browse/LU-16057 Lustre-commit: bee803c6e440ba6b5 ("LU-16057 obdclass: set OBD_MD_FLGROUP for ladvise RPC") Signed-off-by: Li Dongyang Reviewed-on: https://review.whamcloud.com/48080 Reviewed-by: Andreas Dilger Reviewed-by: Arshad Hussain Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/obdclass/obdo.c | 3 --- fs/lustre/osc/osc_io.c | 2 +- 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/fs/lustre/obdclass/obdo.c b/fs/lustre/obdclass/obdo.c index 7df4ff399890..9cafda16a95f 100644 --- a/fs/lustre/obdclass/obdo.c +++ b/fs/lustre/obdclass/obdo.c @@ -123,9 +123,6 @@ EXPORT_SYMBOL(obdo_from_inode); void obdo_to_ioobj(const struct obdo *oa, struct obd_ioobj *ioobj) { ioobj->ioo_oid = oa->o_oi; - if (unlikely(!(oa->o_valid & OBD_MD_FLGROUP))) - ostid_set_seq_mdt0(&ioobj->ioo_oid); - /* Since 2.4 this does not contain o_mode in the low 16 bits. * Instead, it holds (bd_md_max_brw - 1) for multi-bulk BRW RPCs */ diff --git a/fs/lustre/osc/osc_io.c b/fs/lustre/osc/osc_io.c index 655c7c68ab3a..4c9b3d2bb481 100644 --- a/fs/lustre/osc/osc_io.c +++ b/fs/lustre/osc/osc_io.c @@ -1036,7 +1036,7 @@ static int osc_io_ladvise_start(const struct lu_env *env, memset(oa, 0, sizeof(*oa)); oa->o_oi = loi->loi_oi; - oa->o_valid = OBD_MD_FLID; + oa->o_valid = OBD_MD_FLID | OBD_MD_FLGROUP; obdo_set_parent_fid(oa, lio->li_fid); ladvise = ladvise_hdr->lah_advise; From patchwork Fri Oct 14 21:37:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 73A29C433FE for ; Fri, 14 Oct 2022 21:38:32 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BH4XDvz21Gn; Fri, 14 Oct 2022 14:38:31 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B21CXTz1wc2 for ; Fri, 14 Oct 2022 14:38:18 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id EBDB71006F28; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id DEB47DD6C6; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:54 -0400 Message-Id: <1665783491-13827-4-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 03/20] lustre: obdclass: free inst_name correctly X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Emoly Liu In functon class_config_llog_handler(), inst_name should be freed correctly before break. WC-bug-id: https://jira.whamcloud.com/browse/LU-16154 Lustre-commit: e7f17c5e0c95dba3b ("LU-16154 obdclass: free inst_name correctly") Signed-off-by: Emoly Liu Reviewed-on: https://review.whamcloud.com/48542 Reviewed-by: Andreas Dilger Reviewed-by: Zhenyu Xu Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/obdclass/obd_config.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fs/lustre/obdclass/obd_config.c b/fs/lustre/obdclass/obd_config.c index 7d001ffaf121..2b24276880a6 100644 --- a/fs/lustre/obdclass/obd_config.c +++ b/fs/lustre/obdclass/obd_config.c @@ -1230,7 +1230,7 @@ int class_config_llog_handler(const struct lu_env *env, clli->cfg_flags); rc = 0; /* No processing! */ - break; + goto out_inst; } /* @@ -1352,7 +1352,7 @@ int class_config_llog_handler(const struct lu_env *env, lcfg_new = kzalloc(lcfg_len, GFP_NOFS); if (!lcfg_new) { rc = -ENOMEM; - goto out; + goto out_inst; } lustre_cfg_init(lcfg_new, lcfg->lcfg_command, &bufs); @@ -1379,6 +1379,7 @@ int class_config_llog_handler(const struct lu_env *env, rc = class_process_config(lcfg_new); kfree(lcfg_new); +out_inst: kfree(inst_name); break; } From patchwork Fri Oct 14 21:37:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1CC10C433FE for ; Fri, 14 Oct 2022 21:38:28 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0B95tDdz1y2t; Fri, 14 Oct 2022 14:38:25 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B1326wz1xJR for ; Fri, 14 Oct 2022 14:38:17 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id EB6A01006F21; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id E32F2DD6ED; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:55 -0400 Message-Id: <1665783491-13827-5-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 04/20] lustre: osc: take ldlm lock when queue sync pages X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Bobi Jam osc_queue_sync_pages() add osc_extent to osc_object's IO extent list without taking ldlm locks, and then it calls osc_io_unplug_async() to queue the IO work for the client. This patch make sync page queuing take ldlm lock in the osc_extent. WC-bug-id: https://jira.whamcloud.com/browse/LU-16160 Lustre-commit: 67aca1fcc6bed2079 ("LU-16160 osc: take ldlm lock when queue sync pages") Signed-off-by: Bobi Jam Reviewed-on: https://review.whamcloud.com/48557 Reviewed-by: Mikhail Pershin Reviewed-by: Alex Zhuravlev Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_osc.h | 3 +++ fs/lustre/mdc/mdc_dev.c | 3 +++ fs/lustre/osc/osc_cache.c | 7 +++++++ fs/lustre/osc/osc_io.c | 1 + fs/lustre/osc/osc_lock.c | 19 +++++++++++++++++++ 5 files changed, 33 insertions(+) diff --git a/fs/lustre/include/lustre_osc.h b/fs/lustre/include/lustre_osc.h index 89f02c51dbb3..323eeba3da42 100644 --- a/fs/lustre/include/lustre_osc.h +++ b/fs/lustre/include/lustre_osc.h @@ -157,6 +157,7 @@ struct osc_io { /* write osc_lock for this IO, used by osc_extent_find(). */ struct osc_lock *oi_write_osclock; + struct osc_lock *oi_read_osclock; struct obdo oi_oa; struct osc_async_cbargs { bool opc_rpc_sent; @@ -724,6 +725,8 @@ int osc_lock_enqueue_wait(const struct lu_env *env, struct osc_object *obj, struct osc_lock *oscl); void osc_lock_set_writer(const struct lu_env *env, const struct cl_io *io, struct cl_object *obj, struct osc_lock *oscl); +void osc_lock_set_reader(const struct lu_env *env, const struct cl_io *io, + struct cl_object *obj, struct osc_lock *oscl); int osc_lock_print(const struct lu_env *env, void *cookie, lu_printer_t p, const struct cl_lock_slice *slice); void osc_lock_cancel(const struct lu_env *env, diff --git a/fs/lustre/mdc/mdc_dev.c b/fs/lustre/mdc/mdc_dev.c index fd0e36225c13..2fd137d2fa4b 100644 --- a/fs/lustre/mdc/mdc_dev.c +++ b/fs/lustre/mdc/mdc_dev.c @@ -966,6 +966,9 @@ int mdc_lock_init(const struct lu_env *env, struct cl_object *obj, if (io->ci_type == CIT_WRITE || cl_io_is_mkwrite(io)) osc_lock_set_writer(env, io, obj, ols); + else if (io->ci_type == CIT_READ || + (io->ci_type == CIT_FAULT && !io->u.ci_fault.ft_mkwrite)) + osc_lock_set_reader(env, io, obj, ols); LDLM_DEBUG_NOLOCK("lock %p, mdc lock %p, flags %llx\n", lock, ols, ols->ols_flags); diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index b6f0cdb92bdc..36fec837d93e 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -2655,6 +2655,7 @@ int osc_queue_sync_pages(const struct lu_env *env, struct cl_io *io, struct osc_object *obj, struct list_head *list, int brw_flags) { + struct osc_io *oio = osc_env_io(env); struct client_obd *cli = osc_cli(obj); struct osc_extent *ext; struct osc_async_page *oap; @@ -2663,6 +2664,7 @@ int osc_queue_sync_pages(const struct lu_env *env, struct cl_io *io, bool can_merge = true; pgoff_t start = CL_PAGE_EOF; pgoff_t end = 0; + struct osc_lock *oscl; list_for_each_entry(oap, list, oap_pending_item) { struct osc_page *opg = oap2osc_page(oap); @@ -2703,6 +2705,11 @@ int osc_queue_sync_pages(const struct lu_env *env, struct cl_io *io, ext->oe_srvlock = !!(brw_flags & OBD_BRW_SRVLOCK); ext->oe_ndelay = !!(brw_flags & OBD_BRW_NDELAY); ext->oe_dio = !!(brw_flags & OBD_BRW_NOCACHE); + oscl = oio->oi_write_osclock ? : oio->oi_read_osclock; + if (oscl && oscl->ols_dlmlock != NULL) { + ext->oe_dlmlock = LDLM_LOCK_GET(oscl->ols_dlmlock); + lu_ref_add(&ext->oe_dlmlock->l_reference, "osc_extent", ext); + } if (ext->oe_dio && !ext->oe_rw) { /* direct io write */ int grants; int ppc; diff --git a/fs/lustre/osc/osc_io.c b/fs/lustre/osc/osc_io.c index 4c9b3d2bb481..aa8f61d710b9 100644 --- a/fs/lustre/osc/osc_io.c +++ b/fs/lustre/osc/osc_io.c @@ -461,6 +461,7 @@ void osc_io_rw_iter_fini(const struct lu_env *env, oio->oi_lru_reserved = 0; } oio->oi_write_osclock = NULL; + oio->oi_read_osclock = NULL; osc_io_iter_fini(env, ios); } diff --git a/fs/lustre/osc/osc_lock.c b/fs/lustre/osc/osc_lock.c index c8f85020c8fe..dd109496d260 100644 --- a/fs/lustre/osc/osc_lock.c +++ b/fs/lustre/osc/osc_lock.c @@ -1178,6 +1178,22 @@ void osc_lock_set_writer(const struct lu_env *env, const struct cl_io *io, } EXPORT_SYMBOL(osc_lock_set_writer); +void osc_lock_set_reader(const struct lu_env *env, const struct cl_io *io, + struct cl_object *obj, struct osc_lock *oscl) +{ + struct osc_io *oio = osc_env_io(env); + + if (!cl_object_same(io->ci_obj, obj)) + return; + + if (oscl->ols_glimpse || osc_lock_is_lockless(oscl)) + return; + + if (oio->oi_read_osclock == NULL) + oio->oi_read_osclock = oscl; +} +EXPORT_SYMBOL(osc_lock_set_reader); + int osc_lock_init(const struct lu_env *env, struct cl_object *obj, struct cl_lock *lock, const struct cl_io *io) @@ -1224,6 +1240,9 @@ int osc_lock_init(const struct lu_env *env, if (io->ci_type == CIT_WRITE || cl_io_is_mkwrite(io)) osc_lock_set_writer(env, io, obj, oscl); + else if (io->ci_type == CIT_READ || + (io->ci_type == CIT_FAULT && !io->u.ci_fault.ft_mkwrite)) + osc_lock_set_reader(env, io, obj, oscl); LDLM_DEBUG_NOLOCK("lock %p, osc lock %p, flags %llx", From patchwork Fri Oct 14 21:37:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64D8FC433FE for ; Fri, 14 Oct 2022 21:38:39 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BR0S4Hz21H8; Fri, 14 Oct 2022 14:38:39 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B33njJz1y2t for ; Fri, 14 Oct 2022 14:38:19 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id F349A1006F31; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id E71B1DEE01; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:56 -0400 Message-Id: <1665783491-13827-6-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 05/20] lnet: track pinginfo size in bytes, not nis. X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mr NeilBrown When we extend the pinginfo to be able to store large-address nids, there could be nids of different sizes in it. So using the number of nis to track the size won't work. So change to using the number of bytes. i.e. the total size of the 'struct lnet_ping_info'. This affects pb_nnis in the ping_buffer, and the global ln_push_target_nnis. LNET_PING_INFO_SIZE is removed as size won't depend on number of nids any more. When determining the number of bytes expected in a received ping_info, use a new macro lnet_ping_info_size() which can extract information as required from the ping_info. Note that lnet_ping_target_create() now initializes pi_nis to 0. Setting the initial size doesn't seem to be useful. WC-bug-id: https://jira.whamcloud.com/browse/LU-10391 Lustre-commit: 941218e09e1d6bb9b ("LU-10391 lnet: track pinginfo size in bytes, not nis.") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/44627 Reviewed-by: James Simmons Reviewed-by: Serguei Smirnov Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- include/linux/lnet/lib-lnet.h | 6 +- include/linux/lnet/lib-types.h | 13 ++- include/uapi/linux/lnet/lnet-idl.h | 8 +- net/lnet/lnet/api-ni.c | 180 ++++++++++++++++------------- net/lnet/lnet/lib-move.c | 10 +- net/lnet/lnet/lib-msg.c | 14 +-- net/lnet/lnet/peer.c | 58 +++++----- 7 files changed, 157 insertions(+), 132 deletions(-) diff --git a/include/linux/lnet/lib-lnet.h b/include/linux/lnet/lib-lnet.h index fc086dab080e..a95919e69802 100644 --- a/include/linux/lnet/lib-lnet.h +++ b/include/linux/lnet/lib-lnet.h @@ -646,7 +646,7 @@ void lnet_prep_send(struct lnet_msg *msg, int type, int lnet_send(struct lnet_nid *nid, struct lnet_msg *msg, struct lnet_nid *rtr_nid); int lnet_send_ping(struct lnet_nid *dest_nid, struct lnet_handle_md *mdh, - int nnis, void *user_ptr, lnet_handler_t handler, + int bytes, void *user_ptr, lnet_handler_t handler, bool recovery); void lnet_return_tx_credits_locked(struct lnet_msg *msg); void lnet_return_rx_credits_locked(struct lnet_msg *msg); @@ -860,7 +860,7 @@ void lnet_wait_router_start(void); void lnet_swap_pinginfo(struct lnet_ping_buffer *pbuf); int lnet_ping_info_validate(struct lnet_ping_info *pinfo); -struct lnet_ping_buffer *lnet_ping_buffer_alloc(int nnis, gfp_t gfp); +struct lnet_ping_buffer *lnet_ping_buffer_alloc(int bytes, gfp_t gfp); void lnet_ping_buffer_free(struct lnet_ping_buffer *pbuf); static inline void lnet_ping_buffer_addref(struct lnet_ping_buffer *pbuf) @@ -878,7 +878,7 @@ static inline void lnet_ping_buffer_decref(struct lnet_ping_buffer *pbuf) static inline int lnet_push_target_resize_needed(void) { - return the_lnet.ln_push_target->pb_nnis < the_lnet.ln_push_target_nnis; + return the_lnet.ln_push_target->pb_nbytes < the_lnet.ln_push_target_nbytes; } int lnet_push_target_resize(void); diff --git a/include/linux/lnet/lib-types.h b/include/linux/lnet/lib-types.h index 2266d1be16a6..499385bb981b 100644 --- a/include/linux/lnet/lib-types.h +++ b/include/linux/lnet/lib-types.h @@ -567,14 +567,14 @@ struct lnet_ni { * area that may be overwritten by network data. */ struct lnet_ping_buffer { - int pb_nnis; + int pb_nbytes; /* sizeof pb_info */ atomic_t pb_refcnt; bool pb_needs_post; struct lnet_ping_info pb_info; }; -#define LNET_PING_BUFFER_SIZE(NNIDS) \ - offsetof(struct lnet_ping_buffer, pb_info.pi_ni[NNIDS]) +#define LNET_PING_BUFFER_SIZE(bytes) \ + (offsetof(struct lnet_ping_buffer, pb_info) + bytes) #define LNET_PING_BUFFER_LONI(PBUF) ((PBUF)->pb_info.pi_ni[0].ns_nid) #define LNET_PING_BUFFER_SEQNO(PBUF) ((PBUF)->pb_info.pi_ni[0].ns_status) @@ -733,8 +733,8 @@ struct lnet_peer { /* MD handle for push in progress */ struct lnet_handle_md lp_push_mdh; - /* number of NIDs for sizing push data */ - int lp_data_nnis; + /* number of bytes for sizing pb_info in push data */ + int lp_data_bytes; /* NI config sequence number of peer */ u32 lp_peer_seqno; @@ -1255,7 +1255,8 @@ struct lnet { lnet_handler_t ln_push_target_handler; struct lnet_handle_md ln_push_target_md; struct lnet_ping_buffer *ln_push_target; - int ln_push_target_nnis; + /* bytes needed for pb_info to receive push */ + int ln_push_target_nbytes; /* discovery event queue handle */ lnet_handler_t ln_dc_handler; diff --git a/include/uapi/linux/lnet/lnet-idl.h b/include/uapi/linux/lnet/lnet-idl.h index 74036e7ef406..41bbb404af6c 100644 --- a/include/uapi/linux/lnet/lnet-idl.h +++ b/include/uapi/linux/lnet/lnet-idl.h @@ -291,9 +291,13 @@ struct lnet_ping_info { struct lnet_ni_status pi_ni[0]; } __attribute__((packed)); -#define LNET_PING_INFO_SIZE(NNIDS) \ - offsetof(struct lnet_ping_info, pi_ni[NNIDS]) +#define LNET_PING_INFO_HDR_SIZE \ + offsetof(struct lnet_ping_info, pi_ni[0]) +#define LNET_PING_INFO_MIN_SIZE \ + offsetof(struct lnet_ping_info, pi_ni[LNET_INTERFACES_MIN]) #define LNET_PING_INFO_LONI(PINFO) ((PINFO)->pi_ni[0].ns_nid) #define LNET_PING_INFO_SEQNO(PINFO) ((PINFO)->pi_ni[0].ns_status) +#define lnet_ping_info_size(pinfo) \ + offsetof(struct lnet_ping_info, pi_ni[(pinfo)->pi_nnis]) #endif diff --git a/net/lnet/lnet/api-ni.c b/net/lnet/lnet/api-ni.c index 89c7b99e45be..9459fc0f103f 100644 --- a/net/lnet/lnet/api-ni.c +++ b/net/lnet/lnet/api-ni.c @@ -1720,13 +1720,13 @@ lnet_count_acceptor_nets(void) } struct lnet_ping_buffer * -lnet_ping_buffer_alloc(int nnis, gfp_t gfp) +lnet_ping_buffer_alloc(int nbytes, gfp_t gfp) { struct lnet_ping_buffer *pbuf; - pbuf = kmalloc(LNET_PING_BUFFER_SIZE(nnis), gfp); + pbuf = kmalloc(LNET_PING_BUFFER_SIZE(nbytes), gfp); if (pbuf) { - pbuf->pb_nnis = nnis; + pbuf->pb_nbytes = nbytes; /* sizeof of pb_info */ pbuf->pb_needs_post = false; atomic_set(&pbuf->pb_refcnt, 1); } @@ -1742,17 +1742,17 @@ lnet_ping_buffer_free(struct lnet_ping_buffer *pbuf) } static struct lnet_ping_buffer * -lnet_ping_target_create(int nnis) +lnet_ping_target_create(int nbytes) { struct lnet_ping_buffer *pbuf; - pbuf = lnet_ping_buffer_alloc(nnis, GFP_KERNEL); + pbuf = lnet_ping_buffer_alloc(nbytes, GFP_KERNEL); if (!pbuf) { - CERROR("Can't allocate ping source [%d]\n", nnis); + CERROR("Can't allocate ping source [%d]\n", nbytes); return NULL; } - pbuf->pb_info.pi_nnis = nnis; + pbuf->pb_info.pi_nnis = 0; pbuf->pb_info.pi_pid = the_lnet.ln_pid; pbuf->pb_info.pi_magic = LNET_PROTO_PING_MAGIC; pbuf->pb_info.pi_features = @@ -1762,52 +1762,56 @@ lnet_ping_target_create(int nnis) } static inline int -lnet_get_net_ni_count_locked(struct lnet_net *net) +lnet_get_net_ni_bytes_locked(struct lnet_net *net) { struct lnet_ni *ni; - int count = 0; + int bytes = 0; list_for_each_entry(ni, &net->net_ni_list, ni_netlist) - count++; + if (nid_is_nid4(&ni->ni_nid)) + bytes += sizeof(struct lnet_ni_status); - return count; + return bytes; } static inline int -lnet_get_net_ni_count_pre(struct lnet_net *net) +lnet_get_net_ni_bytes_pre(struct lnet_net *net) { struct lnet_ni *ni; - int count = 0; + int bytes = 0; list_for_each_entry(ni, &net->net_ni_added, ni_netlist) - count++; + if (nid_is_nid4(&ni->ni_nid)) + bytes += sizeof(struct lnet_ni_status); - return count; + return bytes; } static inline int -lnet_get_ni_count(void) +lnet_get_ni_bytes(void) { struct lnet_ni *ni; struct lnet_net *net; - int count = 0; + int bytes = 0; lnet_net_lock(0); list_for_each_entry(net, &the_lnet.ln_nets, net_list) { list_for_each_entry(ni, &net->net_ni_list, ni_netlist) - count++; + if (nid_is_nid4(&ni->ni_nid)) + bytes += sizeof(struct lnet_ni_status); + } lnet_net_unlock(0); - return count; + return bytes; } void lnet_swap_pinginfo(struct lnet_ping_buffer *pbuf) { - struct lnet_ni_status *stat; + struct lnet_ni_status *stat, *end; int nnis; int i; @@ -1816,10 +1820,9 @@ lnet_swap_pinginfo(struct lnet_ping_buffer *pbuf) __swab32s(&pbuf->pb_info.pi_pid); __swab32s(&pbuf->pb_info.pi_nnis); nnis = pbuf->pb_info.pi_nnis; - if (nnis > pbuf->pb_nnis) - nnis = pbuf->pb_nnis; - for (i = 0; i < nnis; i++) { - stat = &pbuf->pb_info.pi_ni[i]; + stat = &pbuf->pb_info.pi_ni[0]; + end = (void *)&pbuf->pb_info + pbuf->pb_nbytes; + for (i = 0; i < nnis && stat + 1 <= end; i++, stat++) { __swab64s(&stat->ns_nid); __swab32s(&stat->ns_status); } @@ -1876,7 +1879,7 @@ lnet_ping_target_event_handler(struct lnet_event *event) static int lnet_ping_target_setup(struct lnet_ping_buffer **ppbuf, struct lnet_handle_md *ping_mdh, - int ni_count, bool set_eq) + int ni_bytes, bool set_eq) { struct lnet_processid id = { .nid = LNET_ANY_NID, @@ -1890,7 +1893,7 @@ lnet_ping_target_setup(struct lnet_ping_buffer **ppbuf, the_lnet.ln_ping_target_handler = lnet_ping_target_event_handler; - *ppbuf = lnet_ping_target_create(ni_count); + *ppbuf = lnet_ping_target_create(ni_bytes); if (!*ppbuf) { rc = -ENOMEM; goto fail_free_eq; @@ -1908,7 +1911,7 @@ lnet_ping_target_setup(struct lnet_ping_buffer **ppbuf, /* initialize md content */ md.start = &(*ppbuf)->pb_info; - md.length = LNET_PING_INFO_SIZE((*ppbuf)->pb_nnis); + md.length = (*ppbuf)->pb_nbytes; md.threshold = LNET_MD_THRESH_INF; md.max_size = 0; md.options = LNET_MD_OP_GET | LNET_MD_TRUNCATE | @@ -1949,20 +1952,19 @@ lnet_ping_md_unlink(struct lnet_ping_buffer *pbuf, static void lnet_ping_target_install_locked(struct lnet_ping_buffer *pbuf) { - struct lnet_ni_status *ns; + struct lnet_ni_status *ns, *end; struct lnet_ni *ni; struct lnet_net *net; - int i = 0; int rc; + pbuf->pb_info.pi_nnis = 0; + ns = &pbuf->pb_info.pi_ni[0]; + end = (void *)&pbuf->pb_info + pbuf->pb_nbytes; list_for_each_entry(net, &the_lnet.ln_nets, net_list) { list_for_each_entry(ni, &net->net_ni_list, ni_netlist) { - LASSERT(i < pbuf->pb_nnis); - - ns = &pbuf->pb_info.pi_ni[i]; - if (!nid_is_nid4(&ni->ni_nid)) continue; + LASSERT(ns + 1 <= end); ns->ns_nid = lnet_nid_to_nid4(&ni->ni_nid); lnet_ni_lock(ni); @@ -1970,11 +1972,12 @@ lnet_ping_target_install_locked(struct lnet_ping_buffer *pbuf) ni->ni_status = &ns->ns_status; lnet_ni_unlock(ni); - i++; + pbuf->pb_info.pi_nnis++; + ns++; } } - /* - * We (ab)use the ns_status of the loopback interface to + + /* We (ab)use the ns_status of the loopback interface to * transmit the sequence number. The first interface listed * must be the loopback interface. */ @@ -2043,13 +2046,13 @@ int lnet_push_target_resize(void) struct lnet_handle_md old_mdh; struct lnet_ping_buffer *pbuf; struct lnet_ping_buffer *old_pbuf; - int nnis; + int nbytes; int rc; again: - nnis = the_lnet.ln_push_target_nnis; - if (nnis <= 0) { - CDEBUG(D_NET, "Invalid nnis %d\n", nnis); + nbytes = the_lnet.ln_push_target_nbytes; + if (nbytes <= 0) { + CDEBUG(D_NET, "Invalid nbytes %d\n", nbytes); return -EINVAL; } @@ -2057,9 +2060,9 @@ int lnet_push_target_resize(void) * dropped when we need to resize again (see "old_pbuf" below) or when * LNet is shutdown (see lnet_push_target_fini()) */ - pbuf = lnet_ping_buffer_alloc(nnis, GFP_NOFS); + pbuf = lnet_ping_buffer_alloc(nbytes, GFP_NOFS); if (!pbuf) { - CDEBUG(D_NET, "Can't allocate pbuf for nnis %d\n", nnis); + CDEBUG(D_NET, "Can't allocate pbuf for nbytes %d\n", nbytes); return -ENOMEM; } @@ -2084,10 +2087,10 @@ int lnet_push_target_resize(void) } /* Received another push or reply that requires a larger buffer */ - if (nnis < the_lnet.ln_push_target_nnis) + if (nbytes < the_lnet.ln_push_target_nbytes) goto again; - CDEBUG(D_NET, "nnis %d success\n", nnis); + CDEBUG(D_NET, "nbytes %d success\n", nbytes); return 0; } @@ -2118,7 +2121,7 @@ int lnet_push_target_post(struct lnet_ping_buffer *pbuf, /* initialize md content */ md.start = &pbuf->pb_info; - md.length = LNET_PING_INFO_SIZE(pbuf->pb_nnis); + md.length = pbuf->pb_nbytes; md.threshold = 1; md.max_size = 0; md.options = LNET_MD_OP_PUT | LNET_MD_TRUNCATE; @@ -2175,7 +2178,7 @@ static int lnet_push_target_init(void) LASSERT(rc == 0); /* Start at the required minimum, we'll enlarge if required. */ - the_lnet.ln_push_target_nnis = LNET_INTERFACES_MIN; + the_lnet.ln_push_target_nbytes = LNET_PING_INFO_MIN_SIZE; rc = lnet_push_target_resize(); if (rc) { @@ -2204,7 +2207,7 @@ static void lnet_push_target_fini(void) /* Drop ref set by lnet_ping_buffer_alloc() */ lnet_ping_buffer_decref(the_lnet.ln_push_target); the_lnet.ln_push_target = NULL; - the_lnet.ln_push_target_nnis = 0; + the_lnet.ln_push_target_nbytes = 0; LNetClearLazyPortal(LNET_RESERVED_PORTAL); lnet_assert_handler_unused(the_lnet.ln_push_target_handler); @@ -2865,7 +2868,7 @@ LNetNIInit(lnet_pid_t requested_pid) { int im_a_router = 0; int rc; - int ni_count; + int ni_bytes; struct lnet_ping_buffer *pbuf; struct lnet_handle_md ping_mdh; LIST_HEAD(net_head); @@ -2921,11 +2924,9 @@ LNetNIInit(lnet_pid_t requested_pid) goto err_empty_list; } - ni_count = lnet_startup_lndnets(&net_head); - if (ni_count < 0) { - rc = ni_count; + rc = lnet_startup_lndnets(&net_head); + if (rc < 0) goto err_empty_list; - } if (!the_lnet.ln_nis_from_mod_params) { rc = lnet_parse_routes(lnet_get_routes(), &im_a_router); @@ -2944,7 +2945,11 @@ LNetNIInit(lnet_pid_t requested_pid) the_lnet.ln_refcount = 1; /* Now I may use my own API functions... */ - rc = lnet_ping_target_setup(&pbuf, &ping_mdh, ni_count, true); + ni_bytes = LNET_PING_INFO_HDR_SIZE; + list_for_each_entry(net, &the_lnet.ln_nets, net_list) + ni_bytes += lnet_get_net_ni_bytes_locked(net); + + rc = lnet_ping_target_setup(&pbuf, &ping_mdh, ni_bytes, true); if (rc) goto err_acceptor_stop; @@ -3363,7 +3368,7 @@ static int lnet_add_net_common(struct lnet_net *net, struct lnet_ping_buffer *pbuf; struct lnet_remotenet *rnet; struct lnet_ni *ni; - int net_ni_count; + int net_ni_bytes; u32 net_id; int rc; @@ -3388,12 +3393,13 @@ static int lnet_add_net_common(struct lnet_net *net, * which will be added. * * since ni hasn't been configured yet, use - * lnet_get_net_ni_count_pre() which checks the net_ni_added list + * lnet_get_net_ni_bytes_pre() which checks the net_ni_added list */ - net_ni_count = lnet_get_net_ni_count_pre(net); + net_ni_bytes = lnet_get_net_ni_bytes_pre(net); rc = lnet_ping_target_setup(&pbuf, &ping_mdh, - net_ni_count + lnet_get_ni_count(), + LNET_PING_INFO_HDR_SIZE + + net_ni_bytes + lnet_get_ni_bytes(), false); if (rc < 0) { lnet_net_free(net); @@ -3589,8 +3595,8 @@ int lnet_dyn_del_ni(struct lnet_ioctl_config_ni *conf) u32 net_id = LNET_NIDNET(conf->lic_nid); struct lnet_ping_buffer *pbuf; struct lnet_handle_md ping_mdh; - int rc; - int net_count; + int net_bytes, rc; + bool net_empty; u32 addr; /* don't allow userspace to shutdown the LOLND */ @@ -3616,13 +3622,13 @@ int lnet_dyn_del_ni(struct lnet_ioctl_config_ni *conf) addr = LNET_NIDADDR(conf->lic_nid); if (addr == 0) { /* remove the entire net */ - net_count = lnet_get_net_ni_count_locked(net); + net_bytes = lnet_get_net_ni_bytes_locked(net); lnet_net_unlock(0); /* create and link a new ping info, before removing the old one */ rc = lnet_ping_target_setup(&pbuf, &ping_mdh, - lnet_get_ni_count() - net_count, + lnet_get_ni_bytes() - net_bytes, false); if (rc != 0) goto unlock_api_mutex; @@ -3644,13 +3650,17 @@ int lnet_dyn_del_ni(struct lnet_ioctl_config_ni *conf) goto unlock_net; } - net_count = lnet_get_net_ni_count_locked(net); + net_bytes = lnet_get_net_ni_bytes_locked(net); + net_empty = list_is_singular(&net->net_ni_list); lnet_net_unlock(0); /* create and link a new ping info, before removing the old one */ rc = lnet_ping_target_setup(&pbuf, &ping_mdh, - lnet_get_ni_count() - 1, false); + (LNET_PING_INFO_HDR_SIZE + + lnet_get_ni_bytes() - + sizeof(pbuf->pb_info.pi_ni[0])), + false); if (rc != 0) goto unlock_api_mutex; @@ -3661,7 +3671,7 @@ int lnet_dyn_del_ni(struct lnet_ioctl_config_ni *conf) lnet_ping_target_update(pbuf, ping_mdh); /* check if the net is empty and remove it if it is */ - if (net_count == 1) + if (net_empty) lnet_shutdown_lndnet(net); goto unlock_api_mutex; @@ -3744,8 +3754,7 @@ lnet_dyn_del_net(u32 net_id) struct lnet_net *net; struct lnet_ping_buffer *pbuf; struct lnet_handle_md ping_mdh; - int rc; - int net_ni_count; + int net_ni_bytes, rc; /* don't allow userspace to shutdown the LOLND */ if (LNET_NETTYP(net_id) == LOLND) @@ -3766,13 +3775,15 @@ lnet_dyn_del_net(u32 net_id) goto out; } - net_ni_count = lnet_get_net_ni_count_locked(net); + net_ni_bytes = lnet_get_net_ni_bytes_locked(net); lnet_net_unlock(0); /* create and link a new ping info, before removing the old one */ rc = lnet_ping_target_setup(&pbuf, &ping_mdh, - lnet_get_ni_count() - net_ni_count, false); + LNET_PING_INFO_HDR_SIZE + + lnet_get_ni_bytes() - net_ni_bytes, + false); if (rc) goto out; @@ -4626,6 +4637,12 @@ lnet_ping_event_handler(struct lnet_event *event) complete(&pd->completion); } +/* lnet_ping() only works with nid4 nids, so we can calculate + * size from number of nids + */ +#define LNET_PING_INFO_SIZE(NNIDS) \ + offsetof(struct lnet_ping_info, pi_ni[NNIDS]) + static int lnet_ping(struct lnet_process_id id4, struct lnet_nid *src_nid, signed long timeout, struct lnet_process_id __user *ids, int n_ids) @@ -4635,6 +4652,7 @@ static int lnet_ping(struct lnet_process_id id4, struct lnet_nid *src_nid, struct lnet_ping_buffer *pbuf; struct lnet_process_id tmpid; struct lnet_processid id; + int id_bytes; int i; int nob; int rc; @@ -4653,13 +4671,14 @@ static int lnet_ping(struct lnet_process_id id4, struct lnet_nid *src_nid, if (id4.pid == LNET_PID_ANY) id4.pid = LNET_PID_LUSTRE; - pbuf = lnet_ping_buffer_alloc(n_ids, GFP_NOFS); + id_bytes = LNET_PING_INFO_SIZE(n_ids); + pbuf = lnet_ping_buffer_alloc(id_bytes, GFP_NOFS); if (!pbuf) return -ENOMEM; /* initialize md content */ md.start = &pbuf->pb_info; - md.length = LNET_PING_INFO_SIZE(n_ids); + md.length = id_bytes; md.threshold = 2; /* GET/REPLY */ md.max_size = 0; md.options = LNET_MD_TRUNCATE; @@ -4696,7 +4715,7 @@ static int lnet_ping(struct lnet_process_id id4, struct lnet_nid *src_nid, } nob = pd.rc; - LASSERT(nob >= 0 && nob <= LNET_PING_INFO_SIZE(n_ids)); + LASSERT(nob >= 0 && nob <= id_bytes); rc = -EPROTO; /* if I can't parse... */ @@ -4720,20 +4739,21 @@ static int lnet_ping(struct lnet_process_id id4, struct lnet_nid *src_nid, goto fail_ping_buffer_decref; } - if (nob < LNET_PING_INFO_SIZE(0)) { - CERROR("%s: Short reply %d(%d min)\n", - libcfs_idstr(&id), - nob, (int)LNET_PING_INFO_SIZE(0)); + /* Test if smaller than lnet_pinginfo with no pi_ni status info */ + if (nob < LNET_PING_INFO_HDR_SIZE) { + CERROR("%s: Short reply %d(%lu min)\n", + libcfs_idstr(&id), nob, LNET_PING_INFO_HDR_SIZE); goto fail_ping_buffer_decref; } - if (pbuf->pb_info.pi_nnis < n_ids) + if (pbuf->pb_info.pi_nnis < n_ids) { n_ids = pbuf->pb_info.pi_nnis; + id_bytes = lnet_ping_info_size(&pbuf->pb_info); + } - if (nob < LNET_PING_INFO_SIZE(n_ids)) { + if (nob < id_bytes) { CERROR("%s: Short reply %d(%d expected)\n", - libcfs_idstr(&id), - nob, (int)LNET_PING_INFO_SIZE(n_ids)); + libcfs_idstr(&id), nob, id_bytes); goto fail_ping_buffer_decref; } @@ -4753,6 +4773,8 @@ static int lnet_ping(struct lnet_process_id id4, struct lnet_nid *src_nid, return rc; } +#undef LNET_PING_INFO_SIZE + static int lnet_discover(struct lnet_process_id id4, u32 force, struct lnet_process_id __user *ids, diff --git a/net/lnet/lnet/lib-move.c b/net/lnet/lnet/lib-move.c index a8a5ddbab84a..d46578929d08 100644 --- a/net/lnet/lnet/lib-move.c +++ b/net/lnet/lnet/lib-move.c @@ -3334,7 +3334,7 @@ lnet_recover_local_nis(void) ev_info->mt_type = MT_TYPE_LOCAL_NI; ev_info->mt_nid = nid; - rc = lnet_send_ping(&nid, &mdh, LNET_INTERFACES_MIN, + rc = lnet_send_ping(&nid, &mdh, LNET_PING_INFO_MIN_SIZE, ev_info, the_lnet.ln_mt_handler, true); /* lookup the nid again */ @@ -3563,7 +3563,7 @@ lnet_recover_peer_nis(void) ev_info->mt_type = MT_TYPE_PEER_NI; ev_info->mt_nid = nid; - rc = lnet_send_ping(&nid, &mdh, LNET_INTERFACES_MIN, + rc = lnet_send_ping(&nid, &mdh, LNET_PING_INFO_MIN_SIZE, ev_info, the_lnet.ln_mt_handler, true); lnet_net_lock(0); @@ -3672,7 +3672,7 @@ lnet_monitor_thread(void *arg) */ int lnet_send_ping(struct lnet_nid *dest_nid, - struct lnet_handle_md *mdh, int nnis, + struct lnet_handle_md *mdh, int bytes, void *user_data, lnet_handler_t handler, bool recovery) { struct lnet_md md = { NULL }; @@ -3685,7 +3685,7 @@ lnet_send_ping(struct lnet_nid *dest_nid, goto fail_error; } - pbuf = lnet_ping_buffer_alloc(nnis, GFP_NOFS); + pbuf = lnet_ping_buffer_alloc(bytes, GFP_NOFS); if (!pbuf) { rc = ENOMEM; goto fail_error; @@ -3693,7 +3693,7 @@ lnet_send_ping(struct lnet_nid *dest_nid, /* initialize md content */ md.start = &pbuf->pb_info; - md.length = LNET_PING_INFO_SIZE(nnis); + md.length = bytes; md.threshold = 2; /* GET/REPLY */ md.max_size = 0; md.options = LNET_MD_TRUNCATE | LNET_MD_TRACK_RESPONSE; diff --git a/net/lnet/lnet/lib-msg.c b/net/lnet/lnet/lib-msg.c index 3b1f6a36bfe0..9fb001e5815e 100644 --- a/net/lnet/lnet/lib-msg.c +++ b/net/lnet/lnet/lib-msg.c @@ -814,6 +814,8 @@ lnet_health_check(struct lnet_msg *msg) * messages with a health status != OK. */ if (hstatus != LNET_MSG_STATUS_OK) { + struct lnet_ping_info *pi; + /* Don't further decrement the health value if a recovery * message failed. */ @@ -826,11 +828,10 @@ lnet_health_check(struct lnet_msg *msg) } /* For local failures, health/recovery/resends are not needed if - * I only have a single (non-lolnd) interface. NB: pb_nnis - * includes the lolnd interface, so a single-rail node would - * have pb_nnis == 2. + * I only have a single (non-lolnd) interface. */ - if (the_lnet.ln_ping_target->pb_nnis <= 2) { + pi = &the_lnet.ln_ping_target->pb_info; + if (pi->pi_nnis <= 2) { handle_local_health = false; attempt_local_resend = false; } @@ -840,9 +841,8 @@ lnet_health_check(struct lnet_msg *msg) /* For remote failures, health/recovery/resends are not needed * if the peer only has a single interface. Special case for * routers where we rely on health feature to manage route - * aliveness. NB: unlike pb_nnis above, lp_nnis does _not_ - * include the lolnd, so a single-rail node would have - * lp_nnis == 1. + * aliveness. NB: lp_nnis does _not_ include the lolnd, so a + * single-rail node would have lp_nnis == 1. */ if (lpni && lpni->lpni_peer_net && lpni->lpni_peer_net->lpn_peer && diff --git a/net/lnet/lnet/peer.c b/net/lnet/lnet/peer.c index e7c3c835b528..9b2066028509 100644 --- a/net/lnet/lnet/peer.c +++ b/net/lnet/lnet/peer.c @@ -2254,6 +2254,7 @@ void lnet_peer_push_event(struct lnet_event *ev) { struct lnet_ping_buffer *pbuf; struct lnet_peer *lp; + int infobytes; pbuf = LNET_PING_INFO_TO_BUFFER(ev->md_start + ev->offset); @@ -2298,12 +2299,12 @@ void lnet_peer_push_event(struct lnet_event *ev) goto out; } - /* - * Make sure we'll allocate the correct size ping buffer when + /* Make sure we'll allocate the correct size ping buffer when * pinging the peer. */ - if (lp->lp_data_nnis < pbuf->pb_info.pi_nnis) - lp->lp_data_nnis = pbuf->pb_info.pi_nnis; + infobytes = lnet_ping_info_size(&pbuf->pb_info); + if (lp->lp_data_bytes < infobytes) + lp->lp_data_bytes = infobytes; /* * A non-Multi-Rail peer is not supposed to be capable of @@ -2369,13 +2370,12 @@ void lnet_peer_push_event(struct lnet_event *ev) * and tell discovery to allocate a bigger buffer. */ if (ev->mlength < ev->rlength) { - if (the_lnet.ln_push_target_nnis < pbuf->pb_info.pi_nnis) - the_lnet.ln_push_target_nnis = pbuf->pb_info.pi_nnis; + if (the_lnet.ln_push_target_nbytes < infobytes) + the_lnet.ln_push_target_nbytes = infobytes; lp->lp_state &= ~LNET_PEER_NIDS_UPTODATE; lp->lp_state |= LNET_PEER_FORCE_PING; - CDEBUG(D_NET, "Truncated Push from %s (%d nids)\n", - libcfs_nidstr(&lp->lp_primary_nid), - pbuf->pb_info.pi_nnis); + CDEBUG(D_NET, "Truncated Push from %s (%d bytes)\n", + libcfs_nidstr(&lp->lp_primary_nid), infobytes); goto out; } @@ -2383,8 +2383,7 @@ void lnet_peer_push_event(struct lnet_event *ev) lp->lp_peer_seqno = LNET_PING_BUFFER_SEQNO(pbuf); lp->lp_state &= ~LNET_PEER_NIDS_UPTODATE; - /* - * If there is data present that hasn't been processed yet, + /* If there is data present that hasn't been processed yet, * we'll replace it if the Put contained newer data and it * fits. We're racing with a Ping or earlier Push in this * case. @@ -2392,9 +2391,9 @@ void lnet_peer_push_event(struct lnet_event *ev) if (lp->lp_state & LNET_PEER_DATA_PRESENT) { if (LNET_PING_BUFFER_SEQNO(pbuf) > LNET_PING_BUFFER_SEQNO(lp->lp_data) && - pbuf->pb_info.pi_nnis <= lp->lp_data->pb_nnis) { + infobytes <= lp->lp_data->pb_nbytes) { memcpy(&lp->lp_data->pb_info, &pbuf->pb_info, - LNET_PING_INFO_SIZE(pbuf->pb_info.pi_nnis)); + infobytes); CDEBUG(D_NET, "Ping/Push race from %s: %u vs %u\n", libcfs_nidstr(&lp->lp_primary_nid), LNET_PING_BUFFER_SEQNO(pbuf), @@ -2408,7 +2407,7 @@ void lnet_peer_push_event(struct lnet_event *ev) * the Push and set FORCE_PING to force the discovery * thread to fix the problem by pinging the peer. */ - lp->lp_data = lnet_ping_buffer_alloc(lp->lp_data_nnis, GFP_ATOMIC); + lp->lp_data = lnet_ping_buffer_alloc(lp->lp_data_bytes, GFP_ATOMIC); if (!lp->lp_data) { lp->lp_state |= LNET_PEER_FORCE_PING; CDEBUG(D_NET, "Cannot allocate Push buffer for %s %u\n", @@ -2418,8 +2417,7 @@ void lnet_peer_push_event(struct lnet_event *ev) } /* Success */ - memcpy(&lp->lp_data->pb_info, &pbuf->pb_info, - LNET_PING_INFO_SIZE(pbuf->pb_info.pi_nnis)); + memcpy(&lp->lp_data->pb_info, &pbuf->pb_info, infobytes); lp->lp_state |= LNET_PEER_DATA_PRESENT; CDEBUG(D_NET, "Received Push %s %u\n", libcfs_nidstr(&lp->lp_primary_nid), @@ -2580,6 +2578,7 @@ static void lnet_discovery_event_reply(struct lnet_peer *lp, struct lnet_event *ev) { struct lnet_ping_buffer *pbuf; + int infobytes; int rc; spin_lock(&lp->lp_lock); @@ -2692,25 +2691,24 @@ lnet_discovery_event_reply(struct lnet_peer *lp, struct lnet_event *ev) } } + infobytes = lnet_ping_info_size(&pbuf->pb_info); /* * Make sure we'll allocate the correct size ping buffer when * pinging the peer. */ - if (lp->lp_data_nnis < pbuf->pb_info.pi_nnis) - lp->lp_data_nnis = pbuf->pb_info.pi_nnis; + if (lp->lp_data_bytes < infobytes) + lp->lp_data_bytes = infobytes; - /* - * Check for truncation of the Reply. Clear PING_SENT and set + /* Check for truncation of the Reply. Clear PING_SENT and set * PING_FAILED to trigger a retry. */ - if (pbuf->pb_nnis < pbuf->pb_info.pi_nnis) { - if (the_lnet.ln_push_target_nnis < pbuf->pb_info.pi_nnis) - the_lnet.ln_push_target_nnis = pbuf->pb_info.pi_nnis; + if (pbuf->pb_nbytes < infobytes) { + if (the_lnet.ln_push_target_nbytes < infobytes) + the_lnet.ln_push_target_nbytes = infobytes; lp->lp_state |= LNET_PEER_PING_FAILED; lp->lp_ping_error = 0; - CDEBUG(D_NET, "Truncated Reply from %s (%d nids)\n", - libcfs_nidstr(&lp->lp_primary_nid), - pbuf->pb_info.pi_nnis); + CDEBUG(D_NET, "Truncated Reply from %s (%d bytes)\n", + libcfs_nidstr(&lp->lp_primary_nid), infobytes); goto out; } @@ -3391,7 +3389,7 @@ __must_hold(&lp->lp_lock) static int lnet_peer_send_ping(struct lnet_peer *lp) __must_hold(&lp->lp_lock) { - int nnis; + int bytes; int rc; int cpt; @@ -3404,9 +3402,9 @@ __must_hold(&lp->lp_lock) lnet_peer_addref_locked(lp); lnet_net_unlock(cpt); - nnis = max_t(int, lp->lp_data_nnis, LNET_INTERFACES_MIN); + bytes = max_t(int, lp->lp_data_bytes, LNET_PING_INFO_MIN_SIZE); - rc = lnet_send_ping(&lp->lp_primary_nid, &lp->lp_ping_mdh, nnis, lp, + rc = lnet_send_ping(&lp->lp_primary_nid, &lp->lp_ping_mdh, bytes, lp, the_lnet.ln_dc_handler, false); /* if LNetMDBind in lnet_send_ping fails we need to decrement the * refcount on the peer, otherwise LNetMDUnlink will be called @@ -3514,7 +3512,7 @@ __must_hold(&lp->lp_lock) /* Push source MD */ md.start = &pbuf->pb_info; - md.length = LNET_PING_INFO_SIZE(pbuf->pb_nnis); + md.length = pbuf->pb_nbytes; md.threshold = 2; /* Put/Ack */ md.max_size = 0; md.options = LNET_MD_TRACK_RESPONSE; From patchwork Fri Oct 14 21:37:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55627C4332F for ; Fri, 14 Oct 2022 21:38:29 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BB6lNLz21Hv; Fri, 14 Oct 2022 14:38:26 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B263ntz1xJR for ; Fri, 14 Oct 2022 14:38:18 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id EFD591006F30; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id EAD36DF4C0; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:57 -0400 Message-Id: <1665783491-13827-7-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 06/20] lnet: add iface index to struct lnet_inetdev X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mr NeilBrown When getting list of interfaces, get the index as well, as this can be useful and avoid search the list of interfaces again to find it. WC-bug-id: https://jira.whamcloud.com/browse/LU-10391 Lustre-commit: 860182ee6e84d391a ("LU-10391 lnet: add iface index to struct lnet_inetdev") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48569 Reviewed-by: jsimmons Reviewed-by: Serguei Smirnov Reviewed-by: Frank Sehr Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- include/linux/lnet/lib-lnet.h | 1 + net/lnet/klnds/socklnd/socklnd.c | 2 +- net/lnet/lnet/config.c | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/lnet/lib-lnet.h b/include/linux/lnet/lib-lnet.h index a95919e69802..eb48d2900172 100644 --- a/include/linux/lnet/lib-lnet.h +++ b/include/linux/lnet/lib-lnet.h @@ -826,6 +826,7 @@ struct lnet_inetdev { u32 li_flags; u32 li_ipaddr; u32 li_netmask; + u32 li_index; char li_name[IFNAMSIZ]; }; diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 2b08501133dc..69cb738796e7 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -2522,11 +2522,11 @@ ksocknal_startup(struct lnet_ni *ni) } ni->ni_dev_cpt = ifaces[i].li_cpt; + ksi->ksni_index = ifaces[i].li_index; sa = (void *)&ksi->ksni_addr; memset(sa, 0, sizeof(*sa)); sa->sin_family = AF_INET; sa->sin_addr.s_addr = htonl(ifaces[i].li_ipaddr); - ksi->ksni_index = ksocknal_ip2index((struct sockaddr *)sa, ni); ksi->ksni_netmask = ifaces[i].li_netmask; strlcpy(ksi->ksni_name, ifaces[i].li_name, sizeof(ksi->ksni_name)); diff --git a/net/lnet/lnet/config.c b/net/lnet/lnet/config.c index da3d20e5bebb..083a9a29697f 100644 --- a/net/lnet/lnet/config.c +++ b/net/lnet/lnet/config.c @@ -1538,6 +1538,7 @@ int lnet_inet_enumerate(struct lnet_inetdev **dev_list, struct net *ns) ifaces[nip].li_cpt = cpt; ifaces[nip].li_flags = flags; + ifaces[nip].li_index = dev->ifindex; ifaces[nip].li_ipaddr = ntohl(ifa->ifa_local); ifaces[nip].li_netmask = ntohl(ifa->ifa_mask); strlcpy(ifaces[nip].li_name, ifa->ifa_label, From patchwork Fri Oct 14 21:37:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BFE47C43217 for ; Fri, 14 Oct 2022 21:38:31 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BG518Sz21D4; Fri, 14 Oct 2022 14:38:30 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B421GBz1y57 for ; Fri, 14 Oct 2022 14:38:20 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 0187B1006F3D; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id EFFE8DD535; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:58 -0400 Message-Id: <1665783491-13827-8-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 07/20] lnet: ksocklnd: support IPv6 in ksocknal_ip2index() X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mr NeilBrown ksocknal_ip2index() can now find the interface index for an IPv6 address. WC-bug-id: https://jira.whamcloud.com/browse/LU-10391 Lustre-commit: c0fdf9efbf927db40 ("LU-10391 socklnd: support IPv6 in ksocknal_ip2index()") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48570 Reviewed-by: jsimmons Reviewed-by: Serguei Smirnov Reviewed-by: Frank Sehr Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 40 +++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 9 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 69cb738796e7..89696977ac63 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -40,6 +40,7 @@ #include #include #include +#include #include "socklnd.h" static struct lnet_lnd the_ksocklnd; @@ -79,8 +80,7 @@ static int ksocknal_ip2index(struct sockaddr *addr, struct lnet_ni *ni) int ret = -1; const struct in_ifaddr *ifa; - if (addr->sa_family != AF_INET) - /* No IPv6 support yet */ + if (addr->sa_family != AF_INET && addr->sa_family != AF_INET6) return ret; rcu_read_lock(); @@ -94,14 +94,36 @@ static int ksocknal_ip2index(struct sockaddr *addr, struct lnet_ni *ni) if (!(flags & IFF_UP)) continue; - in_dev = __in_dev_get_rcu(dev); - if (!in_dev) - continue; + switch (addr->sa_family) { + case AF_INET: + in_dev = __in_dev_get_rcu(dev); + if (!in_dev) + continue; - in_dev_for_each_ifa_rcu(ifa, in_dev) { - if (ifa->ifa_local == - ((struct sockaddr_in *)addr)->sin_addr.s_addr) - ret = dev->ifindex; + in_dev_for_each_ifa_rcu(ifa, in_dev) { + if (ifa->ifa_local == + ((struct sockaddr_in *)addr)->sin_addr.s_addr) + ret = dev->ifindex; + } + break; +#if IS_ENABLED(CONFIG_IPV6) + case AF_INET6: { + struct inet6_dev *in6_dev; + const struct inet6_ifaddr *ifa6; + struct sockaddr_in6 *addr6 = (struct sockaddr_in6 *)addr; + + in6_dev = __in6_dev_get(dev); + if (!in6_dev) + continue; + + list_for_each_entry_rcu(ifa6, &in6_dev->addr_list, if_list) { + if (ipv6_addr_cmp(&ifa6->addr, + &addr6->sin6_addr) == 0) + ret = dev->ifindex; + } + break; + } +#endif /* IS_ENABLED(CONFIG_IPV6) */ } if (ret >= 0) break; From patchwork Fri Oct 14 21:37:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60CFFC4332F for ; Fri, 14 Oct 2022 21:38:35 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BM08nzz1y5s; Fri, 14 Oct 2022 14:38:35 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B46slfz1y5k for ; Fri, 14 Oct 2022 14:38:20 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 0674010084A6; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 00709DD53C; Fri, 14 Oct 2022 17:38:13 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:37:59 -0400 Message-Id: <1665783491-13827-9-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 08/20] lnet: only use PUBLIC IP6 addresses for connections X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mr NeilBrown IPv6 can have temporary address. These can be used for short-lives outgoing connections to increase privacy. They are not suitable for long-term connections. So request that only PUBLIC IPv6 addresses are used when making a connection. WC-bug-id: https://jira.whamcloud.com/browse/LU-10391 Lustre-commit: cd3b89be221b4c5b6 ("LU-10391 lnet: only use PUBLIC IP6 addresses for connections") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48571 Reviewed-by: jsimmons Reviewed-by: Frank Sehr Reviewed-by: Serguei Smirnov Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/lnet/lib-socket.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/net/lnet/lnet/lib-socket.c b/net/lnet/lnet/lib-socket.c index 3a99cb69f66f..01f375ed96a3 100644 --- a/net/lnet/lnet/lib-socket.c +++ b/net/lnet/lnet/lib-socket.c @@ -379,6 +379,17 @@ lnet_sock_connect(int interface, int local_port, if (IS_ERR(sock)) return sock; + /* Avoid temporary address, they are bad for long-lived + * connections such as lustre mounts. + * RFC4941, section 3.6 suggests that: + * Individual applications, which have specific + * knowledge about the normal duration of connections, + * MAY override this as appropriate. + */ + if (peeraddr->sa_family == PF_INET6) + ip6_sock_set_addr_preferences(sock->sk, + IPV6_PREFER_SRC_PUBLIC); + rc = kernel_connect(sock, peeraddr, sizeof(struct sockaddr_in6), 0); if (!rc) return sock; From patchwork Fri Oct 14 21:38:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C0D5C43217 for ; Fri, 14 Oct 2022 21:38:35 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BM0NzZz21JS; Fri, 14 Oct 2022 14:38:35 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B54XXqz1y60 for ; Fri, 14 Oct 2022 14:38:21 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 087C510084A7; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 04C5ADD53E; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:00 -0400 Message-Id: <1665783491-13827-10-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 09/20] lustre: osc: Remove oap_magic X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Patrick Farrell oap_magic exists only to debug init and allocation failures, but is allocated for every page of memory, which wastes a lot of memory for something we don't need dedicated debug for. Remove it. WC-bug-id: https://jira.whamcloud.com/browse/LU-15619 Lustre-commit: 721df28648c4b3faa ("LU-15619 osc: Remove oap_magic") Author: Patrick Farrell Signed-off-by: Patrick Farrell Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/46713 Reviewed-by: Zhenyu Xu Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_osc.h | 1 - fs/lustre/osc/osc_cache.c | 6 ------ fs/lustre/osc/osc_page.c | 7 ++----- 3 files changed, 2 insertions(+), 12 deletions(-) diff --git a/fs/lustre/include/lustre_osc.h b/fs/lustre/include/lustre_osc.h index 323eeba3da42..884eafee8a83 100644 --- a/fs/lustre/include/lustre_osc.h +++ b/fs/lustre/include/lustre_osc.h @@ -75,7 +75,6 @@ enum async_flags { }; struct osc_async_page { - int oap_magic; unsigned short oap_cmd; struct list_head oap_pending_item; diff --git a/fs/lustre/osc/osc_cache.c b/fs/lustre/osc/osc_cache.c index 36fec837d93e..12d9ab519e48 100644 --- a/fs/lustre/osc/osc_cache.c +++ b/fs/lustre/osc/osc_cache.c @@ -2313,7 +2313,6 @@ int osc_prep_async_page(struct osc_object *osc, struct osc_page *ops, if (!page) return -EIO; - oap->oap_magic = OAP_MAGIC; oap->oap_obj = osc; oap->oap_page = vmpage; @@ -2354,9 +2353,6 @@ int osc_queue_async_io(const struct lu_env *env, struct cl_io *io, bool need_release = false; int rc = 0; - if (oap->oap_magic != OAP_MAGIC) - return -EINVAL; - if (!cli->cl_import || cli->cl_import->imp_invalid) return -EIO; @@ -2537,8 +2533,6 @@ int osc_teardown_async_page(const struct lu_env *env, struct osc_async_page *oap = &ops->ops_oap; int rc = 0; - LASSERT(oap->oap_magic == OAP_MAGIC); - CDEBUG(D_INFO, "teardown oap %p page %p at index %lu.\n", oap, ops, osc_index(oap2osc(oap))); diff --git a/fs/lustre/osc/osc_page.c b/fs/lustre/osc/osc_page.c index 12ba10827a23..ba10ba320079 100644 --- a/fs/lustre/osc/osc_page.c +++ b/fs/lustre/osc/osc_page.c @@ -125,10 +125,10 @@ static int osc_page_print(const struct lu_env *env, struct client_obd *cli = &osc_export(obj)->exp_obd->u.cli; return (*printer)(env, cookie, LUSTRE_OSC_NAME - "-page@%p %lu: 1< %#x %d %c %c > 2< %llu %u %u %#x %#x | %p %p %p > 3< %d %d > 4< %d %d %d %lu %c | %c %c %c %c > 5< %c %c %c %c | %d %c | %d %c %c>\n", + "-page@%p %lu: 1< %d %c %c > 2< %llu %u %u %#x %#x | %p %p %p > 3< %d %d > 4< %d %d %d %lu %c | %c %c %c %c > 5< %c %c %c %c | %d %c | %d %c %c>\n", opg, osc_index(opg), /* 1 */ - oap->oap_magic, oap->oap_cmd, + oap->oap_cmd, list_empty_marker(&oap->oap_pending_item), list_empty_marker(&oap->oap_rpc_item), /* 2 */ @@ -293,9 +293,6 @@ void osc_page_submit(const struct lu_env *env, struct osc_page *opg, struct osc_io *oio = osc_env_io(env); struct osc_async_page *oap = &opg->ops_oap; - LASSERTF(oap->oap_magic == OAP_MAGIC, - "Bad oap magic: oap %p, magic 0x%x\n", - oap, oap->oap_magic); LASSERT(oap->oap_async_flags & ASYNC_READY); LASSERT(oap->oap_async_flags & ASYNC_COUNT_STABLE); From patchwork Fri Oct 14 21:38:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 02DD0C4332F for ; Fri, 14 Oct 2022 21:38:42 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BV4pntz1y53; Fri, 14 Oct 2022 14:38:42 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B62xx2z1y2q for ; Fri, 14 Oct 2022 14:38:22 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 0B44610084A8; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 09685DD6C6; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:01 -0400 Message-Id: <1665783491-13827-11-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 10/20] lustre: ptlrpc: add assert for ptlrpc_service_purge_all X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Etienne AUJAMES , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Etienne AUJAMES ptlrpc_service_purge_all() calls ptlrpc_server_request_get() with "force=true" to purge all active requests before stopping an NRS policy (when unregistering a service). "force" mode should always return a request if a pending request is present in the NRS policy. BUG: unable to handle kernel NULL pointer dereference at 0000000000000114 IP: [] ptlrpc_nrs_req_stop_nolock+0x5/0x150 ..... ? ptlrpc_server_finish_active_request+0x2b/0x140 [ptlrpc] ptlrpc_service_purge_all+0x137/0x920 [ptlrpc] ptlrpc_unregister_service+0xe7/0x6f0 [ptlrpc] ost_cleanup+0x52/0x1b0 [ost] class_free_dev+0x21d/0x720 [obdclass] class_export_put+0x1f0/0x2c0 [obdclass] class_unlink_export+0x135/0x170 [obdclass] class_decref+0x80/0x160 [obdclass] class_detach+0x1b3/0x2e0 [obdclass] class_process_config+0x1a38/0x2830 [obdclass] ? complete+0x4a/0x60 ? list_del+0xd/0x30 ? wait_for_completion+0x4e/0x140 class_manual_cleanup+0x1e0/0x710 [obdclass] server_stop_servers+0xd5/0x160 [obdclass] server_put_super+0x12d/0xd00 [obdclass] generic_shutdown_super+0x6d/0x100 WC-bug-id: https://jira.whamcloud.com/browse/LU-16144 Lustre-commit: 1bba7dd425d3fc9ef3 ("LU-16144 nrs: implement force mode for nrs_tbf_req_get()") Signed-off-by: Etienne AUJAMES Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48494 Reviewed-by: Nikitas Angelinas Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/ptlrpc/service.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/lustre/ptlrpc/service.c b/fs/lustre/ptlrpc/service.c index 59fe1f4aa18f..aaf7529e25f3 100644 --- a/fs/lustre/ptlrpc/service.c +++ b/fs/lustre/ptlrpc/service.c @@ -2939,6 +2939,7 @@ ptlrpc_service_purge_all(struct ptlrpc_service *svc) while (ptlrpc_server_request_pending(svcpt, true)) { req = ptlrpc_server_request_get(svcpt, true); + LASSERT(req); ptlrpc_server_finish_active_request(svcpt, req); } From patchwork Fri Oct 14 21:38:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00503C4332F for ; Fri, 14 Oct 2022 21:38:39 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BR5806z1y60; Fri, 14 Oct 2022 14:38:39 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B70drqz21Bb for ; Fri, 14 Oct 2022 14:38:23 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 0FF1610084AA; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 0E18DDD535; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:02 -0400 Message-Id: <1665783491-13827-12-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 11/20] lustre: ptlrpc: lower the message level in no resend case X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yang Sheng , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Yang Sheng Don't report the wrong generation as a error message in rq_no_resend case. WC-bug-id: https://jira.whamcloud.com/browse/LU-16166 Lustre-commit: d13cca56a5ae2ad44 ("LU-16166 ptlrpc: lower the message level in no resend case") Signed-off-by: Yang Sheng Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48585 Reviewed-by: Emoly Liu Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/ptlrpc/client.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/lustre/ptlrpc/client.c b/fs/lustre/ptlrpc/client.c index 069ffdf6f93f..5f0ff476db6a 100644 --- a/fs/lustre/ptlrpc/client.c +++ b/fs/lustre/ptlrpc/client.c @@ -1227,7 +1227,8 @@ static int ptlrpc_import_delay_req(struct obd_import *imp, DEBUG_REQ(D_NET, req, "IMP_INVALID"); *status = -ESHUTDOWN; /* bz 12940 */ } else if (req->rq_import_generation != imp->imp_generation) { - DEBUG_REQ(D_ERROR, req, "req wrong generation:"); + DEBUG_REQ(req->rq_no_resend ? D_INFO : D_ERROR, + req, "req wrong generation:"); *status = -EIO; } else if (req->rq_send_state != imp->imp_state) { /* invalidate in progress - any requests should be drop */ From patchwork Fri Oct 14 21:38:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19040C433FE for ; Fri, 14 Oct 2022 21:38:47 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BZ5bVWz1y56; Fri, 14 Oct 2022 14:38:46 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B75Vxtz21GQ for ; Fri, 14 Oct 2022 14:38:23 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 1540210084DE; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 11F44DD6ED; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:03 -0400 Message-Id: <1665783491-13827-13-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 12/20] lustre: obdclass: user netlink to collect devices information X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" Our utilities can report to users a device list with various bits of data using the debugfs file 'devices'. This debugfs file is only by default available to root which prevents regular users from collecting information. Enable non-root users to collect the same information for lctl dl using netlink. The advantage of using netlink is that it also removes the 8K ioctl limit. Add the ability to present this data in YAML format as well. WC-bug-id: https://jira.whamcloud.com/browse/LU-9680 Lustre-commit: 86ba46c24430f67bb ("LU-9680 obdclass: user netlink to collect devices information") Signed-off-by: James Simmons Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/31618 Reviewed-by: Jian Yu Reviewed-by: Neil Brown Reviewed-by: Oleg Drokin --- fs/lustre/include/lustre_kernelcomm.h | 37 ++- fs/lustre/obdclass/class_obd.c | 14 +- fs/lustre/obdclass/kernelcomm.c | 257 +++++++++++++++++- include/uapi/linux/lustre/lustre_kernelcomm.h | 18 ++ 4 files changed, 315 insertions(+), 11 deletions(-) diff --git a/fs/lustre/include/lustre_kernelcomm.h b/fs/lustre/include/lustre_kernelcomm.h index bd5376b2a672..db725ab8a0f9 100644 --- a/fs/lustre/include/lustre_kernelcomm.h +++ b/fs/lustre/include/lustre_kernelcomm.h @@ -41,11 +41,46 @@ /* For declarations shared with userspace */ #include +/** + * enum lustre_device_attrs - Lustre general top-level netlink + * attributes that describe lustre + * 'devices'. These values are used + * to piece togther messages for + * sending and receiving. + * + * @LUSTRE_DEVICE_ATTR_UNSPEC: unspecified attribute to catch errors + * + * @LUSTRE_DEVICE_ATTR_HDR: Netlink group this data is for + * (NLA_NUL_STRING) + * @LUSTRE_DEVICE_ATTR_INDEX: device number used as an index (NLA_U16) + * @LUSTRE_DEVICE_ATTR_STATUS: status of the device (NLA_STRING) + * @LUSTRE_DEVICE_ATTR_CLASS: class the device belongs to (NLA_STRING) + * @LUSTRE_DEVICE_ATTR_NAME: name of the device (NLA_STRING) + * @LUSTRE_DEVICE_ATTR_UUID: UUID of the device (NLA_STRING) + * @LUSTRE_DEVICE_ATTR_REFCOUNT: refcount of the device (NLA_U32) + */ +enum lustre_device_attrs { + LUSTRE_DEVICE_ATTR_UNSPEC = 0, + + LUSTRE_DEVICE_ATTR_HDR, + LUSTRE_DEVICE_ATTR_INDEX, + LUSTRE_DEVICE_ATTR_STATUS, + LUSTRE_DEVICE_ATTR_CLASS, + LUSTRE_DEVICE_ATTR_NAME, + LUSTRE_DEVICE_ATTR_UUID, + LUSTRE_DEVICE_ATTR_REFCOUNT, + + __LUSTRE_DEVICE_ATTR_MAX_PLUS_ONE +}; + +#define LUSTRE_DEVICE_ATTR_MAX (__LUSTRE_DEVICE_ATTR_MAX_PLUS_ONE - 1) + /* prototype for callback function on kuc groups */ typedef int (*libcfs_kkuc_cb_t)(void *data, void *cb_arg); /* Kernel methods */ -void libcfs_kkuc_init(void); +int libcfs_kkuc_init(void); +void libcfs_kkuc_fini(void); int libcfs_kkuc_group_put(const struct obd_uuid *uuid, int group, void *data); int libcfs_kkuc_group_add(struct file *fp, const struct obd_uuid *uuid, int uid, int group, void *data, size_t data_len); diff --git a/fs/lustre/obdclass/class_obd.c b/fs/lustre/obdclass/class_obd.c index f455ed752c15..67a94222a664 100644 --- a/fs/lustre/obdclass/class_obd.c +++ b/fs/lustre/obdclass/class_obd.c @@ -671,15 +671,17 @@ static int __init obdclass_init(void) if (err) return err; - libcfs_kkuc_init(); + err = obd_init_checks(); + if (err) + return err; - err = obd_zombie_impexp_init(); + err = libcfs_kkuc_init(); if (err) return err; - err = obd_init_checks(); + err = obd_zombie_impexp_init(); if (err) - goto cleanup_zombie_impexp; + goto cleanup_kkuc; err = class_handle_init(); if (err) @@ -754,6 +756,9 @@ static int __init obdclass_init(void) cleanup_zombie_impexp: obd_zombie_impexp_stop(); +cleanup_kkuc: + libcfs_kkuc_fini(); + return err; } @@ -771,6 +776,7 @@ static void obdclass_exit(void) class_handle_cleanup(); class_del_uuid(NULL); /* Delete all UUIDs. */ obd_zombie_impexp_stop(); + libcfs_kkuc_fini(); } void obd_heat_clear(struct obd_heat_instance *instance, int count) diff --git a/fs/lustre/obdclass/kernelcomm.c b/fs/lustre/obdclass/kernelcomm.c index e59b6aadf097..5682d4e1ab53 100644 --- a/fs/lustre/obdclass/kernelcomm.c +++ b/fs/lustre/obdclass/kernelcomm.c @@ -38,16 +38,254 @@ #define DEBUG_SUBSYSTEM S_CLASS #include -#include +#include +#include +#include + +#include #include #include +static struct genl_family lustre_family; + +static struct ln_key_list device_list = { + .lkl_maxattr = LUSTRE_DEVICE_ATTR_MAX, + .lkl_list = { + [LUSTRE_DEVICE_ATTR_HDR] = { + .lkp_value = "devices", + .lkp_key_format = LNKF_SEQUENCE | LNKF_MAPPING, + .lkp_data_type = NLA_NUL_STRING, + }, + [LUSTRE_DEVICE_ATTR_INDEX] = { + .lkp_value = "index", + .lkp_data_type = NLA_U16 + }, + [LUSTRE_DEVICE_ATTR_STATUS] = { + .lkp_value = "status", + .lkp_data_type = NLA_STRING + }, + [LUSTRE_DEVICE_ATTR_CLASS] = { + .lkp_value = "type", + .lkp_data_type = NLA_STRING + }, + [LUSTRE_DEVICE_ATTR_NAME] = { + .lkp_value = "name", + .lkp_data_type = NLA_STRING + }, + [LUSTRE_DEVICE_ATTR_UUID] = { + .lkp_value = "uuid", + .lkp_data_type = NLA_STRING + }, + [LUSTRE_DEVICE_ATTR_REFCOUNT] = { + .lkp_value = "refcount", + .lkp_data_type = NLA_U32 + }, + }, +}; + +struct genl_dev_list { + struct obd_device *gdl_target; + unsigned int gdl_start; +}; + +static inline struct genl_dev_list * +device_dump_ctx(struct netlink_callback *cb) +{ + return (struct genl_dev_list *)cb->args[0]; +} + +/* generic ->start() handler for GET requests */ +static int lustre_device_list_start(struct netlink_callback *cb) +{ + struct genlmsghdr *gnlh = nlmsg_data(cb->nlh); + struct netlink_ext_ack *extack = cb->extack; + struct genl_dev_list *glist; + int msg_len, rc = 0; + + glist = kmalloc(sizeof(*glist), GFP_KERNEL); + if (!glist) + return -ENOMEM; + + cb->args[0] = (long)glist; + glist->gdl_target = NULL; + glist->gdl_start = 0; + + msg_len = genlmsg_len(gnlh); + if (msg_len > 0) { + struct nlattr *params = genlmsg_data(gnlh); + struct nlattr *dev; + int rem; + + nla_for_each_attr(dev, params, msg_len, rem) { + struct nlattr *prop; + int rem2; + + nla_for_each_nested(prop, dev, rem2) { + char name[MAX_OBD_NAME]; + struct obd_device *obd; + + if (nla_type(prop) != LN_SCALAR_ATTR_VALUE || + nla_strcmp(prop, "name") != 0) + continue; + + prop = nla_next(prop, &rem2); + if (nla_type(prop) != LN_SCALAR_ATTR_VALUE) { + rc = -EINVAL; + goto report_err; + } + + rc = nla_strlcpy(name, prop, sizeof(name)); + if (rc < 0) + goto report_err; + rc = 0; + + obd = class_name2obd(name); + if (obd) + glist->gdl_target = obd; + } + } + if (!glist->gdl_target) { + NL_SET_ERR_MSG(extack, "No devices found"); + rc = -ENOENT; + } + } +report_err: + if (rc < 0) { + kfree(glist); + cb->args[0] = 0; + } + return rc; +} + +static int lustre_device_list_dump(struct sk_buff *msg, + struct netlink_callback *cb) +{ + struct genl_dev_list *glist = device_dump_ctx(cb); + struct obd_device *filter = glist->gdl_target; + struct netlink_ext_ack *extack = cb->extack; + int portid = NETLINK_CB(cb->skb).portid; + int seq = cb->nlh->nlmsg_seq; + int idx, rc = 0; + + if (glist->gdl_start == 0) { + const struct ln_key_list *all[] = { + &device_list, NULL + }; + + rc = lnet_genl_send_scalar_list(msg, portid, seq, + &lustre_family, + NLM_F_CREATE | NLM_F_MULTI, + LUSTRE_CMD_DEVICES, all); + if (rc < 0) { + NL_SET_ERR_MSG(extack, "failed to send key table"); + return rc; + } + } + + for (idx = glist->gdl_start; idx < class_devno_max(); idx++) { + struct obd_device *obd; + const char *status; + void *hdr; + + obd = class_num2obd(idx); + if (!obd) + continue; + + if (filter && filter != obd) + continue; + + hdr = genlmsg_put(msg, portid, seq, &lustre_family, + NLM_F_MULTI, LUSTRE_CMD_DEVICES); + if (!hdr) { + NL_SET_ERR_MSG(extack, "failed to send values"); + genlmsg_cancel(msg, hdr); + rc = -EMSGSIZE; + break; + } + + if (idx == 0) + nla_put_string(msg, LUSTRE_DEVICE_ATTR_HDR, ""); + + nla_put_u16(msg, LUSTRE_DEVICE_ATTR_INDEX, obd->obd_minor); + + /* Collect only the index value for a single obd */ + if (filter) { + genlmsg_end(msg, hdr); + idx++; + break; + } + + if (obd->obd_stopping) + status = "ST"; + else if (obd->obd_inactive) + status = "IN"; + else if (obd->obd_set_up) + status = "UP"; + else if (obd->obd_attached) + status = "AT"; + else + status = "--"; + + nla_put_string(msg, LUSTRE_DEVICE_ATTR_STATUS, status); + + nla_put_string(msg, LUSTRE_DEVICE_ATTR_CLASS, + obd->obd_type->typ_name); + + nla_put_string(msg, LUSTRE_DEVICE_ATTR_NAME, + obd->obd_name); + + nla_put_string(msg, LUSTRE_DEVICE_ATTR_UUID, + obd->obd_uuid.uuid); + + nla_put_u32(msg, LUSTRE_DEVICE_ATTR_REFCOUNT, + atomic_read(&obd->obd_refcount)); + + genlmsg_end(msg, hdr); + } + + glist->gdl_start = idx; + return rc < 0 ? rc : msg->len; +} + +int lustre_device_done(struct netlink_callback *cb) +{ + struct genl_dev_list *glist; + + glist = device_dump_ctx(cb); + kfree(glist); + cb->args[0] = 0; + + return 0; +} + +static const struct genl_multicast_group lustre_mcast_grps[] = { + { .name = "devices", }, +}; + +static const struct genl_ops lustre_genl_ops[] = { + { + .cmd = LUSTRE_CMD_DEVICES, + .start = lustre_device_list_start, + .dumpit = lustre_device_list_dump, + .done = lustre_device_done, + }, +}; + +static struct genl_family lustre_family = { + .name = LUSTRE_GENL_NAME, + .version = LUSTRE_GENL_VERSION, + .module = THIS_MODULE, + .ops = lustre_genl_ops, + .n_ops = ARRAY_SIZE(lustre_genl_ops), + .mcgrps = lustre_mcast_grps, + .n_mcgrps = ARRAY_SIZE(lustre_mcast_grps), +}; + /** * libcfs_kkuc_msg_put - send an message from kernel to userspace - * - * @fp: to send the message to - * @payload: Payload data. First field of payload is always - * struct kuc_hdr + * @param fp to send the message to + * @param payload Payload data. First field of payload is always + * struct kuc_hdr */ static int libcfs_kkuc_msg_put(struct file *filp, void *payload) { @@ -104,12 +342,19 @@ static inline bool libcfs_kkuc_group_is_valid(unsigned int group) return group < ARRAY_SIZE(kkuc_groups); } -void libcfs_kkuc_init(void) +int libcfs_kkuc_init(void) { int group; for (group = 0; group < ARRAY_SIZE(kkuc_groups); group++) INIT_LIST_HEAD(&kkuc_groups[group]); + + return genl_register_family(&lustre_family); +} + +void libcfs_kkuc_fini(void) +{ + genl_unregister_family(&lustre_family); } /** Add a receiver to a broadcast group diff --git a/include/uapi/linux/lustre/lustre_kernelcomm.h b/include/uapi/linux/lustre/lustre_kernelcomm.h index 744eeb674f72..91bb686d33e9 100644 --- a/include/uapi/linux/lustre/lustre_kernelcomm.h +++ b/include/uapi/linux/lustre/lustre_kernelcomm.h @@ -39,6 +39,24 @@ #include +#define LUSTRE_GENL_NAME "lustre" +#define LUSTRE_GENL_VERSION 0x1 + +/* + * enum lustre_commands - Supported Lustre Netlink commands + * + * @LUSTRE_CMD_UNSPEC: unspecified command to catch errors + * @LUSTRE_CMD_DEVICES: command to manage the Lustre devices + */ +enum lustre_commands { + LUSTRE_CMD_UNSPEC = 0, + LUSTRE_CMD_DEVICES = 1, + + __LUSTRE_CMD_MAX_PLUS_ONE +}; + +#define LUSTRE_CMD_MAX (__LUSTRE_CMD_MAX_PLUS_ONE - 1) + /* KUC message header. * All current and future KUC messages should use this header. * To avoid having to include Lustre headers from libcfs, define this here. From patchwork Fri Oct 14 21:38:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8A481C43217 for ; Fri, 14 Oct 2022 21:38:44 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0BX16P7z1y2t; Fri, 14 Oct 2022 14:38:44 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B83mtKz21H2 for ; Fri, 14 Oct 2022 14:38:24 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 18AE110084E1; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 152E6DD53C; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:04 -0400 Message-Id: <1665783491-13827-14-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 13/20] lnet: use %pISc for formatting IP addresses X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mr NeilBrown The Linux kernel's printf functionality understands %pIS to means that a the address in a 'struct sockaddr' should be formated, either as IPv4 or IPv6. For IPv6, the verbose format showing all 16 bytes whether zero or not is used. To get the more familiar "compressed" format where strings of :0000: are replaced with ::, we need to add the 'c' flag. This is ignored for IPv4. When requesting the port as well ("%pISp), the 'c' and 'p' can appear in either order. So this patch changes all %pIS to %pISc as we always want the compressed format. WC-bug-id: https://jira.whamcloud.com/browse/LU-10391 Lustre-commit: ed6b125bab5f7e383 ("LU-10391 lnet: use %pISc for formatting IP addresses") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48685 Reviewed-by: James Simmons Reviewed-by: Andreas Dilger Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 18 +++++------ net/lnet/klnds/socklnd/socklnd_cb.c | 44 +++++++++++++------------- net/lnet/klnds/socklnd/socklnd_proto.c | 30 +++++++++--------- net/lnet/lnet/acceptor.c | 34 ++++++++++---------- net/lnet/lnet/lib-socket.c | 2 +- 5 files changed, 64 insertions(+), 64 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 89696977ac63..8999580b67b4 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -527,13 +527,13 @@ ksocknal_associate_cb_conn_locked(struct ksock_conn_cb *conn_cb, /* conn_cb wasn't bound locally yet (the initial * conn_cb) */ - CDEBUG(D_NET, "Binding %s %pIS to interface %d\n", + CDEBUG(D_NET, "Binding %s %pISc to interface %d\n", libcfs_idstr(&peer_ni->ksnp_id), &conn_cb->ksnr_addr, conn_iface); } else { CDEBUG(D_NET, - "Rebinding %s %pIS from interface %d to %d\n", + "Rebinding %s %pISc from interface %d to %d\n", libcfs_idstr(&peer_ni->ksnp_id), &conn_cb->ksnr_addr, conn_cb->ksnr_myiface, @@ -835,7 +835,7 @@ ksocknal_accept(struct lnet_ni *ni, struct socket *sock) cr = kzalloc(sizeof(*cr), GFP_NOFS); if (!cr) { LCONSOLE_ERROR_MSG(0x12f, - "Dropping connection request from %pIS: memory exhausted\n", + "Dropping connection request from %pISc: memory exhausted\n", &peer); return -ENOMEM; } @@ -1113,7 +1113,7 @@ ksocknal_create_conn(struct lnet_ni *ni, struct ksock_conn_cb *conn_cb, if (active && !rpc_cmp_addr((struct sockaddr *)&conn_cb->ksnr_addr, (struct sockaddr *)&conn->ksnc_peeraddr)) { - CERROR("Route %s %pIS connected to %pIS\n", + CERROR("Route %s %pISc connected to %pISc\n", libcfs_idstr(&peer_ni->ksnp_id), &conn_cb->ksnr_addr, &conn->ksnc_peeraddr); @@ -1183,7 +1183,7 @@ ksocknal_create_conn(struct lnet_ni *ni, struct ksock_conn_cb *conn_cb, * socket callbacks. */ CDEBUG(D_NET, - "New conn %s p %d.x %pIS -> %pISp incarnation:%lld sched[%d]\n", + "New conn %s p %d.x %pISc -> %pIScp incarnation:%lld sched[%d]\n", libcfs_idstr(&peerid), conn->ksnc_proto->pro_version, &conn->ksnc_myaddr, &conn->ksnc_peeraddr, incarnation, cpt); @@ -1552,7 +1552,7 @@ ksocknal_destroy_conn(struct ksock_conn *conn) case SOCKNAL_RX_LNET_PAYLOAD: last_rcv = conn->ksnc_rx_deadline - ksocknal_timeout(); - CERROR("Completing partial receive from %s[%d], ip %pISp, with error, wanted: %zd, left: %d, last alive is %lld secs ago\n", + CERROR("Completing partial receive from %s[%d], ip %pIScp, with error, wanted: %zd, left: %d, last alive is %lld secs ago\n", libcfs_idstr(&conn->ksnc_peer->ksnp_id), conn->ksnc_type, &conn->ksnc_peeraddr, @@ -1565,21 +1565,21 @@ ksocknal_destroy_conn(struct ksock_conn *conn) break; case SOCKNAL_RX_LNET_HEADER: if (conn->ksnc_rx_started) - CERROR("Incomplete receive of lnet header from %s, ip %pISp, with error, protocol: %d.x.\n", + CERROR("Incomplete receive of lnet header from %s, ip %pIScp, with error, protocol: %d.x.\n", libcfs_idstr(&conn->ksnc_peer->ksnp_id), &conn->ksnc_peeraddr, conn->ksnc_proto->pro_version); break; case SOCKNAL_RX_KSM_HEADER: if (conn->ksnc_rx_started) - CERROR("Incomplete receive of ksock message from %s, ip %pISp, with error, protocol: %d.x.\n", + CERROR("Incomplete receive of ksock message from %s, ip %pIScp, with error, protocol: %d.x.\n", libcfs_idstr(&conn->ksnc_peer->ksnp_id), &conn->ksnc_peeraddr, conn->ksnc_proto->pro_version); break; case SOCKNAL_RX_SLOP: if (conn->ksnc_rx_started) - CERROR("Incomplete receive of slops from %s, ip %pISp, with error\n", + CERROR("Incomplete receive of slops from %s, ip %pIScp, with error\n", libcfs_idstr(&conn->ksnc_peer->ksnp_id), &conn->ksnc_peeraddr); break; diff --git a/net/lnet/klnds/socklnd/socklnd_cb.c b/net/lnet/klnds/socklnd/socklnd_cb.c index 308d8b0d6366..b2da535fbfbe 100644 --- a/net/lnet/klnds/socklnd/socklnd_cb.c +++ b/net/lnet/klnds/socklnd/socklnd_cb.c @@ -533,15 +533,15 @@ ksocknal_process_transmit(struct ksock_conn *conn, struct ksock_tx *tx) if (!conn->ksnc_closing) { switch (rc) { case -ECONNRESET: - LCONSOLE_WARN("Host %pIS reset our connection while we were sending data; it may have rebooted.\n", + LCONSOLE_WARN("Host %pISc reset our connection while we were sending data; it may have rebooted.\n", &conn->ksnc_peeraddr); break; default: - LCONSOLE_WARN("There was an unexpected network error while writing to %pIS: %d.\n", + LCONSOLE_WARN("There was an unexpected network error while writing to %pISc: %d.\n", &conn->ksnc_peeraddr, rc); break; } - CDEBUG(D_NET, "[%p] Error %d on write to %s ip %pISp\n", + CDEBUG(D_NET, "[%p] Error %d on write to %s ip %pIScp\n", conn, rc, libcfs_idstr(&conn->ksnc_peer->ksnp_id), &conn->ksnc_peeraddr); } @@ -676,7 +676,7 @@ ksocknal_queue_tx_locked(struct ksock_tx *tx, struct ksock_conn *conn) */ LASSERT(!conn->ksnc_closing); - CDEBUG(D_NET, "Sending to %s ip %pISp\n", + CDEBUG(D_NET, "Sending to %s ip %pIScp\n", libcfs_idstr(&conn->ksnc_peer->ksnp_id), &conn->ksnc_peeraddr); @@ -777,7 +777,7 @@ ksocknal_find_connectable_conn_cb_locked(struct ksock_peer_ni *peer_ni) if (!(conn_cb->ksnr_retry_interval == 0 || /* first attempt */ now >= conn_cb->ksnr_timeout)) { CDEBUG(D_NET, - "Too soon to retry route %pIS (cnted %d, interval %lld, %lld secs later)\n", + "Too soon to retry route %pISc (cnted %d, interval %lld, %lld secs later)\n", &conn_cb->ksnr_addr, conn_cb->ksnr_connected, conn_cb->ksnr_retry_interval, @@ -1095,11 +1095,11 @@ ksocknal_process_receive(struct ksock_conn *conn) LASSERT(rc != -EAGAIN); if (!rc) - CDEBUG(D_NET, "[%p] EOF from %s ip %pISp\n", + CDEBUG(D_NET, "[%p] EOF from %s ip %pIScp\n", conn, libcfs_idstr(ksnp_id), &conn->ksnc_peeraddr); else if (!conn->ksnc_closing) - CERROR("[%p] Error %d on read from %s ip %pISp\n", + CERROR("[%p] Error %d on read from %s ip %pIScp\n", conn, rc, libcfs_idstr(ksnp_id), &conn->ksnc_peeraddr); @@ -1653,7 +1653,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, rc = lnet_sock_read(sock, &hello->kshm_magic, sizeof(hello->kshm_magic), timeout); if (rc) { - CERROR("Error %d reading HELLO from %pIS\n", + CERROR("Error %d reading HELLO from %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0); return rc; @@ -1663,7 +1663,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, hello->kshm_magic != __swab32(LNET_PROTO_MAGIC) && hello->kshm_magic != le32_to_cpu(LNET_PROTO_TCP_MAGIC)) { /* Unexpected magic! */ - CERROR("Bad magic(1) %#08x (%#08x expected) from %pIS\n", + CERROR("Bad magic(1) %#08x (%#08x expected) from %pISc\n", __cpu_to_le32(hello->kshm_magic), LNET_PROTO_TCP_MAGIC, &conn->ksnc_peeraddr); return -EPROTO; @@ -1672,7 +1672,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, rc = lnet_sock_read(sock, &hello->kshm_version, sizeof(hello->kshm_version), timeout); if (rc) { - CERROR("Error %d reading HELLO from %pIS\n", + CERROR("Error %d reading HELLO from %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0); return rc; @@ -1696,7 +1696,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, hello); } - CERROR("Unknown protocol version (%d.x expected) from %pIS\n", + CERROR("Unknown protocol version (%d.x expected) from %pISc\n", conn->ksnc_proto->pro_version, &conn->ksnc_peeraddr); @@ -1709,7 +1709,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, /* receive the rest of hello message anyway */ rc = conn->ksnc_proto->pro_recv_hello(conn, hello, timeout); if (rc) { - CERROR("Error %d reading or checking hello from from %pIS\n", + CERROR("Error %d reading or checking hello from from %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0); return rc; @@ -1718,7 +1718,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, *incarnation = hello->kshm_src_incarnation; if (LNET_NID_IS_ANY(&hello->kshm_src_nid)) { - CERROR("Expecting a HELLO hdr with a NID, but got LNET_NID_ANY from %pIS\n", + CERROR("Expecting a HELLO hdr with a NID, but got LNET_NID_ANY from %pISc\n", &conn->ksnc_peeraddr); return -EPROTO; } @@ -1746,7 +1746,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, /* peer_ni determines type */ conn->ksnc_type = ksocknal_invert_type(hello->kshm_ctype); if (conn->ksnc_type == SOCKLND_CONN_NONE) { - CERROR("Unexpected type %d from %s ip %pIS\n", + CERROR("Unexpected type %d from %s ip %pISc\n", hello->kshm_ctype, libcfs_idstr(peerid), &conn->ksnc_peeraddr); return -EPROTO; @@ -1758,7 +1758,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, if (peerid->pid != recv_id.pid || !nid_same(&peerid->nid, &recv_id.nid)) { LCONSOLE_ERROR_MSG(0x130, - "Connected successfully to %s on host %pIS, but they claimed they were %s; please check your Lustre configuration.\n", + "Connected successfully to %s on host %pISc, but they claimed they were %s; please check your Lustre configuration.\n", libcfs_idstr(peerid), &conn->ksnc_peeraddr, libcfs_idstr(&recv_id)); @@ -1771,7 +1771,7 @@ ksocknal_recv_hello(struct lnet_ni *ni, struct ksock_conn *conn, } if (ksocknal_invert_type(hello->kshm_ctype) != conn->ksnc_type) { - CERROR("Mismatched types: me %d, %s ip %pIS %d\n", + CERROR("Mismatched types: me %d, %s ip %pISc %d\n", conn->ksnc_type, libcfs_idstr(peerid), &conn->ksnc_peeraddr, hello->kshm_ctype); return -EPROTO; @@ -2149,7 +2149,7 @@ ksocknal_connd(void *arg) if (ksocknal_connect(conn_cb)) { /* consecutive retry */ if (cons_retry++ > SOCKNAL_INSANITY_RECONN) { - CWARN("massive consecutive re-connecting to %pIS\n", + CWARN("massive consecutive re-connecting to %pISc\n", &conn_cb->ksnr_addr); cons_retry = 0; } @@ -2211,17 +2211,17 @@ ksocknal_find_timed_out_conn(struct ksock_peer_ni *peer_ni) switch (error) { case ECONNRESET: - CNETERR("A connection with %s (%pISp) was reset; it may have rebooted.\n", + CNETERR("A connection with %s (%pIScp) was reset; it may have rebooted.\n", libcfs_idstr(&peer_ni->ksnp_id), &conn->ksnc_peeraddr); break; case ETIMEDOUT: - CNETERR("A connection with %s (%pISp) timed out; the network or node may be down.\n", + CNETERR("A connection with %s (%pIScp) timed out; the network or node may be down.\n", libcfs_idstr(&peer_ni->ksnp_id), &conn->ksnc_peeraddr); break; default: - CNETERR("An unexpected network error %d occurred with %s (%pISp\n", + CNETERR("An unexpected network error %d occurred with %s (%pIScp\n", error, libcfs_idstr(&peer_ni->ksnp_id), &conn->ksnc_peeraddr); @@ -2235,7 +2235,7 @@ ksocknal_find_timed_out_conn(struct ksock_peer_ni *peer_ni) ktime_get_seconds() >= conn->ksnc_rx_deadline) { /* Timed out incomplete incoming message */ ksocknal_conn_addref(conn); - CNETERR("Timeout receiving from %s (%pISp), state %d wanted %zd left %d\n", + CNETERR("Timeout receiving from %s (%pIScp), state %d wanted %zd left %d\n", libcfs_idstr(&peer_ni->ksnp_id), &conn->ksnc_peeraddr, conn->ksnc_rx_state, @@ -2257,7 +2257,7 @@ ksocknal_find_timed_out_conn(struct ksock_peer_ni *peer_ni) tx_list) tx->tx_hstatus = LNET_MSG_STATUS_LOCAL_TIMEOUT; - CNETERR("Timeout sending data to %s (%pISp) the network or that node may be down.\n", + CNETERR("Timeout sending data to %s (%pIScp) the network or that node may be down.\n", libcfs_idstr(&peer_ni->ksnp_id), &conn->ksnc_peeraddr); spin_unlock_bh(&sched->kss_lock); diff --git a/net/lnet/klnds/socklnd/socklnd_proto.c b/net/lnet/klnds/socklnd/socklnd_proto.c index 0a93d572ea68..4d92a1c93aad 100644 --- a/net/lnet/klnds/socklnd/socklnd_proto.c +++ b/net/lnet/klnds/socklnd/socklnd_proto.c @@ -548,7 +548,7 @@ ksocknal_send_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello) rc = lnet_sock_write(sock, hdr, sizeof(*hdr), lnet_acceptor_timeout()); if (rc) { - CNETERR("Error %d sending HELLO hdr to %pISp\n", + CNETERR("Error %d sending HELLO hdr to %pIScp\n", rc, &conn->ksnc_peeraddr); goto out; } @@ -563,7 +563,7 @@ ksocknal_send_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello) hello->kshm_nips * sizeof(u32), lnet_acceptor_timeout()); if (rc) { - CNETERR("Error %d sending HELLO payload (%d) to %pISp\n", + CNETERR("Error %d sending HELLO payload (%d) to %pIScp\n", rc, hello->kshm_nips, &conn->ksnc_peeraddr); } @@ -621,7 +621,7 @@ ksocknal_send_hello_v2(struct ksock_conn *conn, struct ksock_hello_msg *hello) lnet_acceptor_timeout()); kfree(hello4); if (rc) { - CNETERR("Error %d sending HELLO hdr to %pISp\n", + CNETERR("Error %d sending HELLO hdr to %pIScp\n", rc, &conn->ksnc_peeraddr); return rc; } @@ -633,7 +633,7 @@ ksocknal_send_hello_v2(struct ksock_conn *conn, struct ksock_hello_msg *hello) hello->kshm_nips * sizeof(u32), lnet_acceptor_timeout()); if (rc) { - CNETERR("Error %d sending HELLO payload (%d) to %pISp\n", + CNETERR("Error %d sending HELLO payload (%d) to %pIScp\n", rc, hello->kshm_nips, &conn->ksnc_peeraddr); } @@ -654,7 +654,7 @@ ksocknal_send_hello_v4(struct ksock_conn *conn, struct ksock_hello_msg *hello) lnet_acceptor_timeout()); if (rc != 0) - CNETERR("Error %d sending HELLO hdr to %pISp\n", + CNETERR("Error %d sending HELLO hdr to %pIScp\n", rc, &conn->ksnc_peeraddr); return rc; } @@ -679,7 +679,7 @@ ksocknal_recv_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello, src_nid), timeout); if (rc) { - CERROR("Error %d reading rest of HELLO hdr from %pIS\n", + CERROR("Error %d reading rest of HELLO hdr from %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0 && rc != -EALREADY); goto out; @@ -687,7 +687,7 @@ ksocknal_recv_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello, /* ...and check we got what we expected */ if (hdr->type != cpu_to_le32(LNET_MSG_HELLO)) { - CERROR("Expecting a HELLO hdr, but got type %d from %pIS\n", + CERROR("Expecting a HELLO hdr, but got type %d from %pISc\n", le32_to_cpu(hdr->type), &conn->ksnc_peeraddr); rc = -EPROTO; @@ -701,7 +701,7 @@ ksocknal_recv_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello, hello->kshm_nips = le32_to_cpu(hdr->payload_length) / sizeof(u32); if (hello->kshm_nips > LNET_INTERFACES_NUM) { - CERROR("Bad nips %d from ip %pIS\n", + CERROR("Bad nips %d from ip %pISc\n", hello->kshm_nips, &conn->ksnc_peeraddr); rc = -EPROTO; goto out; @@ -713,7 +713,7 @@ ksocknal_recv_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello, rc = lnet_sock_read(sock, hello->kshm_ips, hello->kshm_nips * sizeof(u32), timeout); if (rc) { - CERROR("Error %d reading IPs from ip %pIS\n", + CERROR("Error %d reading IPs from ip %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0 && rc != -EALREADY); goto out; @@ -723,7 +723,7 @@ ksocknal_recv_hello_v1(struct ksock_conn *conn, struct ksock_hello_msg *hello, hello->kshm_ips[i] = __le32_to_cpu(hello->kshm_ips[i]); if (!hello->kshm_ips[i]) { - CERROR("Zero IP[%d] from ip %pIS\n", + CERROR("Zero IP[%d] from ip %pISc\n", i, &conn->ksnc_peeraddr); rc = -EPROTO; break; @@ -754,7 +754,7 @@ ksocknal_recv_hello_v2(struct ksock_conn *conn, struct ksock_hello_msg *hello, offsetof(struct ksock_hello_msg_nid4, kshm_src_nid), timeout); if (rc) { - CERROR("Error %d reading HELLO from %pIS\n", + CERROR("Error %d reading HELLO from %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0 && rc != -EALREADY); return rc; @@ -783,7 +783,7 @@ ksocknal_recv_hello_v2(struct ksock_conn *conn, struct ksock_hello_msg *hello, } if (hello->kshm_nips > LNET_INTERFACES_NUM) { - CERROR("Bad nips %d from ip %pIS\n", + CERROR("Bad nips %d from ip %pISc\n", hello->kshm_nips, &conn->ksnc_peeraddr); return -EPROTO; } @@ -794,7 +794,7 @@ ksocknal_recv_hello_v2(struct ksock_conn *conn, struct ksock_hello_msg *hello, rc = lnet_sock_read(sock, hello->kshm_ips, hello->kshm_nips * sizeof(u32), timeout); if (rc) { - CERROR("Error %d reading IPs from ip %pIS\n", + CERROR("Error %d reading IPs from ip %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0 && rc != -EALREADY); return rc; @@ -805,7 +805,7 @@ ksocknal_recv_hello_v2(struct ksock_conn *conn, struct ksock_hello_msg *hello, __swab32s(&hello->kshm_ips[i]); if (!hello->kshm_ips[i]) { - CERROR("Zero IP[%d] from ip %pIS\n", + CERROR("Zero IP[%d] from ip %pISc\n", i, &conn->ksnc_peeraddr); return -EPROTO; } @@ -831,7 +831,7 @@ ksocknal_recv_hello_v4(struct ksock_conn *conn, struct ksock_hello_msg *hello, offsetof(struct ksock_hello_msg, kshm_src_nid), timeout); if (rc) { - CERROR("Error %d reading HELLO from %pIS\n", + CERROR("Error %d reading HELLO from %pISc\n", rc, &conn->ksnc_peeraddr); LASSERT(rc < 0 && rc != -EALREADY); return rc; diff --git a/net/lnet/lnet/acceptor.c b/net/lnet/lnet/acceptor.c index 2306760210ae..6e219085cb53 100644 --- a/net/lnet/lnet/acceptor.c +++ b/net/lnet/lnet/acceptor.c @@ -91,37 +91,37 @@ lnet_connect_console_error(int rc, struct lnet_nid *peer_nid, switch (rc) { /* "normal" errors */ case -ECONNREFUSED: - CNETERR("Connection to %s at host %pISp was refused: check that Lustre is running on that node.\n", + CNETERR("Connection to %s at host %pIScp was refused: check that Lustre is running on that node.\n", libcfs_nidstr(peer_nid), sa); break; case -EHOSTUNREACH: case -ENETUNREACH: - CNETERR("Connection to %s at host %pIS was unreachable: the network or that node may be down, or Lustre may be misconfigured.\n", + CNETERR("Connection to %s at host %pISc was unreachable: the network or that node may be down, or Lustre may be misconfigured.\n", libcfs_nidstr(peer_nid), sa); break; case -ETIMEDOUT: - CNETERR("Connection to %s at host %pISp took too long: that node may be hung or experiencing high load.\n", + CNETERR("Connection to %s at host %pIScp took too long: that node may be hung or experiencing high load.\n", libcfs_nidstr(peer_nid), sa); break; case -ECONNRESET: LCONSOLE_ERROR_MSG(0x11b, - "Connection to %s at host %pISp was reset: is it running a compatible version of Lustre and is %s one of its NIDs?\n", + "Connection to %s at host %pIScp was reset: is it running a compatible version of Lustre and is %s one of its NIDs?\n", libcfs_nidstr(peer_nid), sa, libcfs_nidstr(peer_nid)); break; case -EPROTO: LCONSOLE_ERROR_MSG(0x11c, - "Protocol error connecting to %s at host %pISp: is it running a compatible version of Lustre?\n", + "Protocol error connecting to %s at host %pIScp: is it running a compatible version of Lustre?\n", libcfs_nidstr(peer_nid), sa); break; case -EADDRINUSE: LCONSOLE_ERROR_MSG(0x11d, - "No privileged ports available to connect to %s at host %pISp\n", + "No privileged ports available to connect to %s at host %pIScp\n", libcfs_nidstr(peer_nid), sa); break; default: LCONSOLE_ERROR_MSG(0x11e, - "Unexpected error %d connecting to %s at host %pISp\n", + "Unexpected error %d connecting to %s at host %pIScp\n", rc, libcfs_nidstr(peer_nid), sa); break; } @@ -237,7 +237,7 @@ lnet_accept(struct socket *sock, u32 magic) accept_timeout); if (rc) - CERROR("Error sending magic+version in response to LNET magic from %pIS: %d\n", + CERROR("Error sending magic+version in response to LNET magic from %pISc: %d\n", &peer, rc); return -EPROTO; } @@ -248,7 +248,7 @@ lnet_accept(struct socket *sock, u32 magic) str = "unrecognised"; LCONSOLE_ERROR_MSG(0x11f, - "Refusing connection from %pIS magic %08x: %s acceptor protocol\n", + "Refusing connection from %pISc magic %08x: %s acceptor protocol\n", &peer, magic, str); return -EPROTO; } @@ -258,7 +258,7 @@ lnet_accept(struct socket *sock, u32 magic) rc = lnet_sock_read(sock, &cr.acr_version, sizeof(cr.acr_version), accept_timeout); if (rc) { - CERROR("Error %d reading connection request version from %pIS\n", + CERROR("Error %d reading connection request version from %pISc\n", rc, &peer); return -EIO; } @@ -281,7 +281,7 @@ lnet_accept(struct socket *sock, u32 magic) rc = lnet_sock_write(sock, &cr, sizeof(cr), accept_timeout); if (rc) - CERROR("Error sending magic+version in response to version %d from %pIS: %d\n", + CERROR("Error sending magic+version in response to version %d from %pISc: %d\n", peer_version, &peer, rc); return -EPROTO; @@ -311,7 +311,7 @@ lnet_accept(struct socket *sock, u32 magic) break; } if (rc) { - CERROR("Error %d reading connection request from %pIS\n", + CERROR("Error %d reading connection request from %pISc\n", rc, &peer); return -EIO; } @@ -323,7 +323,7 @@ lnet_accept(struct socket *sock, u32 magic) if (ni) lnet_ni_decref(ni); LCONSOLE_ERROR_MSG(0x120, - "Refusing connection from %pIS for %s: No matching NI\n", + "Refusing connection from %pISc for %s: No matching NI\n", &peer, libcfs_nidstr(&nid)); return -EPERM; } @@ -332,12 +332,12 @@ lnet_accept(struct socket *sock, u32 magic) /* This catches a request for the loopback LND */ lnet_ni_decref(ni); LCONSOLE_ERROR_MSG(0x121, - "Refusing connection from %pIS for %s: NI doesn not accept IP connections\n", + "Refusing connection from %pISc for %s: NI doesn not accept IP connections\n", &peer, libcfs_nidstr(&nid)); return -EPERM; } - CDEBUG(D_NET, "Accept %s from %pIS\n", libcfs_nidstr(&nid), &peer); + CDEBUG(D_NET, "Accept %s from %pISc\n", libcfs_nidstr(&nid), &peer); rc = ni->ni_net->net_lnd->lnd_accept(ni, sock); @@ -430,7 +430,7 @@ lnet_acceptor(void *arg) if (secure && rpc_get_port((struct sockaddr *)&peer) > LNET_ACCEPTOR_MAX_RESERVED_PORT) { - CERROR("Refusing connection from %pISp: insecure port\n", + CERROR("Refusing connection from %pIScp: insecure port\n", &peer); goto failed; } @@ -438,7 +438,7 @@ lnet_acceptor(void *arg) rc = lnet_sock_read(newsock, &magic, sizeof(magic), accept_timeout); if (rc) { - CERROR("Error %d reading connection request from %pIS\n", + CERROR("Error %d reading connection request from %pISc\n", rc, &peer); goto failed; } diff --git a/net/lnet/lnet/lib-socket.c b/net/lnet/lnet/lib-socket.c index 01f375ed96a3..e6236c98261d 100644 --- a/net/lnet/lnet/lib-socket.c +++ b/net/lnet/lnet/lib-socket.c @@ -401,7 +401,7 @@ lnet_sock_connect(int interface, int local_port, * port... */ CDEBUG_LIMIT(rc == -EADDRNOTAVAIL ? D_NET : D_NETERROR, - "Error %d connecting %d -> %pISp\n", rc, + "Error %d connecting %d -> %pIScp\n", rc, local_port, peeraddr); sock_release(sock); From patchwork Fri Oct 14 21:38:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 105FBC43217 for ; Fri, 14 Oct 2022 21:38:48 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bb5FMWz1yDV; Fri, 14 Oct 2022 14:38:47 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0B927kZz21HL for ; Fri, 14 Oct 2022 14:38:25 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 1CE4810084E2; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 19859DD53E; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:05 -0400 Message-Id: <1665783491-13827-15-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 14/20] lustre: llog: correct llog FID and path output X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mikhail Pershin , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mikhail Pershin - fix wrong LLOG_ID-to-FID convestion to output llog FID by introducing PLOGID macro to expand llog ID for DFID format - stop printing lgl_ogen along with llog FID as it always zero since 2.3.51 and is not used anymore Fixes: e813619f0a3b ("lustre: llog: update llog print format to use FIDs") WC-bug-id: https://jira.whamcloud.com/browse/LU-15646 Lustre-commit: e28f3ee185b2ef7ba ("LU-15646 llog: correct llog FID and path output") Signed-off-by: Mikhail Pershin Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48430 Reviewed-by: Andreas Dilger Reviewed-by: Lai Siyao Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/obdclass/llog.c | 11 ++++---- fs/lustre/obdclass/llog_cat.c | 37 ++++++++----------------- fs/lustre/obdclass/llog_swab.c | 2 +- include/uapi/linux/lustre/lustre_user.h | 1 + 4 files changed, 19 insertions(+), 32 deletions(-) diff --git a/fs/lustre/obdclass/llog.c b/fs/lustre/obdclass/llog.c index 38904b6b5da5..eb8f7e5a8c48 100644 --- a/fs/lustre/obdclass/llog.c +++ b/fs/lustre/obdclass/llog.c @@ -422,7 +422,7 @@ static int llog_process_thread(void *arg) if (rc) { CERROR("%s: invalid record in llog "DFID" record for index %d/%d: rc = %d\n", loghandle2name(loghandle), - PFID(&loghandle->lgh_id.lgl_oi.oi_fid), + PLOGID(&loghandle->lgh_id), rec->lrh_len, index, rc); /* * the block seem to be corrupted, let's try @@ -448,7 +448,7 @@ static int llog_process_thread(void *arg) */ CERROR("%s: "DFID" index %u, expected %u\n", loghandle2name(loghandle), - PFID(&loghandle->lgh_id.lgl_oi.oi_fid), + PLOGID(&loghandle->lgh_id), rec->lrh_index, index); index = rec->lrh_index; } @@ -481,10 +481,9 @@ static int llog_process_thread(void *arg) } out: - CDEBUG(D_HA, "stop processing %s " DOSTID ":%x index %d count %d\n", + CDEBUG(D_HA, "stop processing %s "DFID" index %d count %d\n", ((llh->llh_flags & LLOG_F_IS_CAT) ? "catalog" : "plain"), - POSTID(&loghandle->lgh_id.lgl_oi), loghandle->lgh_id.lgl_ogen, - index, llh->llh_count); + PLOGID(&loghandle->lgh_id), index, llh->llh_count); if (cd) cd->lpcd_last_idx = last_called_index; @@ -534,7 +533,7 @@ int llog_process_or_fork(const struct lu_env *env, CDEBUG(D_OTHER, "Processing " DFID " flags 0x%03x startcat %d startidx %d first_idx %d last_idx %d read_mode %d\n", - PFID(&loghandle->lgh_id.lgl_oi.oi_fid), flags, + PLOGID(&loghandle->lgh_id), flags, (flags & LLOG_F_IS_CAT) && d ? d->lpd_startcat : -1, (flags & LLOG_F_IS_CAT) && d ? d->lpd_startidx : -1, cd ? cd->lpcd_first_idx : -1, cd ? cd->lpcd_last_idx : -1, diff --git a/fs/lustre/obdclass/llog_cat.c b/fs/lustre/obdclass/llog_cat.c index 753422be3185..95bfa65d25b1 100644 --- a/fs/lustre/obdclass/llog_cat.c +++ b/fs/lustre/obdclass/llog_cat.c @@ -77,18 +77,11 @@ static int llog_cat_id2handle(const struct lu_env *env, if (ostid_id(&cgl->lgl_oi) == ostid_id(&logid->lgl_oi) && ostid_seq(&cgl->lgl_oi) == ostid_seq(&logid->lgl_oi)) { - if (cgl->lgl_ogen != logid->lgl_ogen) { - CWARN("%s: log " DFID " generation %x != %x\n", - loghandle2name(loghandle), - PFID(&logid->lgl_oi.oi_fid), - cgl->lgl_ogen, logid->lgl_ogen); - continue; - } *res = llog_handle_get(loghandle); if (!*res) { CERROR("%s: log "DFID" refcount is zero!\n", loghandle2name(loghandle), - PFID(&logid->lgl_oi.oi_fid)); + PLOGID(logid)); continue; } loghandle->u.phd.phd_cat_handle = cathandle; @@ -101,9 +94,8 @@ static int llog_cat_id2handle(const struct lu_env *env, rc = llog_open(env, cathandle->lgh_ctxt, &loghandle, logid, NULL, LLOG_OPEN_EXISTS); if (rc < 0) { - CERROR("%s: error opening log id " DFID ":%x: rc = %d\n", - loghandle2name(cathandle), PFID(&logid->lgl_oi.oi_fid), - logid->lgl_ogen, rc); + CERROR("%s: error opening log id "DFID": rc = %d\n", + loghandle2name(cathandle), PLOGID(logid), rc); return rc; } @@ -153,25 +145,20 @@ static int llog_cat_process_common(const struct lu_env *env, if (rec->lrh_type != le32_to_cpu(LLOG_LOGID_MAGIC)) { rc = -EINVAL; - CWARN("%s: invalid record in catalog " DFID ":%x: rc = %d\n", - loghandle2name(cat_llh), - PFID(&cat_llh->lgh_id.lgl_oi.oi_fid), - cat_llh->lgh_id.lgl_ogen, rc); - + CWARN("%s: invalid record in catalog "DFID": rc = %d\n", + loghandle2name(cat_llh), PLOGID(&cat_llh->lgh_id), rc); return rc; } CDEBUG(D_HA, - "processing log " DFID ":%x at index %u of catalog " DFID "\n", - PFID(&lir->lid_id.lgl_oi.oi_fid), lir->lid_id.lgl_ogen, - le32_to_cpu(rec->lrh_index), - PFID(&cat_llh->lgh_id.lgl_oi.oi_fid)); + "processing log "DFID" at index %u of catalog "DFID"\n", + PLOGID(&lir->lid_id), le32_to_cpu(rec->lrh_index), + PLOGID(&cat_llh->lgh_id)); rc = llog_cat_id2handle(env, cat_llh, llhp, &lir->lid_id); if (rc) { - CWARN("%s: can't find llog handle " DFID ":%x: rc = %d\n", - loghandle2name(cat_llh), - PFID(&lir->lid_id.lgl_oi.oi_fid), - lir->lid_id.lgl_ogen, rc); + CWARN("%s: can't find llog handle "DFID": rc = %d\n", + loghandle2name(cat_llh), PLOGID(&lir->lid_id), + rc); return rc; } @@ -238,7 +225,7 @@ static int llog_cat_process_or_fork(const struct lu_env *env, CWARN("%s: catlog " DFID " crosses index zero\n", loghandle2name(cat_llh), - PFID(&cat_llh->lgh_id.lgl_oi.oi_fid)); + PLOGID(&cat_llh->lgh_id)); /*startcat = 0 is default value for general processing */ if ((startcat != LLOG_CAT_FIRST && startcat >= llh->llh_cat_idx) || !startcat) { diff --git a/fs/lustre/obdclass/llog_swab.c b/fs/lustre/obdclass/llog_swab.c index fcc2a48b50b0..5d2936429b9f 100644 --- a/fs/lustre/obdclass/llog_swab.c +++ b/fs/lustre/obdclass/llog_swab.c @@ -45,7 +45,7 @@ static void print_llogd_body(struct llogd_body *d) { CDEBUG(D_OTHER, "llogd body: %p\n", d); CDEBUG(D_OTHER, "\tlgd_logid.lgl_oi: " DFID "\n", - PFID(&d->lgd_logid.lgl_oi.oi_fid)); + PLOGID(&d->lgd_logid)); CDEBUG(D_OTHER, "\tlgd_logid.lgl_ogen: %#x\n", d->lgd_logid.lgl_ogen); CDEBUG(D_OTHER, "\tlgd_ctxt_idx: %#x\n", d->lgd_ctxt_idx); CDEBUG(D_OTHER, "\tlgd_llh_flags: %#x\n", d->lgd_llh_flags); diff --git a/include/uapi/linux/lustre/lustre_user.h b/include/uapi/linux/lustre/lustre_user.h index 6577202517af..db18cd5f1fa8 100644 --- a/include/uapi/linux/lustre/lustre_user.h +++ b/include/uapi/linux/lustre/lustre_user.h @@ -987,6 +987,7 @@ static inline void obd_uuid2fsname(char *buf, char *uuid, int buflen) */ #define SFID "0x%llx:0x%x:0x%x" #define RFID(fid) &((fid)->f_seq), &((fid)->f_oid), &((fid)->f_ver) +#define PLOGID(logid) ((unsigned long long)(logid)->lgl_oi.oi.oi_seq, (__u32)(logid)->lgl_oi.oi.oi_id, 0) /********* Quotas **********/ From patchwork Fri Oct 14 21:38:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 041FEC433FE for ; Fri, 14 Oct 2022 21:38:51 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bg3ynXz21HS; Fri, 14 Oct 2022 14:38:51 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0BB0LfCz21HL for ; Fri, 14 Oct 2022 14:38:26 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 207D310087C5; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 1D2E7DD535; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:06 -0400 Message-Id: <1665783491-13827-16-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 15/20] lnet: o2iblnd: fix handling of RDMA_CM_EVENT_UNREACHABLE X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Serguei Smirnov , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Serguei Smirnov RDMA_CM_EVENT_UNREACHABLE may be received not only when connection is being connected, but also when it is being closed. Fix handing of this event accordingly. WC-bug-id: https://jira.whamcloud.com/browse/LU-15885 Lustre-commit: 3925b1669d519e6c0 ("LU-15885 o2iblnd: fix handling of RDMA_CM_EVENT_UNREACHABLE") Signed-off-by: Serguei Smirnov Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48492 Reviewed-by: Frank Sehr Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c index 30e77c0382f1..919b83d5c6e2 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -3171,13 +3171,17 @@ kiblnd_cm_callback(struct rdma_cm_id *cmid, struct rdma_cm_event *event) return event->status; /* rc destroys cmid */ case RDMA_CM_EVENT_UNREACHABLE: - conn = (struct kib_conn *)cmid->context; - LASSERT(conn->ibc_state == IBLND_CONN_ACTIVE_CONNECT || - conn->ibc_state == IBLND_CONN_PASSIVE_WAIT); - CNETERR("%s: UNREACHABLE %d\n", - libcfs_nid2str(conn->ibc_peer->ibp_nid), event->status); - kiblnd_connreq_done(conn, -ENETDOWN); - kiblnd_conn_decref(conn); + CNETERR("%s: UNREACHABLE %d, ibc_state: %d\n", + libcfs_nid2str(conn->ibc_peer->ibp_nid), + event->status, + conn->ibc_state); + LASSERT(conn->ibc_state != IBLND_CONN_ESTABLISHED && + conn->ibc_state != IBLND_CONN_INIT); + if (conn->ibc_state == IBLND_CONN_ACTIVE_CONNECT || + conn->ibc_state == IBLND_CONN_PASSIVE_WAIT) { + kiblnd_connreq_done(conn, -ENETDOWN); + kiblnd_conn_decref(conn); + } return 0; case RDMA_CM_EVENT_CONNECT_ERROR: From patchwork Fri Oct 14 21:38:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D3C2C4332F for ; Fri, 14 Oct 2022 21:38:51 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bg01VZz1y5c; Fri, 14 Oct 2022 14:38:50 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0BB55jGz21Bx for ; Fri, 14 Oct 2022 14:38:26 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 228C3100CA0F; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 2034ADD6C6; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:07 -0400 Message-Id: <1665783491-13827-17-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 16/20] lnet: socklnd: remove remnants of tcp bonding X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Mr NeilBrown ->ksnp_n_passive_ips is now always zero, so remove it and all uses of it. ->ksnp_passive_ips is gone too, as is ksocknal_ip2iface(). WC-bug-id: https://jira.whamcloud.com/browse/LU-13641 Lustre-commit: 3630e1eaf9db562a1 ("LU-13641 socklnd: remove remnants of tcp bonding") Signed-off-by: Mr NeilBrown Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48568 Reviewed-by: James Simmons Reviewed-by: Serguei Smirnov Reviewed-by: Frank Sehr Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 73 +++----------------------------- net/lnet/klnds/socklnd/socklnd.h | 4 -- 2 files changed, 6 insertions(+), 71 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 8999580b67b4..9c8b75f0b2a2 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -46,20 +46,6 @@ static struct lnet_lnd the_ksocklnd; struct ksock_nal_data ksocknal_data; -static struct ksock_interface * -ksocknal_ip2iface(struct lnet_ni *ni, struct sockaddr *addr) -{ - struct ksock_net *net = ni->ni_data; - struct ksock_interface *iface; - - iface = &net->ksnn_interface; - - if (rpc_cmp_addr((struct sockaddr *)&iface->ksni_addr, addr)) - return iface; - - return NULL; -} - static struct ksock_interface * ksocknal_index2iface(struct lnet_ni *ni, int index) { @@ -281,28 +267,6 @@ ksocknal_find_peer(struct lnet_ni *ni, struct lnet_processid *id) static void ksocknal_unlink_peer_locked(struct ksock_peer_ni *peer_ni) { - int i; - struct ksock_interface *iface; - - for (i = 0; i < peer_ni->ksnp_n_passive_ips; i++) { - struct sockaddr_in sa = { .sin_family = AF_INET }; - - LASSERT(i < LNET_INTERFACES_NUM); - sa.sin_addr.s_addr = htonl(peer_ni->ksnp_passive_ips[i]); - - iface = ksocknal_ip2iface(peer_ni->ksnp_ni, - (struct sockaddr *)&sa); - /* - * All IPs in peer_ni->ksnp_passive_ips[] come from the - * interface list, therefore the call must succeed. - */ - LASSERT(iface); - - CDEBUG(D_NET, "peer_ni=%p iface=%p ksni_nroutes=%d\n", - peer_ni, iface, iface->ksni_nroutes); - iface->ksni_npeers--; - } - LASSERT(list_empty(&peer_ni->ksnp_conns)); LASSERT(!peer_ni->ksnp_conn_cb); LASSERT(!peer_ni->ksnp_closing); @@ -320,7 +284,6 @@ ksocknal_get_peer_info(struct lnet_ni *ni, int index, struct ksock_peer_ni *peer_ni; struct ksock_conn_cb *conn_cb; int i; - int j; int rc = -ENOENT; read_lock(&ksocknal_data.ksnd_global_lock); @@ -328,12 +291,11 @@ ksocknal_get_peer_info(struct lnet_ni *ni, int index, hash_for_each(ksocknal_data.ksnd_peers, i, peer_ni, ksnp_list) { if (peer_ni->ksnp_ni != ni) continue; + if (index-- > 0) + continue; - if (!peer_ni->ksnp_n_passive_ips && - !peer_ni->ksnp_conn_cb) { - if (index-- > 0) - continue; - + conn_cb = peer_ni->ksnp_conn_cb; + if (!conn_cb) { *id = peer_ni->ksnp_id; *myip = 0; *peer_ip = 0; @@ -341,29 +303,7 @@ ksocknal_get_peer_info(struct lnet_ni *ni, int index, *conn_count = 0; *share_count = 0; rc = 0; - goto out; - } - - for (j = 0; j < peer_ni->ksnp_n_passive_ips; j++) { - if (index-- > 0) - continue; - - *id = peer_ni->ksnp_id; - *myip = peer_ni->ksnp_passive_ips[j]; - *peer_ip = 0; - *port = 0; - *conn_count = 0; - *share_count = 0; - rc = 0; - goto out; - } - - if (peer_ni->ksnp_conn_cb) { - if (index-- > 0) - continue; - - conn_cb = peer_ni->ksnp_conn_cb; - + } else { *id = peer_ni->ksnp_id; if (conn_cb->ksnr_addr.ss_family == AF_INET) { struct sockaddr_in *sa; @@ -383,10 +323,9 @@ ksocknal_get_peer_info(struct lnet_ni *ni, int index, } *conn_count = conn_cb->ksnr_conn_count; *share_count = 1; - goto out; } + break; } -out: read_unlock(&ksocknal_data.ksnd_global_lock); return rc; } diff --git a/net/lnet/klnds/socklnd/socklnd.h b/net/lnet/klnds/socklnd/socklnd.h index 93368bd4139f..dcb4b2952f8e 100644 --- a/net/lnet/klnds/socklnd/socklnd.h +++ b/net/lnet/klnds/socklnd/socklnd.h @@ -438,10 +438,6 @@ struct ksock_peer_ni { */ time64_t ksnp_send_keepalive; /* time to send keepalive */ struct lnet_ni *ksnp_ni; /* which network */ - int ksnp_n_passive_ips; /* # of... */ - - /* preferred local interfaces */ - u32 ksnp_passive_ips[LNET_INTERFACES_NUM]; }; struct ksock_connreq { From patchwork Fri Oct 14 21:38:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E8A07C433FE for ; Fri, 14 Oct 2022 21:38:54 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bk4hZDz1y2t; Fri, 14 Oct 2022 14:38:54 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0BC3QC9z21J5 for ; Fri, 14 Oct 2022 14:38:27 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 25D78100CA13; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 246B6DD53C; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:08 -0400 Message-Id: <1665783491-13827-18-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 17/20] lnet: Router test interop check and aarch fix X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Horn , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Chris Horn Enabling routing may fail on nodes with small amount of memory (like aarch config). Define small number of router buffers to work around this issue. Modify the functions which calculate the number of buffers to allow small sizes to be specified via parameters. WC-bug-id: https://jira.whamcloud.com/browse/LU-15595 Lustre-commit: 1aba6b0d9b661d369 ("LU-15595 tests: Router test interop check and aarch fix") Signed-off-by: Chris Horn Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48578 Reviewed-by: Frank Sehr Reviewed-by: Shaun Tancheff Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/lnet/router.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/net/lnet/lnet/router.c b/net/lnet/lnet/router.c index 5d1e5a05a9fb..ee4f1d84dc9f 100644 --- a/net/lnet/lnet/router.c +++ b/net/lnet/lnet/router.c @@ -1405,11 +1405,15 @@ lnet_nrb_tiny_calculate(void) return -EINVAL; } - if (tiny_router_buffers > 0) + if (tiny_router_buffers > 0) { + if (tiny_router_buffers < LNET_NRB_TINY_MIN) + CWARN("tiny_router_buffers=%d less than recommended minimum %d\n", + tiny_router_buffers, LNET_NRB_TINY_MIN); nrbs = tiny_router_buffers; + } nrbs /= LNET_CPT_NUMBER; - return max(nrbs, LNET_NRB_TINY_MIN); + return max(nrbs, 1); } static int @@ -1424,11 +1428,15 @@ lnet_nrb_small_calculate(void) return -EINVAL; } - if (small_router_buffers > 0) + if (small_router_buffers > 0) { + if (small_router_buffers < LNET_NRB_SMALL_MIN) + CWARN("small_router_buffers=%d less than recommended minimum %d\n", + small_router_buffers, LNET_NRB_SMALL_MIN); nrbs = small_router_buffers; + } nrbs /= LNET_CPT_NUMBER; - return max(nrbs, LNET_NRB_SMALL_MIN); + return max(nrbs, 1); } static int @@ -1443,11 +1451,15 @@ lnet_nrb_large_calculate(void) return -EINVAL; } - if (large_router_buffers > 0) + if (large_router_buffers > 0) { + if (large_router_buffers < LNET_NRB_LARGE_MIN) + CWARN("large_router_buffers=%d less than recommended minimum %d\n", + large_router_buffers, LNET_NRB_LARGE_MIN); nrbs = large_router_buffers; + } nrbs /= LNET_CPT_NUMBER; - return max(nrbs, LNET_NRB_LARGE_MIN); + return max(nrbs, 1); } int From patchwork Fri Oct 14 21:38:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 315B7C43217 for ; Fri, 14 Oct 2022 21:38:58 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bn6BPXz21JS; Fri, 14 Oct 2022 14:38:57 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0BD19dPz21D4 for ; Fri, 14 Oct 2022 14:38:28 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 2D9B5100CA14; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 29A37DD53E; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:09 -0400 Message-Id: <1665783491-13827-19-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 18/20] lnet: o2iblnd: fix deadline for tx on peer queue X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Serguei Smirnov , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Serguei Smirnov In o2iblnd, deadline is checked for txs on peer queue, but not set prior to adding the tx to the queue. This may cause the tx to be dropped unnecessarily with "Timed out tx for ..." warning. Fix it by setting the tx_deadline when adding tx to peer queue. WC-bug-id: https://jira.whamcloud.com/browse/LU-16184 Lustre-commit: 4c89ee7d7b098c7f1 ("LU-16184 o2iblnd: fix deadline for tx on peer queue") Signed-off-by: Serguei Smirnov Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48640 Reviewed-by: Cyril Bordage Reviewed-by: Frank Sehr Reviewed-by: Amir Shehata Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c index 919b83d5c6e2..6f040964121c 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -1422,6 +1422,7 @@ kiblnd_launch_tx(struct lnet_ni *ni, struct kib_tx *tx, lnet_nid_t nid) int rc; int i; struct lnet_ioctl_config_o2iblnd_tunables *tunables; + s64 timeout_ns; /* * If I get here, I've committed to send, so I complete the tx with @@ -1450,6 +1451,7 @@ kiblnd_launch_tx(struct lnet_ni *ni, struct kib_tx *tx, lnet_nid_t nid) return; } + timeout_ns = kiblnd_timeout() * NSEC_PER_SEC; read_unlock(g_lock); /* Re-try with a write lock */ write_lock(g_lock); @@ -1459,9 +1461,12 @@ kiblnd_launch_tx(struct lnet_ni *ni, struct kib_tx *tx, lnet_nid_t nid) if (list_empty(&peer_ni->ibp_conns)) { /* found a peer_ni, but it's still connecting... */ LASSERT(kiblnd_peer_connecting(peer_ni)); - if (tx) + if (tx) { + tx->tx_deadline = ktime_add_ns(ktime_get(), + timeout_ns); list_add_tail(&tx->tx_list, &peer_ni->ibp_tx_queue); + } write_unlock_irqrestore(g_lock, flags); } else { conn = kiblnd_get_conn_locked(peer_ni); @@ -1498,9 +1503,12 @@ kiblnd_launch_tx(struct lnet_ni *ni, struct kib_tx *tx, lnet_nid_t nid) if (list_empty(&peer2->ibp_conns)) { /* found a peer_ni, but it's still connecting... */ LASSERT(kiblnd_peer_connecting(peer2)); - if (tx) + if (tx) { + tx->tx_deadline = ktime_add_ns(ktime_get(), + timeout_ns); list_add_tail(&tx->tx_list, &peer2->ibp_tx_queue); + } write_unlock_irqrestore(g_lock, flags); } else { conn = kiblnd_get_conn_locked(peer2); @@ -1525,8 +1533,10 @@ kiblnd_launch_tx(struct lnet_ni *ni, struct kib_tx *tx, lnet_nid_t nid) /* always called with a ref on ni, which prevents ni being shutdown */ LASSERT(!((struct kib_net *)ni->ni_data)->ibn_shutdown); - if (tx) + if (tx) { + tx->tx_deadline = ktime_add_ns(ktime_get(), timeout_ns); list_add_tail(&tx->tx_list, &peer_ni->ibp_tx_queue); + } kiblnd_peer_addref(peer_ni); hash_add(kiblnd_data.kib_peers, &peer_ni->ibp_list, nid); From patchwork Fri Oct 14 21:38:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0AF4AC4332F for ; Fri, 14 Oct 2022 21:38:57 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bm5Lg3z21Jp; Fri, 14 Oct 2022 14:38:56 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0BD6Wfgz21D4 for ; Fri, 14 Oct 2022 14:38:28 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 2FCC3100CA15; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 2E049DD535; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:10 -0400 Message-Id: <1665783491-13827-20-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 19/20] lnet: o2iblnd: detect link state to set fatal error on ni X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Serguei Smirnov , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Serguei Smirnov To avoid selecting lnet ni which corresponds to a downed link for sending, add a mechanism for detecting ip-layer link events in o2iblnd. On ip link up/down events, find corresponding ni and toggle ni_fatal_error_on flag. This complements the existing mechanism for ib-layer link event handling. WC-bug-id: https://jira.whamcloud.com/browse/LU-16051 Lustre-commit: 30d73908087d5b2f0 ("LU-16051 o2iblnd: detect link state to set fatal error on ni") Signed-off-by: Serguei Smirnov Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48644 Reviewed-by: Frank Sehr Reviewed-by: Chris Horn Reviewed-by: Cyril Bordage Reviewed-by: James Simmons Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.c | 219 ++++++++++++++++++++++++++----- 1 file changed, 186 insertions(+), 33 deletions(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.c b/net/lnet/klnds/o2iblnd/o2iblnd.c index c713528a7e7c..d5ca1a3dd25c 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd.c @@ -2527,6 +2527,184 @@ void kiblnd_destroy_dev(struct kib_dev *dev) kfree(dev); } +static struct kib_dev * +kiblnd_dev_search(char *ifname) +{ + struct kib_dev *alias = NULL; + struct kib_dev *dev; + char *colon; + char *colon2; + + colon = strchr(ifname, ':'); + list_for_each_entry(dev, &kiblnd_data.kib_devs, ibd_list) { + if (strcmp(&dev->ibd_ifname[0], ifname) == 0) + return dev; + + if (alias) + continue; + + colon2 = strchr(dev->ibd_ifname, ':'); + if (colon) + *colon = 0; + if (colon2) + *colon2 = 0; + + if (strcmp(&dev->ibd_ifname[0], ifname) == 0) + alias = dev; + + if (colon) + *colon = ':'; + if (colon2) + *colon2 = ':'; + } + return alias; +} + +static int +kiblnd_handle_link_state_change(struct net_device *dev, + unsigned char operstate) +{ + struct lnet_ni *ni = NULL; + struct kib_dev *event_kibdev; + struct kib_net *net; + struct kib_net *cnxt; + bool link_down = !(operstate == IF_OPER_UP); + struct in_device *in_dev; + bool found_ip = false; + const struct in_ifaddr *ifa; + + event_kibdev = kiblnd_dev_search(dev->name); + + if (!event_kibdev) + goto out; + + list_for_each_entry_safe(net, cnxt, &event_kibdev->ibd_nets, ibn_list) { + found_ip = false; + + ni = net->ibn_ni; + + in_dev = __in_dev_get_rtnl(dev); + if (!in_dev) { + CDEBUG(D_NET, "Interface %s has no IPv4 status.\n", + dev->name); + CDEBUG(D_NET, "%s: set link fatal state to 1\n", + libcfs_nidstr(&net->ibn_ni->ni_nid)); + atomic_set(&ni->ni_fatal_error_on, 1); + continue; + } + in_dev_for_each_ifa_rtnl(ifa, in_dev) { + if (htonl(event_kibdev->ibd_ifip) == ifa->ifa_local) + found_ip = true; + } + + if (!found_ip) { + CDEBUG(D_NET, "Interface %s has no matching ip\n", + dev->name); + CDEBUG(D_NET, "%s: set link fatal state to 1\n", + libcfs_nidstr(&net->ibn_ni->ni_nid)); + atomic_set(&ni->ni_fatal_error_on, 1); + continue; + } + + if (link_down) { + CDEBUG(D_NET, "%s: set link fatal state to 1\n", + libcfs_nidstr(&net->ibn_ni->ni_nid)); + atomic_set(&ni->ni_fatal_error_on, link_down); + } else { + CDEBUG(D_NET, "%s: set link fatal state to %u\n", + libcfs_nidstr(&net->ibn_ni->ni_nid), + (kiblnd_get_link_status(dev) == 0)); + atomic_set(&ni->ni_fatal_error_on, + (kiblnd_get_link_status(dev) == 0)); + } + } +out: + return 0; +} + +static int +kiblnd_handle_inetaddr_change(struct in_ifaddr *ifa, unsigned long event) +{ + struct kib_dev *event_kibdev; + struct kib_net *net; + struct kib_net *cnxt; + struct net_device *event_netdev = ifa->ifa_dev->dev; + + event_kibdev = kiblnd_dev_search(event_netdev->name); + + if (!event_kibdev) + goto out; + + if (htonl(event_kibdev->ibd_ifip) != ifa->ifa_local) + goto out; + + list_for_each_entry_safe(net, cnxt, &event_kibdev->ibd_nets, + ibn_list) { + CDEBUG(D_NET, "%s: set link fatal state to %u\n", + libcfs_nidstr(&net->ibn_ni->ni_nid), + (event == NETDEV_DOWN)); + atomic_set(&net->ibn_ni->ni_fatal_error_on, + (event == NETDEV_DOWN)); + } +out: + return 0; +} + +/************************************ + * Net device notifier event handler + ************************************/ +static int kiblnd_device_event(struct notifier_block *unused, + unsigned long event, void *ptr) +{ + struct net_device *dev = netdev_notifier_info_to_dev(ptr); + unsigned char operstate; + + operstate = dev->operstate; + + CDEBUG(D_NET, "devevent: status=%ld, iface=%s ifindex %d state %u\n", + event, dev->name, dev->ifindex, operstate); + + switch (event) { + case NETDEV_UP: + case NETDEV_DOWN: + case NETDEV_CHANGE: + kiblnd_handle_link_state_change(dev, operstate); + break; + } + + return NOTIFY_OK; +} + +/************************************ + * Inetaddr notifier event handler + ************************************/ +static int kiblnd_inetaddr_event(struct notifier_block *unused, + unsigned long event, void *ptr) +{ + struct in_ifaddr *ifa = ptr; + + CDEBUG(D_NET, "addrevent: status %ld ip addr %pI4, netmask %pI4.\n", + event, &ifa->ifa_address, &ifa->ifa_mask); + + switch (event) { + case NETDEV_UP: + case NETDEV_DOWN: + case NETDEV_CHANGE: + kiblnd_handle_inetaddr_change(ifa, event); + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block kiblnd_dev_notifier_block = { + .notifier_call = kiblnd_device_event, +}; + +static struct notifier_block kiblnd_inetaddr_notifier_block = { + .notifier_call = kiblnd_inetaddr_event, +}; + static void kiblnd_base_shutdown(void) { struct kib_sched_info *sched; @@ -2535,6 +2713,11 @@ static void kiblnd_base_shutdown(void) LASSERT(list_empty(&kiblnd_data.kib_devs)); + if (kiblnd_data.kib_init == IBLND_INIT_ALL) { + unregister_netdevice_notifier(&kiblnd_dev_notifier_block); + unregister_inetaddr_notifier(&kiblnd_inetaddr_notifier_block); + } + switch (kiblnd_data.kib_init) { default: LBUG(); @@ -2723,6 +2906,9 @@ static int kiblnd_base_startup(struct net *ns) goto failed; } + register_netdevice_notifier(&kiblnd_dev_notifier_block); + register_inetaddr_notifier(&kiblnd_inetaddr_notifier_block); + /* flag everything initialised */ kiblnd_data.kib_init = IBLND_INIT_ALL; /*****************************************************/ @@ -2799,39 +2985,6 @@ static int kiblnd_dev_start_threads(struct kib_dev *dev, bool newdev, u32 *cpts, return 0; } -static struct kib_dev * -kiblnd_dev_search(char *ifname) -{ - struct kib_dev *alias = NULL; - struct kib_dev *dev; - char *colon; - char *colon2; - - colon = strchr(ifname, ':'); - list_for_each_entry(dev, &kiblnd_data.kib_devs, ibd_list) { - if (strcmp(&dev->ibd_ifname[0], ifname) == 0) - return dev; - - if (alias) - continue; - - colon2 = strchr(dev->ibd_ifname, ':'); - if (colon) - *colon = 0; - if (colon2) - *colon2 = 0; - - if (strcmp(&dev->ibd_ifname[0], ifname) == 0) - alias = dev; - - if (colon) - *colon = ':'; - if (colon2) - *colon2 = ':'; - } - return alias; -} - static int kiblnd_startup(struct lnet_ni *ni) { char *ifname = NULL; From patchwork Fri Oct 14 21:38:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 13007364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from pdx1-mailman-customer002.dreamhost.com (listserver-buz.dreamhost.com [69.163.136.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8449EC433FE for ; Fri, 14 Oct 2022 21:39:01 +0000 (UTC) Received: from pdx1-mailman-customer002.dreamhost.com (localhost [127.0.0.1]) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTP id 4Mq0Bs1mdRz1y30; Fri, 14 Oct 2022 14:39:01 -0700 (PDT) Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pdx1-mailman-customer002.dreamhost.com (Postfix) with ESMTPS id 4Mq0BF4trQz21D4 for ; Fri, 14 Oct 2022 14:38:29 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 34294100CA16; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 31851DD53C; Fri, 14 Oct 2022 17:38:14 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Fri, 14 Oct 2022 17:38:11 -0400 Message-Id: <1665783491-13827-21-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> References: <1665783491-13827-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 20/20] lnet: socklnd: limit retries on conns_per_peer mismatch X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Serguei Smirnov , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Serguei Smirnov If connection initiator has a higher conns-per-peer setting than its peer, don't try to create extra connections forever as the peer will keep rejecting them. A few retries should suffice to resolve a valid race. Fixes: 511ace4a ("lnet: socklnd: add conns_per_peer parameter") WC-bug-id: https://jira.whamcloud.com/browse/LU-16191 Lustre-commit: da893c6c9707ca3b2 ("LU-16191 socklnd: limit retries on conns_per_peer mismatch") Signed-off-by: Serguei Smirnov Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/48664 Reviewed-by: Frank Sehr Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/socklnd/socklnd.c | 1 + net/lnet/klnds/socklnd/socklnd.h | 4 ++++ net/lnet/klnds/socklnd/socklnd_cb.c | 25 +++++++++++++++++++------ 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/net/lnet/klnds/socklnd/socklnd.c b/net/lnet/klnds/socklnd/socklnd.c index 9c8b75f0b2a2..00e33c88dfaa 100644 --- a/net/lnet/klnds/socklnd/socklnd.c +++ b/net/lnet/klnds/socklnd/socklnd.c @@ -144,6 +144,7 @@ ksocknal_create_conn_cb(struct sockaddr *addr) conn_cb->ksnr_blki_conn_count = 0; conn_cb->ksnr_blko_conn_count = 0; conn_cb->ksnr_max_conns = 0; + conn_cb->ksnr_busy_retry_count = 0; return conn_cb; } diff --git a/net/lnet/klnds/socklnd/socklnd.h b/net/lnet/klnds/socklnd/socklnd.h index dcb4b2952f8e..bb68a3df596a 100644 --- a/net/lnet/klnds/socklnd/socklnd.h +++ b/net/lnet/klnds/socklnd/socklnd.h @@ -379,6 +379,7 @@ struct ksock_conn { }; #define SOCKNAL_CONN_COUNT_MAX_BITS 8 /* max conn count bits */ +#define SOCKNAL_MAX_BUSY_RETRIES 3 struct ksock_conn_cb { struct list_head ksnr_connd_list; /* chain on ksnr_connd_routes */ @@ -407,6 +408,9 @@ struct ksock_conn_cb { unsigned int ksnr_max_conns; /* conns_per_peer at * peer creation */ + unsigned int ksnr_busy_retry_count; /* counts retry attempts + * due to EALREADY rc + */ }; #define SOCKNAL_KEEPALIVE_PING 1 /* cookie for keepalive ping */ diff --git a/net/lnet/klnds/socklnd/socklnd_cb.c b/net/lnet/klnds/socklnd/socklnd_cb.c index b2da535fbfbe..f358875a2afe 100644 --- a/net/lnet/klnds/socklnd/socklnd_cb.c +++ b/net/lnet/klnds/socklnd/socklnd_cb.c @@ -1785,7 +1785,7 @@ ksocknal_connect(struct ksock_conn_cb *conn_cb) { LIST_HEAD(zombies); struct ksock_peer_ni *peer_ni = conn_cb->ksnr_peer; - int type; + int type = SOCKLND_CONN_NONE; int wanted; struct socket *sock; time64_t deadline; @@ -1863,14 +1863,18 @@ ksocknal_connect(struct ksock_conn_cb *conn_cb) goto failed; } - /* - * A +ve RC means I have to retry because I lost the connection + if (rc == EALREADY && conn_cb->ksnr_conn_count > 0) + conn_cb->ksnr_busy_retry_count += 1; + else + conn_cb->ksnr_busy_retry_count = 0; + + /* A +ve RC means I have to retry because I lost the connection * race or I have to renegotiate protocol version */ - retry_later = (rc); + retry_later = (rc != 0); if (retry_later) - CDEBUG(D_NET, "peer_ni %s: conn race, retry later.\n", - libcfs_nidstr(&peer_ni->ksnp_id.nid)); + CDEBUG(D_NET, "peer_ni %s: conn race, retry later. rc %d\n", + libcfs_nidstr(&peer_ni->ksnp_id.nid), rc); write_lock_bh(&ksocknal_data.ksnd_global_lock); } @@ -1878,6 +1882,15 @@ ksocknal_connect(struct ksock_conn_cb *conn_cb) conn_cb->ksnr_scheduled = 0; conn_cb->ksnr_connecting = 0; + if (conn_cb->ksnr_busy_retry_count >= SOCKNAL_MAX_BUSY_RETRIES && + type > SOCKLND_CONN_NONE) { + /* After so many retries due to EALREADY assume that + * the peer doesn't support as many connections as we want + */ + conn_cb->ksnr_connected |= BIT(type); + retry_later = false; + } + if (retry_later) { /* * re-queue for attention; this frees me up to handle