From patchwork Thu Jan 18 10:50:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 13522694 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96B9522F06 for ; Thu, 18 Jan 2024 10:53:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705575189; cv=none; b=EmM9b8LFVx0PIQrGN1AUYPpuD9OtFZL77u2YSsTSXV9gs4HJnQAePtSoEgNC93OZVP9ONcBokd4ZX4642Jar8Tb8Rj+DFX9cwpUdl3gTZLJ73VXZ1UFE1nIZAg9Ezh4LFUc1+wR7xYuRbW5oysM8aIC/P1n3u0IcNbsFyVQfV2w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705575189; c=relaxed/simple; bh=lARG60+yEeBfctx2lgyF7Mf/1TOgOxqBtXjvdSfY39k=; h=DKIM-Signature:Received:X-MC-Unique:Received:Received:From:To:Cc: Subject:Date:Message-ID:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:X-Scanned-By; b=DwiX/yMWYoeCZkpCemnlDddtfTAxVRMxFDC6IAMfn6krc809wzb04Z/RlxAGf/A0JBjkCNohK8otJH5S31UISzcFIn3FSaG4HuFeGLMDFMUCNi5Xvo08cg8EvmCuiyvR10dJA5HigEfUlQpo6woPxxVVTavk3cHsPKjvI8unFhs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=AU3t21js; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AU3t21js" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705575186; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zKgYZs6xoNW0t7R/S+SqB3PMTQfMwTdhZMSMbm4mr6I=; b=AU3t21jssHqPVaE5EE+QYYtxF5QmrwOTWtnI8dgKrP20kvFy5HJvsmIJH7UM3weemEd4O/ puF27Kyy6zlOCEV8QUU38A36aI+XNpElx5w7ZzNJUfb0mTiJJDaOFjsM2+ux+FjvSQ52eu Q6aB43ExXGSUx5tt1gB7W2ddyHw7gx0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-632-gLpzClq9NCKgD7aOZ2wIOw-1; Thu, 18 Jan 2024 05:53:03 -0500 X-MC-Unique: gLpzClq9NCKgD7aOZ2wIOw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EB67D1C0BB6E; Thu, 18 Jan 2024 10:53:02 +0000 (UTC) Received: from li-a71a4dcc-35d1-11b2-a85c-951838863c8d.ibm.com.com (unknown [10.72.112.62]) by smtp.corp.redhat.com (Postfix) with ESMTP id D65172166B33; Thu, 18 Jan 2024 10:52:59 +0000 (UTC) From: xiubli@redhat.com To: ceph-devel@vger.kernel.org Cc: idryomov@gmail.com, jlayton@kernel.org, vshankar@redhat.com, mchangir@redhat.com, Xiubo Li Subject: [PATCH v4 1/3] libceph: fail the sparse-read if there still has data in socket Date: Thu, 18 Jan 2024 18:50:45 +0800 Message-ID: <20240118105047.792879-2-xiubli@redhat.com> In-Reply-To: <20240118105047.792879-1-xiubli@redhat.com> References: <20240118105047.792879-1-xiubli@redhat.com> Precedence: bulk X-Mailing-List: ceph-devel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 From: Xiubo Li Once this happens that means there have bugs. URL: https://tracker.ceph.com/issues/63586 Signed-off-by: Xiubo Li --- net/ceph/osd_client.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index 9be80d01c1dc..f8029b30a3fb 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -5912,6 +5912,13 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough; case CEPH_SPARSE_READ_DATA: if (sr->sr_index >= count) { + if (sr->sr_datalen) { + pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n", + sr->sr_datalen, sr->sr_index, + count); + return -EREMOTEIO; + } + sr->sr_state = CEPH_SPARSE_READ_HDR; goto next_op; } @@ -5919,6 +5926,8 @@ static int osd_sparse_read(struct ceph_connection *con, eoff = sr->sr_extent[sr->sr_index].off; elen = sr->sr_extent[sr->sr_index].len; + sr->sr_datalen -= elen; + dout("[%d] ext %d off 0x%llx len 0x%llx\n", o->o_osd, sr->sr_index, eoff, elen); From patchwork Thu Jan 18 10:50:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 13522695 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47BAF22F06 for ; Thu, 18 Jan 2024 10:53:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705575191; cv=none; b=ILQvMq9BPjpwDsitPszd82siqDsZoKWb24Rt425yqGqpddlgQoAnRVbEHgon8eu1FHVzL34vXndhZLGxcLy0D5Nl+SdwTAr95nYFQqJAo6ZPgK3YMcshkVYhLqS6QecBtnsxozpCGJA42K1rZ37jgcC/yBgbQj6XjfWn8uQ18JU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705575191; c=relaxed/simple; bh=tlOJBCzF3WjE+x++MEry8mQ20R73UO2CO22oo9s8Qyg=; h=DKIM-Signature:Received:X-MC-Unique:Received:Received:From:To:Cc: Subject:Date:Message-ID:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:X-Scanned-By; b=nopttUhfurUHYYqyeZdF9xwbEfF8Xb5SEhRq7rEnXkHCPTEp+fOFjR2tsFVYFf1IlwTtp8YJ+Yvq/U93Soj7aGDTZ9GvvJpmXRA9UPlhScsHSUlTcfIYLeTPrbMl9CjlqbVyFtm2hUp6vsFGqsYzZB6Eo3UaEVjlH+N4J/6/aG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=exKUms8G; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="exKUms8G" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705575189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=twQBZwt/0/ZoadeigsXW1CtTVxmntH2XtZanBw6LaVQ=; b=exKUms8G8P3/WBI5NjKTTs+amthvJEcjoBZXc3R/2Q3V3FrikJVBoLPNjcxcQlyK0UA3rI 7Rspae0+9mn4qTNt3I0REqFJ/n/DcXmoSrfQAe1Df0UYuW6qfPTuy4R2C3XehNckw7PU6w hqZd/sSOajL8Sua3G42EEKpnAjlxarg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-435-AHe3jEBWPVy4RYKrpjikbQ-1; Thu, 18 Jan 2024 05:53:06 -0500 X-MC-Unique: AHe3jEBWPVy4RYKrpjikbQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6CCD0101A526; Thu, 18 Jan 2024 10:53:06 +0000 (UTC) Received: from li-a71a4dcc-35d1-11b2-a85c-951838863c8d.ibm.com.com (unknown [10.72.112.62]) by smtp.corp.redhat.com (Postfix) with ESMTP id A2C602166B33; Thu, 18 Jan 2024 10:53:03 +0000 (UTC) From: xiubli@redhat.com To: ceph-devel@vger.kernel.org Cc: idryomov@gmail.com, jlayton@kernel.org, vshankar@redhat.com, mchangir@redhat.com, Xiubo Li Subject: [PATCH v4 2/3] libceph: rename read_sparse_msg_XX to read_partial_sparse_msg_XX Date: Thu, 18 Jan 2024 18:50:46 +0800 Message-ID: <20240118105047.792879-3-xiubli@redhat.com> In-Reply-To: <20240118105047.792879-1-xiubli@redhat.com> References: <20240118105047.792879-1-xiubli@redhat.com> Precedence: bulk X-Mailing-List: ceph-devel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 From: Xiubo Li Actually the read_sparse_msg_XX functions allow to continue reading and parsing the socket buffer when handling of short receives. Just rename it with _partial_ prefixed. URL: https://tracker.ceph.com/issues/63586 Signed-off-by: Xiubo Li Acked-by: Jeff Layton --- net/ceph/messenger_v1.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index f9a50d7f0d20..4cb60bacf5f5 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -991,7 +991,7 @@ static inline int read_partial_message_section(struct ceph_connection *con, return read_partial_message_chunk(con, section, sec_len, crc); } -static int read_sparse_msg_extent(struct ceph_connection *con, u32 *crc) +static int read_partial_sparse_msg_extent(struct ceph_connection *con, u32 *crc) { struct ceph_msg_data_cursor *cursor = &con->in_msg->cursor; bool do_bounce = ceph_test_opt(from_msgr(con->msgr), RXBOUNCE); @@ -1026,7 +1026,7 @@ static int read_sparse_msg_extent(struct ceph_connection *con, u32 *crc) return 1; } -static int read_sparse_msg_data(struct ceph_connection *con) +static int read_partial_sparse_msg_data(struct ceph_connection *con) { struct ceph_msg_data_cursor *cursor = &con->in_msg->cursor; bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); @@ -1043,7 +1043,7 @@ static int read_sparse_msg_data(struct ceph_connection *con) con->v1.in_sr_len, &crc); else if (cursor->sr_resid > 0) - ret = read_sparse_msg_extent(con, &crc); + ret = read_partial_sparse_msg_extent(con, &crc); if (ret <= 0) { if (do_datacrc) @@ -1254,7 +1254,7 @@ static int read_partial_message(struct ceph_connection *con) return -EIO; if (m->sparse_read) - ret = read_sparse_msg_data(con); + ret = read_partial_sparse_msg_data(con); else if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) ret = read_partial_msg_data_bounce(con); else From patchwork Thu Jan 18 10:50:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 13522696 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95A3F22F0F for ; Thu, 18 Jan 2024 10:53:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705575197; cv=none; b=Pde60yxK5lcQ/Wn1QTpAlQNRgNZkDPP3wf67C38qMmuhsDDdVviHO94UXI2CEWUukAFhwu7RJ4Ex5j7ZGloDiwcOSyrjWq+Sc3W7cCtW/i6CLjgB9W7SM7p+qSKLFvVwb/MAFl2N14Vav8Dybqy/hJRExN5mYwe3nB85rFRQ2lI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705575197; c=relaxed/simple; bh=9aubr3nQ5w3kYoMxTBdRM2PE9DSFRr+t4EdyLmDRby8=; h=DKIM-Signature:Received:X-MC-Unique:Received:Received:From:To:Cc: Subject:Date:Message-ID:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:X-Scanned-By; b=eW6Jlr9FLeLBO5CSr94ytrk5zA6PzVDTN6ODS0Ld4xEGvCTll8qieTAMtlf/rbAt9qO2UMPUMv2p1AID9tx7oWNglogOwKl4GvChHaux4R5OXNn7oYDkodVdUNzkie8ALxuHFQF11UNuayKculiG2+3jG1Z4GozPCJgDwB+vnXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=MbmF2AhC; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MbmF2AhC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705575194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tWHtwQsevLlnlhr5I9nOk0sG2hz2p8edzgg45EIgciA=; b=MbmF2AhCBbvrq9I407xMD30Zoss71zMqCwlXApxwEGRYEKUhe3VZSrHi5F5rsPrZ3qp1oq jLMp70k62jtIpOtvdO64o2N0hIIf2fOV/B+DsHAbaDr1ZPClJUXHwKiHXD1EKriKVmz2n/ IGM86HYiLxZCejSHviJ5rT8lgAGD+cY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-286-s9LI8D2xO-27dVdEy9dJ8A-1; Thu, 18 Jan 2024 05:53:10 -0500 X-MC-Unique: s9LI8D2xO-27dVdEy9dJ8A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7234080007C; Thu, 18 Jan 2024 10:53:10 +0000 (UTC) Received: from li-a71a4dcc-35d1-11b2-a85c-951838863c8d.ibm.com.com (unknown [10.72.112.62]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3C5CB2166B35; Thu, 18 Jan 2024 10:53:06 +0000 (UTC) From: xiubli@redhat.com To: ceph-devel@vger.kernel.org Cc: idryomov@gmail.com, jlayton@kernel.org, vshankar@redhat.com, mchangir@redhat.com, Xiubo Li Subject: [PATCH v4 3/3] libceph: just wait for more data to be available on the socket Date: Thu, 18 Jan 2024 18:50:47 +0800 Message-ID: <20240118105047.792879-4-xiubli@redhat.com> In-Reply-To: <20240118105047.792879-1-xiubli@redhat.com> References: <20240118105047.792879-1-xiubli@redhat.com> Precedence: bulk X-Mailing-List: ceph-devel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.6 From: Xiubo Li The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet. This will add 'sr_total_resid' to record the total length for all data items for sparse-read message and 'sr_resid_elen' to record the current extent total length. URL: https://tracker.ceph.com/issues/63586 Signed-off-by: Xiubo Li --- include/linux/ceph/messenger.h | 1 + net/ceph/messenger_v1.c | 32 +++++++++++++++++++++----------- 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h index 2eaaabbe98cb..ca6f82abed62 100644 --- a/include/linux/ceph/messenger.h +++ b/include/linux/ceph/messenger.h @@ -231,6 +231,7 @@ struct ceph_msg_data { struct ceph_msg_data_cursor { size_t total_resid; /* across all data items */ + size_t sr_total_resid; /* across all data items for sparse-read */ struct ceph_msg_data *data; /* current data item */ size_t resid; /* bytes not yet consumed */ diff --git a/net/ceph/messenger_v1.c b/net/ceph/messenger_v1.c index 4cb60bacf5f5..2733da891688 100644 --- a/net/ceph/messenger_v1.c +++ b/net/ceph/messenger_v1.c @@ -160,7 +160,9 @@ static size_t sizeof_footer(struct ceph_connection *con) static void prepare_message_data(struct ceph_msg *msg, u32 data_len) { /* Initialize data cursor if it's not a sparse read */ - if (!msg->sparse_read) + if (msg->sparse_read) + msg->cursor.sr_total_resid = data_len; + else ceph_msg_data_cursor_init(&msg->cursor, msg, data_len); } @@ -1032,35 +1034,43 @@ static int read_partial_sparse_msg_data(struct ceph_connection *con) bool do_datacrc = !ceph_test_opt(from_msgr(con->msgr), NOCRC); u32 crc = 0; int ret = 1; + int len; if (do_datacrc) crc = con->in_data_crc; - do { - if (con->v1.in_sr_kvec.iov_base) + while (cursor->sr_total_resid) { + len = 0; + if (con->v1.in_sr_kvec.iov_base) { + len = con->v1.in_sr_kvec.iov_len; ret = read_partial_message_chunk(con, &con->v1.in_sr_kvec, con->v1.in_sr_len, &crc); - else if (cursor->sr_resid > 0) + len = con->v1.in_sr_kvec.iov_len - len; + } else if (cursor->sr_resid > 0) { + len = cursor->sr_resid; ret = read_partial_sparse_msg_extent(con, &crc); - - if (ret <= 0) { - if (do_datacrc) - con->in_data_crc = crc; - return ret; + len -= cursor->sr_resid; } + cursor->sr_total_resid -= len; + if (ret <= 0) + break; memset(&con->v1.in_sr_kvec, 0, sizeof(con->v1.in_sr_kvec)); ret = con->ops->sparse_read(con, cursor, (char **)&con->v1.in_sr_kvec.iov_base); + if (ret <= 0) { + ret = ret ? : 1; /* must return > 0 to indicate success */ + break; + } con->v1.in_sr_len = ret; - } while (ret > 0); + } if (do_datacrc) con->in_data_crc = crc; - return ret < 0 ? ret : 1; /* must return > 0 to indicate success */ + return ret; } static int read_partial_msg_data(struct ceph_connection *con)