From patchwork Thu Jul 28 10:12:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 9251057 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 08E5A6077C for ; Thu, 28 Jul 2016 10:17:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE9512621F for ; Thu, 28 Jul 2016 10:17:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E2CA526861; Thu, 28 Jul 2016 10:17:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 30FFD2621F for ; Thu, 28 Jul 2016 10:17:27 +0000 (UTC) Received: from localhost ([::1]:52143 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bSiNa-0006eg-5l for patchwork-qemu-devel@patchwork.kernel.org; Thu, 28 Jul 2016 06:17:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59586) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bSiJH-0002vV-Ob for qemu-devel@nongnu.org; Thu, 28 Jul 2016 06:13:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bSiJF-0000Ml-Sv for qemu-devel@nongnu.org; Thu, 28 Jul 2016 06:12:59 -0400 Received: from [59.151.112.132] (port=52428 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bSiJF-0000Jl-0c for qemu-devel@nongnu.org; Thu, 28 Jul 2016 06:12:57 -0400 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="9251084" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Jul 2016 18:12:48 +0800 Received: from G08CNEXCHPEKD02.g08.fujitsu.local (unknown [10.167.33.83]) by cn.fujitsu.com (Postfix) with ESMTP id 8BAC442984E7; Thu, 28 Jul 2016 18:12:45 +0800 (CST) Received: from G08FNSTD140215.g08.fujitsu.local (10.167.226.56) by G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.279.2; Thu, 28 Jul 2016 18:12:45 +0800 From: Zhang Chen To: qemu devel , Jason Wang Date: Thu, 28 Jul 2016 18:12:23 +0800 Message-ID: <1469700748-19754-5-git-send-email-zhangchen.fnst@cn.fujitsu.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1469700748-19754-1-git-send-email-zhangchen.fnst@cn.fujitsu.com> References: <1469700748-19754-1-git-send-email-zhangchen.fnst@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.56] X-yoursite-MailScanner-ID: 8BAC442984E7.AF8AE X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: zhangchen.fnst@cn.fujitsu.com X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Subject: [Qemu-devel] [PATCH V11 4/9] colo-compare: track connection and enqueue packet X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Li Zhijian , "eddie . dong" , "Dr . David Alan Gilbert" , Zhang Chen , zhanghailiang Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP In this patch we use kernel jhash table to track connection, and then enqueue net packet like this: + CompareState ++ | | +---------------+ +---------------+ +---------------+ |conn list +--->conn +--------->conn | +---------------+ +---------------+ +---------------+ | | | | | | +---------------+ +---v----+ +---v----+ +---v----+ +---v----+ |primary | |secondary |primary | |secondary |packet | |packet + |packet | |packet + +--------+ +--------+ +--------+ +--------+ | | | | +---v----+ +---v----+ +---v----+ +---v----+ |primary | |secondary |primary | |secondary |packet | |packet + |packet | |packet + +--------+ +--------+ +--------+ +--------+ | | | | +---v----+ +---v----+ +---v----+ +---v----+ |primary | |secondary |primary | |secondary |packet | |packet + |packet | |packet + +--------+ +--------+ +--------+ +--------+ We use conn_list to record connection info. When we want to enqueue a packet, firstly get the connection from connection_track_table. then push the packet to g_queue(pri/sec) in it's own conn. Signed-off-by: Zhang Chen Signed-off-by: Li Zhijian Signed-off-by: Wen Congyang --- net/colo-base.c | 108 +++++++++++++++++++++++++++++++++++++++++++++++++++++ net/colo-base.h | 27 ++++++++++++++ net/colo-compare.c | 70 +++++++++++++++++++++++++++++----- 3 files changed, 195 insertions(+), 10 deletions(-) diff --git a/net/colo-base.c b/net/colo-base.c index f5d5de9..7e91dec 100644 --- a/net/colo-base.c +++ b/net/colo-base.c @@ -16,6 +16,29 @@ #include "qemu/error-report.h" #include "net/colo-base.h" +uint32_t connection_key_hash(const void *opaque) +{ + const ConnectionKey *key = opaque; + uint32_t a, b, c; + + /* Jenkins hash */ + a = b = c = JHASH_INITVAL + sizeof(*key); + a += key->src.s_addr; + b += key->dst.s_addr; + c += (key->src_port | key->dst_port << 16); + __jhash_mix(a, b, c); + + a += key->ip_proto; + __jhash_final(a, b, c); + + return c; +} + +int connection_key_equal(const void *key1, const void *key2) +{ + return memcmp(key1, key2, sizeof(ConnectionKey)) == 0; +} + int parse_packet_early(Packet *pkt) { int network_length; @@ -47,6 +70,62 @@ int parse_packet_early(Packet *pkt) return 0; } +void fill_connection_key(Packet *pkt, ConnectionKey *key) +{ + uint32_t tmp_ports; + + key->ip_proto = pkt->ip->ip_p; + + switch (key->ip_proto) { + case IPPROTO_TCP: + case IPPROTO_UDP: + case IPPROTO_DCCP: + case IPPROTO_ESP: + case IPPROTO_SCTP: + case IPPROTO_UDPLITE: + tmp_ports = *(uint32_t *)(pkt->transport_layer); + key->src = pkt->ip->ip_src; + key->dst = pkt->ip->ip_dst; + key->src_port = ntohs(tmp_ports & 0xffff); + key->dst_port = ntohs(tmp_ports >> 16); + break; + case IPPROTO_AH: + tmp_ports = *(uint32_t *)(pkt->transport_layer + 4); + key->src = pkt->ip->ip_src; + key->dst = pkt->ip->ip_dst; + key->src_port = ntohs(tmp_ports & 0xffff); + key->dst_port = ntohs(tmp_ports >> 16); + break; + default: + key->src_port = 0; + key->dst_port = 0; + break; + } +} + +Connection *connection_new(ConnectionKey *key) +{ + Connection *conn = g_slice_new(Connection); + + conn->ip_proto = key->ip_proto; + conn->processing = false; + g_queue_init(&conn->primary_list); + g_queue_init(&conn->secondary_list); + + return conn; +} + +void connection_destroy(void *opaque) +{ + Connection *conn = opaque; + + g_queue_foreach(&conn->primary_list, packet_destroy, NULL); + g_queue_free(&conn->primary_list); + g_queue_foreach(&conn->secondary_list, packet_destroy, NULL); + g_queue_free(&conn->secondary_list); + g_slice_free(Connection, conn); +} + Packet *packet_new(const void *data, int size) { Packet *pkt = g_slice_new(Packet); @@ -72,3 +151,32 @@ void connection_hashtable_reset(GHashTable *connection_track_table) { g_hash_table_remove_all(connection_track_table); } + +/* if not found, create a new connection and add to hash table */ +Connection *connection_get(GHashTable *connection_track_table, + ConnectionKey *key, + uint32_t *hashtable_size) +{ + Connection *conn = g_hash_table_lookup(connection_track_table, key); + + if (conn == NULL) { + ConnectionKey *new_key = g_memdup(key, sizeof(*key)); + + conn = connection_new(key); + + (*hashtable_size) += 1; + if (*hashtable_size > HASHTABLE_MAX_SIZE) { + error_report("colo proxy connection hashtable full, clear it"); + connection_hashtable_reset(connection_track_table); + /* + * when hashtable_size == 0, clear the conn_list + * in place where be called. + */ + *hashtable_size = 0; + } + + g_hash_table_insert(connection_track_table, new_key, conn); + } + + return conn; +} diff --git a/net/colo-base.h b/net/colo-base.h index 48835e7..cc60405 100644 --- a/net/colo-base.h +++ b/net/colo-base.h @@ -30,7 +30,34 @@ typedef struct Packet { int size; } Packet; +typedef struct ConnectionKey { + /* (src, dst) must be grouped, in the same way than in IP header */ + struct in_addr src; + struct in_addr dst; + uint16_t src_port; + uint16_t dst_port; + uint8_t ip_proto; +} QEMU_PACKED ConnectionKey; + +typedef struct Connection { + /* connection primary send queue: element type: Packet */ + GQueue primary_list; + /* connection secondary send queue: element type: Packet */ + GQueue secondary_list; + /* flag to enqueue unprocessed_connections */ + bool processing; + uint8_t ip_proto; +} Connection; + +uint32_t connection_key_hash(const void *opaque); +int connection_key_equal(const void *opaque1, const void *opaque2); int parse_packet_early(Packet *pkt); +void fill_connection_key(Packet *pkt, ConnectionKey *key); +Connection *connection_new(ConnectionKey *key); +void connection_destroy(void *opaque); +Connection *connection_get(GHashTable *connection_track_table, + ConnectionKey *key, + uint32_t *hashtable_size); void connection_hashtable_reset(GHashTable *connection_track_table); Packet *packet_new(const void *data, int size); void packet_destroy(void *opaque, void *user_data); diff --git a/net/colo-compare.c b/net/colo-compare.c index 32357f7..4807f55 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -72,6 +72,11 @@ typedef struct CompareState { SocketReadState pri_rs; SocketReadState sec_rs; + /* connection list: the connections belonged to this NIC could be found + * in this list. + * element type: Connection + */ + GQueue conn_list; /* hashtable to save connection */ GHashTable *connection_track_table; /* proxy current hash size */ @@ -91,13 +96,30 @@ static int compare_chr_send(CharDriverState *out, const uint8_t *buf, uint32_t size); +static void colo_rm_connection(void *opaque, void *user_data) +{ + Connection *conn = opaque; + Packet *pkt = NULL; + + while (!g_queue_is_empty(&conn->primary_list)) { + pkt = g_queue_pop_head(&conn->primary_list); + packet_destroy(pkt, NULL); + } + while (!g_queue_is_empty(&conn->secondary_list)) { + pkt = g_queue_pop_head(&conn->secondary_list); + packet_destroy(pkt, NULL); + } +} + /* * Return 0 on success, if return -1 means the pkt * is unsupported(arp and ipv6) and will be sent later */ static int packet_enqueue(CompareState *s, int mode) { + ConnectionKey key = {{ 0 } }; Packet *pkt = NULL; + Connection *conn; if (mode == PRIMARY_IN) { pkt = packet_new(s->pri_rs.buf, s->pri_rs.packet_len); @@ -110,17 +132,38 @@ static int packet_enqueue(CompareState *s, int mode) pkt = NULL; return -1; } - /* TODO: get connection key from pkt */ + fill_connection_key(pkt, &key); - /* - * TODO: use connection key get conn from - * connection_track_table - */ + conn = connection_get(s->connection_track_table, + &key, + &s->hashtable_size); - /* - * TODO: insert pkt to it's conn->primary_list - * or conn->secondary_list - */ + if (!s->hashtable_size) { + g_queue_foreach(&s->conn_list, colo_rm_connection, NULL); + } + + if (!conn->processing) { + g_queue_push_tail(&s->conn_list, conn); + conn->processing = true; + } + + if (mode == PRIMARY_IN) { + if (g_queue_get_length(&conn->primary_list) < + MAX_QUEUE_SIZE) { + g_queue_push_tail(&conn->primary_list, pkt); + } else { + error_report("colo compare primary queue size too big," + "drop packet"); + } + } else { + if (g_queue_get_length(&conn->secondary_list) < + MAX_QUEUE_SIZE) { + g_queue_push_tail(&conn->secondary_list, pkt); + } else { + error_report("colo compare secondary queue size too big," + "drop packet"); + } + } return 0; } @@ -265,9 +308,14 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize); net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize); + g_queue_init(&s->conn_list); + s->hashtable_size = 0; - /* use g_hash_table_new_full() to new a hashtable */ + s->connection_track_table = g_hash_table_new_full(connection_key_hash, + connection_key_equal, + g_free, + connection_destroy); return; } @@ -308,6 +356,8 @@ static void colo_compare_finalize(Object *obj) qemu_chr_fe_release(s->chr_out); } + g_queue_free(&s->conn_list); + g_free(s->pri_indev); g_free(s->sec_indev); g_free(s->outdev);