From patchwork Fri Mar 25 09:02:08 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 8669121 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id DE0B1C0553 for ; Fri, 25 Mar 2016 09:02:51 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id AB420201F5 for ; Fri, 25 Mar 2016 09:02:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 617C720166 for ; Fri, 25 Mar 2016 09:02:49 +0000 (UTC) Received: from localhost ([::1]:54789 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ajNdo-0006Gm-P1 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 25 Mar 2016 05:02:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56048) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ajNdc-0006CM-DY for qemu-devel@nongnu.org; Fri, 25 Mar 2016 05:02:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ajNda-0007wH-Ic for qemu-devel@nongnu.org; Fri, 25 Mar 2016 05:02:36 -0400 Received: from [59.151.112.132] (port=21889 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ajNdZ-0007rA-JA for qemu-devel@nongnu.org; Fri, 25 Mar 2016 05:02:34 -0400 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="4963596" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 25 Mar 2016 17:02:10 +0800 Received: from G08CNEXCHPEKD02.g08.fujitsu.local (unknown [10.167.33.83]) by cn.fujitsu.com (Postfix) with ESMTP id 155E4408D26B; Fri, 25 Mar 2016 17:02:07 +0800 (CST) Received: from G08FNSTD140215.g08.fujitsu.local (10.167.226.56) by G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.279.2; Fri, 25 Mar 2016 17:02:06 +0800 From: Zhang Chen To: qemu devel , Jason Wang Date: Fri, 25 Mar 2016 17:02:08 +0800 Message-ID: <1458896529-7891-3-git-send-email-zhangchen.fnst@cn.fujitsu.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1458896529-7891-1-git-send-email-zhangchen.fnst@cn.fujitsu.com> References: <1458896529-7891-1-git-send-email-zhangchen.fnst@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.56] X-yoursite-MailScanner-ID: 155E4408D26B.A04E1 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: zhangchen.fnst@cn.fujitsu.com X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Cc: Li Zhijian , Gui jianfeng , "eddie.dong" , zhanghailiang , "Dr. David Alan Gilbert" , Zhang Chen , Yang Hongyang Subject: [Qemu-devel] [PATCH V1 2/3] colo-compare: track connection and enqueue packet X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In this patch we use kernel jhash table to track connection, and then enqueue net packet like this: + CompareState ++ | | +---------------+ +---------------+ +---------------+ |conn list +--->conn +--------->conn | +---------------+ +---------------+ +---------------+ | | | | | | +---------------+ +---v----+ +---v----+ +---v----+ +---v----+ |primary | |secondary |primary | |secondary |packet | |packet + |packet | |packet + +--------+ +--------+ +--------+ +--------+ | | | | +---v----+ +---v----+ +---v----+ +---v----+ |primary | |secondary |primary | |secondary |packet | |packet + |packet | |packet + +--------+ +--------+ +--------+ +--------+ | | | | +---v----+ +---v----+ +---v----+ +---v----+ |primary | |secondary |primary | |secondary |packet | |packet + |packet | |packet + +--------+ +--------+ +--------+ +--------+ Signed-off-by: Zhang Chen Signed-off-by: Li Zhijian Signed-off-by: Wen Congyang --- net/colo-compare.c | 324 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 321 insertions(+), 3 deletions(-) diff --git a/net/colo-compare.c b/net/colo-compare.c index 62c66df..0bb5a51 100644 --- a/net/colo-compare.c +++ b/net/colo-compare.c @@ -20,15 +20,22 @@ #include "net/queue.h" #include "sysemu/char.h" #include "qemu/sockets.h" +#include +#include "slirp/slirp.h" +#include "qemu/jhash.h" +#include #define TYPE_COLO_COMPARE "colo-compare" #define COLO_COMPARE(obj) \ OBJECT_CHECK(CompareState, (obj), TYPE_COLO_COMPARE) #define COMPARE_READ_LEN_MAX NET_BUFSIZE +#define PAGE_SIZE 4096 +#define ETH_HLEN 14 static QTAILQ_HEAD(, CompareState) net_compares = QTAILQ_HEAD_INITIALIZER(net_compares); +static ssize_t hashtable_max_size; typedef struct ReadState { int state; /* 0 = getting length, 1 = getting data */ @@ -37,6 +44,28 @@ typedef struct ReadState { uint8_t buf[COMPARE_READ_LEN_MAX]; } ReadState; +/* + + CompareState ++ + | | + +---------------+ +---------------+ +---------------+ + |conn list +--->conn +--------->conn | + +---------------+ +---------------+ +---------------+ + | | | | | | + +---------------+ +---v----+ +---v----+ +---v----+ +---v----+ + |primary | |secondary |primary | |secondary + |packet | |packet + |packet | |packet + + +--------+ +--------+ +--------+ +--------+ + | | | | + +---v----+ +---v----+ +---v----+ +---v----+ + |primary | |secondary |primary | |secondary + |packet | |packet + |packet | |packet + + +--------+ +--------+ +--------+ +--------+ + | | | | + +---v----+ +---v----+ +---v----+ +---v----+ + |primary | |secondary |primary | |secondary + |packet | |packet + |packet | |packet + + +--------+ +--------+ +--------+ +--------+ +*/ typedef struct CompareState { Object parent; @@ -49,8 +78,268 @@ typedef struct CompareState { QTAILQ_ENTRY(CompareState) next; ReadState pri_rs; ReadState sec_rs; + + /* connection list: the connections belonged to this NIC could be found + * in this list. + * element type: Connection + */ + GQueue conn_list; + QemuMutex conn_list_lock; /* to protect conn_list */ + /* hashtable to save connection */ + GHashTable *connection_track_table; + /* to save unprocessed_connections */ + GQueue unprocessed_connections; + /* proxy current hash size */ + ssize_t hashtable_size; } CompareState; +typedef struct Packet { + void *data; + union { + uint8_t *network_layer; + struct ip *ip; + }; + uint8_t *transport_layer; + int size; + CompareState *s; +} Packet; + +typedef struct ConnectionKey { + /* (src, dst) must be grouped, in the same way than in IP header */ + struct in_addr src; + struct in_addr dst; + uint16_t src_port; + uint16_t dst_port; + uint8_t ip_proto; +} QEMU_PACKED ConnectionKey; + +typedef struct Connection { + QemuMutex list_lock; + /* connection primary send queue: element type: Packet */ + GQueue primary_list; + /* connection secondary send queue: element type: Packet */ + GQueue secondary_list; + /* flag to enqueue unprocessed_connections */ + bool processing; + int ip_proto; +} Connection; + +enum { + PRIMARY_IN = 0, + SECONDARY_IN, +}; + +static void packet_destroy(void *opaque, void *user_data); +static int compare_chr_send(CharDriverState *out, const uint8_t *buf, int size); + +static uint32_t connection_key_hash(const void *opaque) +{ + const ConnectionKey *key = opaque; + uint32_t a, b, c; + + /* Jenkins hash */ + a = b = c = JHASH_INITVAL + sizeof(*key); + a += key->src.s_addr; + b += key->dst.s_addr; + c += (key->src_port | key->dst_port << 16); + __jhash_mix(a, b, c); + + a += key->ip_proto; + __jhash_final(a, b, c); + + return c; +} + +static int connection_key_equal(const void *opaque1, const void *opaque2) +{ + return memcmp(opaque1, opaque2, sizeof(ConnectionKey)) == 0; +} + +/* + * initialize connecon_key for packet + * Return 0 on success, if return 1 the pkt will be sent later + */ +static int connection_key_init(Packet *pkt, ConnectionKey *key) +{ + int network_length; + uint8_t *data = pkt->data; + uint16_t l3_proto; + uint32_t tmp_ports; + ssize_t l2hdr_len = eth_get_l2_hdr_length(data); + + pkt->network_layer = data + ETH_HLEN; + l3_proto = eth_get_l3_proto(data, l2hdr_len); + if (l3_proto != ETH_P_IP) { + return 1; + } + + network_length = pkt->ip->ip_hl * 4; + pkt->transport_layer = pkt->network_layer + network_length; + key->ip_proto = pkt->ip->ip_p; + key->src = pkt->ip->ip_src; + key->dst = pkt->ip->ip_dst; + + switch (key->ip_proto) { + case IPPROTO_TCP: + case IPPROTO_UDP: + case IPPROTO_DCCP: + case IPPROTO_ESP: + case IPPROTO_SCTP: + case IPPROTO_UDPLITE: + tmp_ports = *(uint32_t *)(pkt->transport_layer); + key->src_port = tmp_ports & 0xffff; + key->dst_port = tmp_ports >> 16; + break; + case IPPROTO_AH: + tmp_ports = *(uint32_t *)(pkt->transport_layer + 4); + key->src_port = tmp_ports & 0xffff; + key->dst_port = tmp_ports >> 16; + break; + default: + break; + } + + return 0; +} + +static Connection *connection_new(ConnectionKey *key) +{ + Connection *conn = g_slice_new(Connection); + + qemu_mutex_init(&conn->list_lock); + conn->ip_proto = key->ip_proto; + conn->processing = false; + g_queue_init(&conn->primary_list); + g_queue_init(&conn->secondary_list); + + return conn; +} + +/* + * Clear hashtable, stop this hash growing really huge + */ +static void connection_hashtable_reset(CompareState *s) +{ + s->hashtable_size = 0; + g_hash_table_remove_all(s->connection_track_table); +} + +/* if not found, creata a new connection and add to hash table */ +static Connection *connection_get(CompareState *s, ConnectionKey *key) +{ + /* FIXME: protect connection_track_table */ + Connection *conn = g_hash_table_lookup(s->connection_track_table, key); + + if (conn == NULL) { + ConnectionKey *new_key = g_memdup(key, sizeof(*key)); + + conn = connection_new(key); + + s->hashtable_size++; + if (s->hashtable_size > hashtable_max_size) { + error_report("colo proxy connection hashtable full, clear it"); + connection_hashtable_reset(s); + /* TODO:clear conn_list */ + } else { + g_hash_table_insert(s->connection_track_table, new_key, conn); + } + } + + return conn; +} + +static void connection_destroy(void *opaque) +{ + Connection *conn = opaque; + + qemu_mutex_lock(&conn->list_lock); + g_queue_foreach(&conn->primary_list, packet_destroy, NULL); + g_queue_free(&conn->primary_list); + g_queue_foreach(&conn->secondary_list, packet_destroy, NULL); + g_queue_free(&conn->secondary_list); + qemu_mutex_unlock(&conn->list_lock); + qemu_mutex_destroy(&conn->list_lock); + g_slice_free(Connection, conn); +} + +static Packet *packet_new(CompareState *s, const void *data, + int size, ConnectionKey *key) +{ + Packet *pkt = g_slice_new(Packet); + + pkt->data = g_memdup(data, size); + pkt->size = size; + pkt->s = s; + + if (connection_key_init(pkt, key)) { + packet_destroy(pkt, NULL); + pkt = NULL; + } + + return pkt; +} + +static int packet_enqueue(CompareState *s, int mode) +{ + ConnectionKey key = {{ 0 } }; + Packet *pkt = NULL; + Connection *conn; + + /* arp packet will be sent */ + if (mode == PRIMARY_IN) { + pkt = packet_new(s, s->pri_rs.buf, s->pri_rs.packet_len, &key); + } else { + pkt = packet_new(s, s->sec_rs.buf, s->sec_rs.packet_len, &key); + } + if (!pkt) { + return -1; + } + + conn = connection_get(s, &key); + if (!conn->processing) { + qemu_mutex_lock(&s->conn_list_lock); + g_queue_push_tail(&s->conn_list, conn); + qemu_mutex_unlock(&s->conn_list_lock); + conn->processing = true; + } + + qemu_mutex_lock(&conn->list_lock); + if (mode == PRIMARY_IN) { + g_queue_push_tail(&conn->primary_list, pkt); + } else { + g_queue_push_tail(&conn->secondary_list, pkt); + } + qemu_mutex_unlock(&conn->list_lock); + + return 0; +} + +static void packet_destroy(void *opaque, void *user_data) +{ + Packet *pkt = opaque; + + g_free(pkt->data); + g_slice_free(Packet, pkt); +} + +static inline void colo_flush_connection(void *opaque, void *user_data) +{ + Connection *conn = opaque; + Packet *pkt = NULL; + + qemu_mutex_lock(&conn->list_lock); + while (!g_queue_is_empty(&conn->primary_list)) { + pkt = g_queue_pop_head(&conn->primary_list); + compare_chr_send(pkt->s->chr_out, pkt->data, pkt->size); + /* FIXME: destroy pkt ?*/ + } + while (!g_queue_is_empty(&conn->secondary_list)) { + pkt = g_queue_pop_head(&conn->secondary_list); + packet_destroy(pkt, NULL); + } + qemu_mutex_unlock(&conn->list_lock); +} + static int compare_chr_send(CharDriverState *out, const uint8_t *buf, int size) { int ret = 0; @@ -142,8 +431,10 @@ static void compare_pri_chr_in(void *opaque, const uint8_t *buf, int size) ret = compare_chr_fill_rstate(&s->pri_rs, buf, size); if (ret == 1) { - /* FIXME: enqueue to primary packet list */ - compare_chr_send(s->chr_out, buf, size); + if (packet_enqueue(s, PRIMARY_IN)) { + error_report("primary: unsupported packet in"); + compare_chr_send(s->chr_out, buf, size); + } } else if (ret == -1) { qemu_chr_add_handlers(s->chr_pri_in, NULL, NULL, NULL, NULL); } @@ -156,7 +447,9 @@ static void compare_sec_chr_in(void *opaque, const uint8_t *buf, int size) ret = compare_chr_fill_rstate(&s->sec_rs, buf, size); if (ret == 1) { - /* TODO: enqueue to secondary packet list*/ + if (packet_enqueue(s, SECONDARY_IN)) { + error_report("secondary: unsupported packet in"); + } } else if (ret == -1) { qemu_chr_add_handlers(s->chr_sec_in, NULL, NULL, NULL, NULL); } @@ -210,6 +503,7 @@ static void compare_set_outdev(Object *obj, const char *value, Error **errp) static void colo_compare_complete(UserCreatable *uc, Error **errp) { CompareState *s = COLO_COMPARE(uc); + struct sysinfo si; if (!s->pri_indev || !s->sec_indev || !s->outdev) { error_setg(errp, "colo compare needs 'primary_in' ," @@ -255,6 +549,29 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp) QTAILQ_INSERT_TAIL(&net_compares, s, next); + g_queue_init(&s->conn_list); + qemu_mutex_init(&s->conn_list_lock); + + s->hashtable_size = 0; + /* + * Idea from kernel tcp.c: use 1/16384 of memory. On i386: 32MB + * machine has 512 buckets. >= 1GB machines have 16384 buckets. + */ + sysinfo(&si); + hashtable_max_size = si.totalram / 16384; + if (si.totalram > (1024 * 1024 * 1024 / PAGE_SIZE)) { + hashtable_max_size = 16384; + } + if (hashtable_max_size < 32) { + hashtable_max_size = 32; + } + hashtable_max_size = hashtable_max_size * 8; /* default factor = 8 */ + + s->connection_track_table = g_hash_table_new_full(connection_key_hash, + connection_key_equal, + g_free, + connection_destroy); + return; out: @@ -297,6 +614,7 @@ static void colo_compare_class_finalize(ObjectClass *oc, void *data) if (!QTAILQ_EMPTY(&net_compares)) { QTAILQ_REMOVE(&net_compares, s, next); } + qemu_mutex_destroy(&s->conn_list_lock); } static void colo_compare_init(Object *obj)