From patchwork Tue Sep 27 02:22:36 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhang Chen X-Patchwork-Id: 9351453 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B33DE601C2 for ; Tue, 27 Sep 2016 02:37:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A143B28581 for ; Tue, 27 Sep 2016 02:37:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 94F5828F97; Tue, 27 Sep 2016 02:37:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A239228581 for ; Tue, 27 Sep 2016 02:37:32 +0000 (UTC) Received: from localhost ([::1]:47591 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1boiGx-0004Gb-SE for patchwork-qemu-devel@patchwork.kernel.org; Mon, 26 Sep 2016 22:37:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45117) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1boi3N-0007po-S6 for qemu-devel@nongnu.org; Mon, 26 Sep 2016 22:23:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1boi3J-00036t-UV for qemu-devel@nongnu.org; Mon, 26 Sep 2016 22:23:28 -0400 Received: from [59.151.112.132] (port=3609 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1boi3I-000317-Uw for qemu-devel@nongnu.org; Mon, 26 Sep 2016 22:23:25 -0400 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="11402788" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 27 Sep 2016 10:23:23 +0800 Received: from G08CNEXCHPEKD02.g08.fujitsu.local (unknown [10.167.33.83]) by cn.fujitsu.com (Postfix) with ESMTP id 1C5544623492; Tue, 27 Sep 2016 10:23:21 +0800 (CST) Received: from localhost.localdomain (10.167.226.56) by G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.279.2; Tue, 27 Sep 2016 10:23:20 +0800 From: Zhang Chen To: qemu devel , Jason Wang Date: Tue, 27 Sep 2016 10:22:36 +0800 Message-ID: <1474942956-27182-13-git-send-email-zhangchen.fnst@cn.fujitsu.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1474942956-27182-1-git-send-email-zhangchen.fnst@cn.fujitsu.com> References: <1474942956-27182-1-git-send-email-zhangchen.fnst@cn.fujitsu.com> MIME-Version: 1.0 X-Originating-IP: [10.167.226.56] X-yoursite-MailScanner-ID: 1C5544623492.AA8DE X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: zhangchen.fnst@cn.fujitsu.com X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Subject: [Qemu-devel] [PATCH V15 12/12] docs: Add documentation for COLO-proxy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Li Zhijian , "eddie . dong" , "Dr . David Alan Gilbert" , Zhang Chen , zhanghailiang Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Introduce the design of COLO-proxy, and how to use it. Signed-off-by: Zhang Chen --- docs/colo-proxy.txt | 188 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 docs/colo-proxy.txt diff --git a/docs/colo-proxy.txt b/docs/colo-proxy.txt new file mode 100644 index 0000000..76767cb --- /dev/null +++ b/docs/colo-proxy.txt @@ -0,0 +1,188 @@ +COLO-proxy +---------- +Copyright (c) 2016 Intel Corporation +Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. +Copyright (c) 2016 Fujitsu, Corp. + +This work is licensed under the terms of the GNU GPL, version 2 or later. +See the COPYING file in the top-level directory. + +This document gives an overview of COLO proxy's design. + +== Background == +COLO-proxy is a part of COLO project. It is used +to compare the network package to help COLO decide +whether to do checkpoint. With COLO-proxy's help, +COLO greatly improves the performance. + +The filter-redirector, filter-mirror, colo-compare +and filter-rewriter compose the COLO-proxy. + +== Architecture == + +COLO-Proxy is based on qemu netfilter and it's a plugin for qemu netfilter +(except colo-compare). It keep Secondary VM connect normally to +client and compare packets sent by PVM with sent by SVM. +If the packet difference, notify COLO-frame to do checkpoint and send +all primary packet has queued. Otherwise just send the queued primary +packet and drop the queued secondary packet. + +Below is a COLO proxy ascii figure: + + Primary qemu Secondary qemu ++--------------------------------------------------------------+ +----------------------------------------------------------------+ +| +----------------------------------------------------------+ | | +-----------------------------------------------------------+ | +| | | | | | | | +| | guest | | | | guest | | +| | | | | | | | +| +-------^--------------------------+-----------------------+ | | +---------------------+--------+----------------------------+ | +| | | | | ^ | | +| | | | | | | | +| | +------------------------------------------------------+ | | | | +|netfilter| | | | | | netfilter | | | +| +----------+ +----------------------------+ | | | +-----------------------------------------------------------+ | +| | | | | | out | | | | | | filter excute order | | +| | | | +-----------------------------+ | | | | | | +-------------------> | | +| | | | | | | | | | | | | | TCP | | +| | +-----+--+-+ +-----v----+ +-----v----+ |pri +----+----+sec| | | | +------------+ +---+----+---v+rewriter++ +------------+ | | +| | | | | | | | |in | |in | | | | | | | | | | | | | +| | | filter | | filter | | filter +------> colo <------+ +--------> filter +--> adjust | adjust +--> filter | | | +| | | mirror | |redirector| |redirector| | | compare | | | | | | redirector | | ack | seq | | redirector | | | +| | | | | | | | | | | | | | | | | | | | | | | | +| | +----^-----+ +----+-----+ +----------+ | +---------+ | | | | +------------+ +--------+--------------+ +---+--------+ | | +| | | tx | rx rx | | | | | tx all | rx | | +| | | | | | | | +-----------------------------------------------------------+ | +| | | +--------------+ | | | | | | +| | | filter excute order | | | | | | | +| | | +----------------> | | | +--------------------------------------------------------+ | +| +-----------------------------------------+ | | | +| | | | | | ++--------------------------------------------------------------+ +----------------------------------------------------------------+ + |guest receive | guest send + | | ++--------+----------------------------v------------------------+ +| | NOTE: filter direction is rx/tx/all +| tap | rx:receive packets sent to the netdev +| | tx:receive packets sent by the netdev ++--------------------------------------------------------------+ + +1.Guest receive packet route: + +Primary: + +Tap --> Mirror Client Filter +Mirror client will send packet to guest,at the +same time, copy and forward packet to secondary +mirror server. + +Secondary: + +Mirror Server Filter --> TCP Rewriter +If receive packet is TCP packet,we will adjust ack +and update TCP checksum, then send to secondary +guest. Otherwise directly send to guest. + +2.Guest send packet route: + +Primary: + +Guest --> Redirect Server Filter +Redirect server filter receive primary guest packet +but do nothing, just pass to next filter. + +Redirect Server Filter --> COLO-Compare +COLO-compare receive primary guest packet then +waiting scondary redirect packet to compare it. +If packet same,send queued primary packet and clear +queued secondary packet, Otherwise send primary packet +and do checkpoint. + +COLO-Compare --> Another Redirector Filter +The redirector get packet from colo-compare by use +chardev socket. + +Redirector Filter --> Tap +Send the packet. + +Secondary: + +Guest --> TCP Rewriter Filter +If the packet is TCP packet,we will adjust seq +and update TCP checksum. Then send it to +redirect client filter. Otherwise directly send to +redirect client filter. + +Redirect Client Filter --> Redirect Server Filter +Forward packet to primary. + +== Components introduction == + +Filter-mirror is a netfilter plugin. +It gives qemu the ability to mirror +packets to a chardev. + +Filter-redirector is a netfilter plugin. +It gives qemu the ability to redirect net packet. +Redirector can redirect filter's net packet to outdev, +and redirect indev's packet to filter. + + filter + + + redirector | + +--------------+ + | | | + | | | + | | | + indev +---------+ +----------> outdev + | | | + | | | + | | | + +--------------+ + | + v + filter + +COLO-compare, we do packet comparing job. +Packets coming from the primary char indev will be sent to outdev. +Packets coming from the secondary char dev will be dropped after comparing. +COLO-comapre need two input chardev and one output chardev: +primary_in=chardev1-id (source: primary send packet) +secondary_in=chardev2-id (source: secondary send packet) +outdev=chardev3-id + +Filter-rewriter will rewrite some of secondary packet to make +secondary guest's tcp connection established successfully. +In this module we will rewrite tcp packet's ack to the secondary +from primary,and rewrite tcp packet's seq to the primary from +secondary. + +== Usage == + +Here, we use demo ip and port discribe more clearly. +Primary(ip:3.3.3.3): +-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown +-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 +-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait +-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait +-chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait +-chardev socket,id=compare0-0,host=3.3.3.3,port=9001 +-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait +-chardev socket,id=compare_out0,host=3.3.3.3,port=9005 +-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 +-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out +-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 +-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0 + +Secondary(ip:3.3.3.8): +-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown +-device e1000,netdev=hn0,mac=52:a4:00:12:78:66 +-chardev socket,id=red0,host=3.3.3.3,port=9003 +-chardev socket,id=red1,host=3.3.3.3,port=9004 +-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 +-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 + +Note: + a.COLO-proxy must work with COLO-frame and Block-replication. + b.Primary COLO must be started firstly, because COLO-proxy needs + chardev socket server running before secondary started. + c.Filter-rewriter only rewrite tcp packet.