diff mbox

[1/1] vnc: Add support for color map

Message ID 1464099559-20789-1-git-send-email-den@openvz.org
State New, archived
Headers show

Commit Message

Denis V. Lunev May 24, 2016, 2:19 p.m. UTC
From: Alexander Graf <agraf@suse.de>

Our current VNC code does not handle color maps (aka non-true-color) at all
and aborts if a client requests them. There are 2 major issues with this:

 1) A VNC viewer on an 8-bit X11 system may request color maps
 2) RealVNC _always_ starts requesting color maps, then moves on to full color

In order to support these 2 use cases, let's just create a fake color map
that covers exactly our normal true color 8 bit color space. That way we don't
lose anything over a client that wants true color.

Reported-by: Sascha Wehnert <swehnert@suse.com>
Signed-off-by: Alexander Graf <agraf@suse.de>

Actually this is a very old patch originally submitted in 2013 by
Alexander. The situation is still the same with RealVNC, it does not
connect by default to QEMU VNC. The problem is that this client is
really popular. This is better to be kludged.

Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
CC: Gerd Hoffmann <kraxel@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>
---
 ui/vnc.c | 33 +++++++++++++++++++++++++++++++--
 1 file changed, 31 insertions(+), 2 deletions(-)

Comments

Denis V. Lunev May 31, 2016, 8:14 a.m. UTC | #1
On 05/24/2016 05:19 PM, Denis V. Lunev wrote:
> From: Alexander Graf <agraf@suse.de>
>
> Our current VNC code does not handle color maps (aka non-true-color) at all
> and aborts if a client requests them. There are 2 major issues with this:
>
>   1) A VNC viewer on an 8-bit X11 system may request color maps
>   2) RealVNC _always_ starts requesting color maps, then moves on to full color
>
> In order to support these 2 use cases, let's just create a fake color map
> that covers exactly our normal true color 8 bit color space. That way we don't
> lose anything over a client that wants true color.
>
> Reported-by: Sascha Wehnert <swehnert@suse.com>
> Signed-off-by: Alexander Graf <agraf@suse.de>
>
> Actually this is a very old patch originally submitted in 2013 by
> Alexander. The situation is still the same with RealVNC, it does not
> connect by default to QEMU VNC. The problem is that this client is
> really popular. This is better to be kludged.
>
> Signed-off-by: Pavel Butsykin <pbutsykin@virtuozzo.com>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
> CC: Gerd Hoffmann <kraxel@redhat.com>
> CC: Paolo Bonzini <pbonzini@redhat.com>
> ---
>   ui/vnc.c | 33 +++++++++++++++++++++++++++++++--
>   1 file changed, 31 insertions(+), 2 deletions(-)
>
> diff --git a/ui/vnc.c b/ui/vnc.c
> index d2ebf1f..ddd01fd 100644
> --- a/ui/vnc.c
> +++ b/ui/vnc.c
> @@ -2094,6 +2094,24 @@ static void set_pixel_conversion(VncState *vs)
>       }
>   }
>   
> +static void send_color_map(VncState *vs)
> +{
> +    int i;
> +
> +    vnc_write_u8(vs, VNC_MSG_SERVER_SET_COLOUR_MAP_ENTRIES);
> +    vnc_write_u8(vs,  0);    /* padding     */
> +    vnc_write_u16(vs, 0);    /* first color */
> +    vnc_write_u16(vs, 256);  /* # of colors */
> +
> +    for (i = 0; i < 256; i++) {
> +        PixelFormat *pf = &vs->client_pf;
> +
> +        vnc_write_u16(vs, (((i >> pf->rshift) & pf->rmax) << (16 - pf->rbits)));
> +        vnc_write_u16(vs, (((i >> pf->gshift) & pf->gmax) << (16 - pf->gbits)));
> +        vnc_write_u16(vs, (((i >> pf->bshift) & pf->bmax) << (16 - pf->bbits)));
> +    }
> +}
> +
>   static void set_pixel_format(VncState *vs,
>                                int bits_per_pixel, int depth,
>                                int big_endian_flag, int true_color_flag,
> @@ -2101,8 +2119,15 @@ static void set_pixel_format(VncState *vs,
>                                int red_shift, int green_shift, int blue_shift)
>   {
>       if (!true_color_flag) {
> -        vnc_client_error(vs);
> -        return;
> +        /* Expose a reasonable default 256 color map */
> +        bits_per_pixel = 8;
> +        depth = 8;
> +        red_max = 7;
> +        green_max = 7;
> +        blue_max = 3;
> +        red_shift = 0;
> +        green_shift = 3;
> +        blue_shift = 6;
>       }
>   
>       switch (bits_per_pixel) {
> @@ -2132,6 +2157,10 @@ static void set_pixel_format(VncState *vs,
>       vs->client_pf.depth = bits_per_pixel == 32 ? 24 : bits_per_pixel;
>       vs->client_be = big_endian_flag;
>   
> +    if (!true_color_flag) {
> +        send_color_map(vs);
> +    }
> +
>       set_pixel_conversion(vs);
>   
>       graphic_hw_invalidate(vs->vd->dcl.con);
ping
Gerd Hoffmann May 31, 2016, 2:06 p.m. UTC | #2
On Di, 2016-05-31 at 11:14 +0300, Denis V. Lunev wrote:
> On 05/24/2016 05:19 PM, Denis V. Lunev wrote:
> > From: Alexander Graf <agraf@suse.de>
> >
> > Our current VNC code does not handle color maps (aka non-true-color) at all
> > and aborts if a client requests them. There are 2 major issues with this:

> ping

In the vnc queue already.  Just forgot to send a notice.

cheers,
  Gerd
Denis V. Lunev May 31, 2016, 2:07 p.m. UTC | #3
On 05/31/2016 05:06 PM, Gerd Hoffmann wrote:
> On Di, 2016-05-31 at 11:14 +0300, Denis V. Lunev wrote:
>> On 05/24/2016 05:19 PM, Denis V. Lunev wrote:
>>> From: Alexander Graf <agraf@suse.de>
>>>
>>> Our current VNC code does not handle color maps (aka non-true-color) at all
>>> and aborts if a client requests them. There are 2 major issues with this:
>> ping
> In the vnc queue already.  Just forgot to send a notice.
>
> cheers,
>    Gerd
>
thanks a lot!
diff mbox

Patch

diff --git a/ui/vnc.c b/ui/vnc.c
index d2ebf1f..ddd01fd 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -2094,6 +2094,24 @@  static void set_pixel_conversion(VncState *vs)
     }
 }
 
+static void send_color_map(VncState *vs)
+{
+    int i;
+
+    vnc_write_u8(vs, VNC_MSG_SERVER_SET_COLOUR_MAP_ENTRIES);
+    vnc_write_u8(vs,  0);    /* padding     */
+    vnc_write_u16(vs, 0);    /* first color */
+    vnc_write_u16(vs, 256);  /* # of colors */
+
+    for (i = 0; i < 256; i++) {
+        PixelFormat *pf = &vs->client_pf;
+
+        vnc_write_u16(vs, (((i >> pf->rshift) & pf->rmax) << (16 - pf->rbits)));
+        vnc_write_u16(vs, (((i >> pf->gshift) & pf->gmax) << (16 - pf->gbits)));
+        vnc_write_u16(vs, (((i >> pf->bshift) & pf->bmax) << (16 - pf->bbits)));
+    }
+}
+
 static void set_pixel_format(VncState *vs,
                              int bits_per_pixel, int depth,
                              int big_endian_flag, int true_color_flag,
@@ -2101,8 +2119,15 @@  static void set_pixel_format(VncState *vs,
                              int red_shift, int green_shift, int blue_shift)
 {
     if (!true_color_flag) {
-        vnc_client_error(vs);
-        return;
+        /* Expose a reasonable default 256 color map */
+        bits_per_pixel = 8;
+        depth = 8;
+        red_max = 7;
+        green_max = 7;
+        blue_max = 3;
+        red_shift = 0;
+        green_shift = 3;
+        blue_shift = 6;
     }
 
     switch (bits_per_pixel) {
@@ -2132,6 +2157,10 @@  static void set_pixel_format(VncState *vs,
     vs->client_pf.depth = bits_per_pixel == 32 ? 24 : bits_per_pixel;
     vs->client_be = big_endian_flag;
 
+    if (!true_color_flag) {
+        send_color_map(vs);
+    }
+
     set_pixel_conversion(vs);
 
     graphic_hw_invalidate(vs->vd->dcl.con);