[vlc-devel] [PATCH] libvlc: add API to propagate mouse move and down/up events

[Felix Paul Kühne]

This is needed when using OSD menus with a vout which can't detect mouse movements because it doesn't have access to the window server it lives in, e.g. Android or vmem.
---
include/vlc/libvlc_media_player.h | 36 ++++++++++++++++++++++++++++++++
lib/libvlc.sym | 3 +++
lib/video.c | 43 +++++++++++++++++++++++++++++++++++++++
3 files changed, 82 insertions(+)

diff --git a/include/vlc/libvlc_media_player.h b/include/vlc/libvlc_media_player.h
index fb91f59..43d66a5 100644
--- a/include/vlc/libvlc_media_player.h
+++ b/include/vlc/libvlc_media_player.h
@@ -951,6 +951,42 @@ void libvlc_video_set_key_input( libvlc_media_player_t *p_mi, unsigned on );
LIBVLC_API
void libvlc_video_set_mouse_input( libvlc_media_player_t *p_mi, unsigned on );

+
+/* Propagate a mouse position in coordinates relative to the video
+ *
+ * \param p_mi media player
+ * \param num number of the video (starting from, and most commonly 0)
+ * \param px to set the abscissa
+ * \param py to set the ordinate
+ * \return 0 on success, -1 if the specified video does not exist
+ * \warning it is your duty to convert screen coordinates to video coordinates
+ */
+ LIBVLC_API
+int libvlc_video_set_cursor( libvlc_media_player_t *p_mi, unsigned num,
+ int px, int py );
+
+/* propagate a mouse down event at the last known mouse position
+ *
+ * \param p_mi media player
+ * \param num number of the video (starting from, and most commonly 0)
+ * \param buttonNumber to differenciate the clicked button
+ * 0 = left, 1 = right, >=2 = other
+ */
+LIBVLC_API
+int libvlc_video_set_mouse_down( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber );
+
+/* propagate a mouse up event at the last known mouse position
+ *
+ * \param p_mi media player
+ * \param num number of the video (starting from, and most commonly 0)
+ * \param buttonNumber to differenciate the clicked button
+ * 0 = left, 1 = right, >=2 = other
+ */
+LIBVLC_API
+int libvlc_video_set_mouse_up( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber );
+
/**
* Get the pixel dimensions of a video.
*
diff --git a/lib/libvlc.sym b/lib/libvlc.sym
index fdfd164..a4ae3b9 100644
--- a/lib/libvlc.sym
+++ b/lib/libvlc.sym
@@ -225,6 +225,7 @@ libvlc_video_set_adjust_float
libvlc_video_set_adjust_int
libvlc_video_set_aspect_ratio
libvlc_video_set_callbacks
+libvlc_video_set_cursor
libvlc_video_set_crop_geometry
libvlc_video_set_deinterlace
libvlc_video_set_format
@@ -234,7 +235,9 @@ libvlc_video_set_logo_int
libvlc_video_set_logo_string
libvlc_video_set_marquee_int
libvlc_video_set_marquee_string
+libvlc_video_set_mouse_down
libvlc_video_set_mouse_input
+libvlc_video_set_mouse_up
libvlc_video_set_scale
libvlc_video_set_spu
libvlc_video_set_spu_delay
diff --git a/lib/video.c b/lib/video.c
index 43471b9..161ffda 100644
--- a/lib/video.c
+++ b/lib/video.c
@@ -217,6 +217,49 @@ int libvlc_video_get_cursor( libvlc_media_player_t *mp, unsigned num,
return 0;
}

+int libvlc_video_set_cursor( libvlc_media_player_t *mp, unsigned num,
+ int px, int py )
+{
+ vout_thread_t *p_vout = GetVout (mp, num);
+ if (p_vout == NULL)
+ return -1;
+
+ var_SetCoords (p_vout, "mouse-moved", px, py);
+ vlc_object_release (p_vout);
+ return 0;
+}
+
+
+int libvlc_video_set_mouse_down( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber )
+{
+ vout_thread_t *p_vout = GetVout (mp , num);
+ if (p_vout == NULL)
+ return -1;
+
+ var_OrInteger(p_vout, "mouse-button-down", 1 << buttonNumber);
+
+ int x, y;
+ var_GetCoords(p_vout, "mouse-moved", &x, &y);
+ var_SetCoords(p_vout, "mouse-clicked", x, y);
+
+ return 0;
+}
+
+
+int libvlc_video_set_mouse_up( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber )
+{
+ vout_thread_t *p_vout = GetVout (mp , num);
+ if (p_vout == NULL)
+ return -1;
+
+ var_NAndInteger(p_vout, "mouse-button-down", 1 << buttonNumber);
+
+ return 0;
+}
+
+
unsigned libvlc_media_player_has_vout( libvlc_media_player_t *p_mi )
{
size_t n;

[Felix Paul Kühne]

This is needed when using OSD menus with a vout which can't detect mouse movements because it doesn't have access to the window server it lives in, e.g. Android or vmem.
---
include/vlc/libvlc_media_player.h | 36 +++++++++++++++++++++++++++++++
lib/libvlc.sym | 3 +++
lib/video.c | 45 +++++++++++++++++++++++++++++++++++++++
3 files changed, 84 insertions(+)

diff --git a/include/vlc/libvlc_media_player.h b/include/vlc/libvlc_media_player.h
index fb91f59..43d66a5 100644
--- a/include/vlc/libvlc_media_player.h
+++ b/include/vlc/libvlc_media_player.h
@@ -951,6 +951,42 @@ void libvlc_video_set_key_input( libvlc_media_player_t *p_mi, unsigned on );
LIBVLC_API
void libvlc_video_set_mouse_input( libvlc_media_player_t *p_mi, unsigned on );

+
+/* Propagate a mouse position in coordinates relative to the video
+ *
+ * \param p_mi media player
+ * \param num number of the video (starting from, and most commonly 0)
+ * \param px to set the abscissa
+ * \param py to set the ordinate
+ * \return 0 on success, -1 if the specified video does not exist
+ * \warning it is your duty to convert screen coordinates to video coordinates
+ */
+ LIBVLC_API
+int libvlc_video_set_cursor( libvlc_media_player_t *p_mi, unsigned num,
+ int px, int py );
+
+/* propagate a mouse down event at the last known mouse position
+ *
+ * \param p_mi media player
+ * \param num number of the video (starting from, and most commonly 0)
+ * \param buttonNumber to differenciate the clicked button
+ * 0 = left, 1 = right, >=2 = other
+ */
+LIBVLC_API
+int libvlc_video_set_mouse_down( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber );
+
+/* propagate a mouse up event at the last known mouse position
+ *
+ * \param p_mi media player
+ * \param num number of the video (starting from, and most commonly 0)
+ * \param buttonNumber to differenciate the clicked button
+ * 0 = left, 1 = right, >=2 = other
+ */
+LIBVLC_API
+int libvlc_video_set_mouse_up( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber );
+
/**
* Get the pixel dimensions of a video.
*
diff --git a/lib/libvlc.sym b/lib/libvlc.sym
index fdfd164..a4ae3b9 100644
--- a/lib/libvlc.sym
+++ b/lib/libvlc.sym
@@ -225,6 +225,7 @@ libvlc_video_set_adjust_float
libvlc_video_set_adjust_int
libvlc_video_set_aspect_ratio
libvlc_video_set_callbacks
+libvlc_video_set_cursor
libvlc_video_set_crop_geometry
libvlc_video_set_deinterlace
libvlc_video_set_format
@@ -234,7 +235,9 @@ libvlc_video_set_logo_int
libvlc_video_set_logo_string
libvlc_video_set_marquee_int
libvlc_video_set_marquee_string
+libvlc_video_set_mouse_down
libvlc_video_set_mouse_input
+libvlc_video_set_mouse_up
libvlc_video_set_scale
libvlc_video_set_spu
libvlc_video_set_spu_delay
diff --git a/lib/video.c b/lib/video.c
index 43471b9..bc4e156 100644
--- a/lib/video.c
+++ b/lib/video.c
@@ -217,6 +217,51 @@ int libvlc_video_get_cursor( libvlc_media_player_t *mp, unsigned num,
return 0;
}

+int libvlc_video_set_cursor( libvlc_media_player_t *mp, unsigned num,
+ int px, int py )
+{
+ vout_thread_t *p_vout = GetVout (mp, num);
+ if (p_vout == NULL)
+ return -1;
+
+ var_SetCoords (p_vout, "mouse-moved", px, py);
+ vlc_object_release (p_vout);
+ return 0;
+}
+
+
+int libvlc_video_set_mouse_down( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber )
+{
+ vout_thread_t *p_vout = GetVout (mp , num);
+ if (p_vout == NULL)
+ return -1;
+
+ var_OrInteger (p_vout, "mouse-button-down", 1 << buttonNumber);
+
+ int x, y;
+ var_GetCoords (p_vout, "mouse-moved", &x, &y);
+ var_SetCoords (p_vout, "mouse-clicked", x, y);
+ vlc_object_release (p_vout);
+
+ return 0;
+}
+
+
+int libvlc_video_set_mouse_up( libvlc_media_player_t *mp, unsigned num,
+ unsigned buttonNumber )
+{
+ vout_thread_t *p_vout = GetVout (mp , num);
+ if (p_vout == NULL)
+ return -1;
+
+ var_NAndInteger (p_vout, "mouse-button-down", 1 << buttonNumber);
+ vlc_object_release (p_vout);
+
+ return 0;
+}
+
+
unsigned libvlc_media_player_has_vout( libvlc_media_player_t *p_mi )
{
size_t n;

[Rémi Denis-Courmont]

Post by Felix Paul Kühne
This is needed when using OSD menus with a vout which can't detect mouse
movements because it doesn't have access to the window server it lives in
vmem.

You don't know which vmem instance you are referring to. This is totally
broken if you have more than one window.

[Rémi Denis-Courmont]

Post by Rémi Denis-Courmont

Post by Felix Paul Kühne
This is needed when using OSD menus with a vout which can't detect mouse
movements because it doesn't have access to the window server it lives in
vmem.

You don't know which vmem instance you are referring to. This is totally
broken if you have more than one window.

(And I think this is solved by implementing a proper vout plugin, not by
hacking vmem further.)

[Felix Paul Kühne]

Hello R?mi,

Thanks for the review.

Post by Rémi Denis-Courmont

Post by Rémi Denis-Courmont

Post by Felix Paul Kühne
This is needed when using OSD menus with a vout which can't detect mouse
movements because it doesn't have access to the window server it lives in
vmem.

You don't know which vmem instance you are referring to. This is totally
broken if you have more than one window.

This is absolutely true.

However: libvlc_video_get_cursor() already uses the same pattern I do in this patch. It lets you access the video output by N.

Additionally, if you consider the remainder of the video API listed in libvlc_medoa_player.h, it appears to me as if more than one video output per libvlc instance was never a supported scenario. This starts with the ability to select a single video track only, but doesn?t end there.

Post by Rémi Denis-Courmont
(And I think this is solved by implementing a proper vout plugin, not by
hacking vmem further.)

I fully agree, however, this is not really feasible in the scenario targeted by the patch.

For the NPAPI plugin, we support a special rendering mode called ?windowless?, which does not make use of the existing libvlc vout modules. It is blitting the vmem buffer directly into the HTML context of a website. This allows fancy stuff like the application of HTML5 effects on the vout and overlays of HTML contents like buttons or text.
In this scenario, vmem is not aware of a mouse cursor, which is the correct way to do things. However, the web plugin is notified about mouse movements and clicks within the drawing rectangle. Those events need to be forwarded to libvlc in a way, so the user can interact with an OSD like a DVD menu, etc. Of course, it is the web plugin?s duty to convert relative screen coordinates of the mouse to screen coordinates within the video by taking account of current scaling, possibly adapted aspect ratio, etc. This is part of a second patch (implemented for the mac only so far), which I push as soon as this patch is merged.

Please let me know if you stumble across ideas how to improve this patch.

Thanks,

Felix

[Rémi Denis-Courmont]

Post by Felix Paul Kühne

Post by Rémi Denis-Courmont
You don't know which vmem instance you are referring to. This is totally
broken if you have more than one window.

This is absolutely true.
However: libvlc_video_get_cursor() already uses the same pattern I do in
this patch. It lets you access the video output by N.

That is a known design bug that should not be repeated. I don't remember why
get_cursor() was even needed way back; it seems useless/unusable in practice
or almost. But at least it exhibits dubious correctness in returning the
correct coordinates for _the_last_ focused video area.

Post by Felix Paul Kühne
Additionally, if you consider the remainder of the video API listed in
libvlc_medoa_player.h, it appears to me as if more than one video output
per libvlc instance was never a supported scenario.

You've got to be kidding. libvlc_video_get_size() supports it (for meta data)
The vmem API supports it too; there was even a discussion a few months back on
how to add new per-vmem instance functions.

Also several video functions apply correctly to all video outputs, regardless
of how many there are.

(...)

Post by Felix Paul Kühne
I fully agree, however, this is not really feasible in the scenario targeted by the patch.

vmem is neither designed nor capable of supporting video rendering (it was
designed by Sam to feed video post-processing filters). Mouse handling is not
the (worst) limitation inhibiting proper rendering support. Adding a function
pretending the contrary is misleading to external developers and plain wrong.

Post by Felix Paul Kühne
For the NPAPI plugin, we support a special rendering mode called
?windowless?, which does not make use of the existing libvlc vout modules.

That is a bug in the NPAPI plugin and a gap in modules/video_output/, not a
problem in LibVLC at this point. Using vmem as it currently stands cannot work
correctly for that scenario, regardless of this patch.