Interaction Rectangle

Interaction Rectangle

Rectangular geometry collecting and sending interaction events via UDP or ANC channel

The Interaction Rectangle node renders a rectangle geometry, collects all interaction (mouse, touch, keyboard) and sends this data in an Open Sound Control based protocol via UDP or the VIO ANC channel for further processing by the Live Video texture provider.

Node Usage

This node was basically introduced to provide the possibility of interaction with a Live Video texture source. Possible scenarios could be:

• A VIO implementation in a game engine providing the render output and giving the possibility to interact with the game engine scene
• Another VIO application could be the interaction with a Virtual Machine like ​Virtual Box (Remote Desktop from within Ventuz)
• A simpler 'Virtual Machine' application could be accomplished by using a Video Input card to grab the content of another computer; this computer would require a service which receives the interaction data via UDP and injects mouse, keyboard and touch commands

All these applications of course require software development and custom modifications on side of the 'video texture provider'.

The Interaction Rectangle provides some input properties to control the behavior of the node. With InteractionReceiver you define if the interaction data should be sent via UDP or the ANCillary Channel of the Live Video Input. The ReceiverID is needed to specify the receiver of the interaction messages. In ANC mode the ReceiverID must match the Ordinal (index) of the Live Video input. IP and Port define where to send via UDP. The ScreenAligned property toggles the geometry size between 1x1 and full layer. The Interaction property defines which types of interaction events (Keyboard, Mouse, Touch) should be sent. If you disable the node via the Enabled property, it does not react to any interaction and does not send any data. The TouchGroupFilter property which all Interaction nodes have in common is described here.
The mouse and touch events are only sent if they were generated inside the boundaries of the Interaction Rectangle geometry. The position is always mapped to 0-1, 0-1. Keyboard events are always sent if the renderer window has the focus and Keyboard is enabled in the Interaction property.
Via the Custom Verbs of the Property Editor it is possible to add output property sets which provide information about active Mouse and Multi-Touch events. A description of the fixed output properties and events, that all Interaction nodes have in common, can be found here.

Message Format

The interaction events collected by the node are formatted as OSC bundles. For details about the OSC format see the Remoting 4 Protocol Specification. The message format may look complicated. The reason for this is that it is based on our internal Input Subsystem format that is used for input synchronisation in a multi-machine Cluster.

The message structure looks like follows:

OscBundle		Address	Arguments Array
	OscElement	/ventuz/<i_ReceiverID>/InputTimecode	<d_Timecode>
	OscElement[n]	/ventuz/<i_ReceiverID>/<s_DeviceType>/<i_DeviceID>/<s_ID>/<i_TouchGroup>	<i_TouchType>, <s_TouchID>, <f_X>, <f_Y>, <f_Z>, <f_Pressure>, <f_Angle>
			-
			<i_Value>
			<f_X>, <f_Y>, <f_Z>

The OscBundle contains at least two OscElements. The first OscElement carries the timecode message. Its address is /ventuz/<i_ReceiverID>/InputTimecode where <i_ReceiverID> is an integer number representing the receiver ID. The argument carried by this OscElement is a double value representing the timecode of the frame in which the interaction data was recorded. All subsequent OscElements carry interaction data; the address is always /ventuz/<i_ReceiverID>/<s_DeviceType>/<i_DeviceID>/<s_ID>/<i_TouchGroup>.

The address parameters have following meaning and content:

The different argument types and values are explained here: