You can also use the GestureManager to create such GestureCombinations and then load the resulting file with a MiVRy actor.

(3) How do I identify gestures without having to trigger the “start” and “end” of the gesture.

To identify gestures continuously without a clear “beginning” and “end”, use the “contdIdentify” function. You still have to call “startStroke” once (for example at the start of the level), and have to continuously provide the controller position with the “contdStroke” function. Then, after the “contdStroke”, use the “contdIdentify” function to identify the currently performed gesture. Use the contdIdentificationPeriod value to control how long of a time frame to consider in the identification. You can also use contdIdentificationSmoothing to avoid the identification result from jumping from one gesture ID to another too easily.

(4) How can I open and edit gesture database (.DAT) files?

Please use the “GestureManager” ( https://www.marui-plugin.com/documentation-mivry-unreal/#gesturemanager ) to open and edit “.DAT” gesture database files. Please note that the GestureManager is based on Unity, so if you record gestures in the GestureManager and want to use them in Unreal, enable the “Unity Compatibility Mode” in your MiVRyActor, GestureRecognitionActor, or GestureCombinationsActor.

(5) The Gesture Recognition library does not detect if a gesture is different from all recorded gestures. How do I find out if the user makes a completely new (or “wrong”) gesture?

The gesture recognition plug-in will always return the number of the gesture which is most similar to the one you just performed.
If you want to check if the gesture you made is different from all the recorded gestures, check the “Similarity” value that you receive when identifying a gesture. This is a value between “0” and “1”, where “zero” means that the gestures are completely different, and “one” means that the performed gesture is a perfect average of all the recorded samples.

Thus, a value of one will indicate perfect similarity, a low value close to zero indicate great differences between the performed gesture and the recorded gesture samples. You can use this value to judge if the performed gesture is sufficiently similar to the recorded one.

(6) I want to use Gesture Recognition in my commercial project. What commercial licensing options do you provide?

Please contact us at support@marui-plugin.com for details.

(7) Do I have to use the “startTraining” function every time I start my game? Does it have to keep running in the background while my app is running?

No, you only need to call startTraining after you have recorded new gesture data (samples) and want these new recordings to be used by the AI. However, you need to save the AI after training to a database file (.DAT) and load this file in your game before using the other gesture recognition functions.
While the training is running, you cannot use any of the other functions, so you cannot let training run in the background. You must start (and stop) training in between using the AI.

(8) How long should I let the training run to achieve optimal recognition performance?

Usually, the AI will reach its peak performance within one minute of training, but if you’re using a large number of gestures and samples, it may take longer. You can check the current recognition performance from the training callback functions and see if the performance still keeps increasing. If not, feel free to stop the training.

(9) Gestures aren’t recognized correctly when I look up/down/left/right or tilt my head.

You can choose if the frame of reference for your gestures are the players point of view (“head”) or the real world or game world (“world”). For example, if the player is looking up to the sky when performing a gesture towards the sky, then from a “world” frame-of-reference the direction is “up”, but from players “head” point-of-view, the direction is “forward”. Therefore, if you consider your gestures to be relative to the world “up” (sky) and “down” (ground) rather than the visual “upper end of the screen” and “lower end of the screen”, then change the frameOfReferenceUpDownPitch to FrameOfReference.World. The same setting is available for the yaw (compass direction) and head tilt.

(10) MiVRy identifies any motion as some gesture, even when it doesn’t resemble any of the recorded gestures. Why? How can I tell if no valid gesture motion was performed?

MiVRy will always tell you the “most likely” best guess as to which gesture was just performed, no matter how different the currently performed motion is from all recorded gestures. This is because we cannot decide for you how much difference is tolerable.
In order to disqualify “wrong” motions, you have two options:
(A) you can check the “similarity” value returned by MiVRy. This value describes how similar the gesture motion was compared to previous recordings on a scale from 0 (very different) to 1 (very similar).
(B) you can check the “probability” value. Especially when you compare the probability values for all recorded gestures (for example via the “endStrokeAndGetAllProbabilitiesAndSimilarities” function) and see that they are all very low and not very different from one another, you may want to decide that the current gesture performance was not valid.

(11) What exactly does die “similarity” value of a gesture performance mean? How is it different from the probability value?

The “similarity” value expresses how much the identified gesture differs from the average of the recorded samples for that gesture. When you record several samples, MiVRy internally calculates a “mean” (“average”, “typical”) gesture motion based on those samples. It also calculates how much the recorded samples differ from this “mean” (ie. the “variance” of the samples). The “similarity” value is then calculated based on this “mean”. If your newly performed gesture motion hits exactly this “average”, then the similarity value will be one. The more it differs, the lower the “similarity” value will be, going towards zero. How fast it will fall depends on how similar the recorded samples were. If all recorded samples looked exactly the same, then MiVRy will be very strict, and the “similarity” value will fall fast when the currently performed motion isn’t also exactly alike. If, however the samples differed a lot, MiVRy will be more tolerant when calculating the “similarity” value and it will be higher. The value is always between 0 and 1. This “similarity” is different from the “probability” values, which are estimates by the artificial intelligence (neural network). “Probability” may contain many more considerations, for example if there are other gestures who resemble the identified gesture (probability may drop, similarity is unaffected), or if there are a multitude of distinct motions lumped together as one “gesture” (for example: having a gesture “alphabet” which contains drawings of “A”, “B”, “C” etc all lumped together as one gesture – then “similarity” will be calculated based on an “average” character that doesn’t resemble any sample, but the AI may successfully understand what you mean and give high “probability” values).

(12) Instead of triggering the start and end of a gesture motion, I want MiVRy to constantly run in the background and detect gestures as they occur.

You can use the “Continuous Gesture Identification” feature of MiVRy. When using the “GestureRecognition” or “GestureCombinations” objects directly, use the “contdIdentify” function – you can call this function repeatedly (for example on every frame or when something in your app happens) and every time it will tell you which gesture is currently being performed. When using the Unity “Mivry” component or the UnrealEngine “MivryActor”, use the “Continuous Gesture Identification” switch. Either way, two settings are important for Continuous Gesture Identification: “Continuous Gesture Period” and “Continuous Gesture Smoothing”. “Continuous Gesture Period” is the time frame (in milliseconds) that continuous gestures are expected to be. So if your gestures take 1 second to perform, set this to “1000” so that MiVRy will consider the last 1000 milliseconds to identify the gesture. “Continuous Gesture Smoothing” is the number of samples (previous calls to “contdIdentify” to use for smoothing continuous gesture identification results). When setting this to zero, each attempt to identify the gesture will stand alone, which may lead to sudden changes when switching from one gesture to another. If ContinuousGestureSmoothing is higher than zero, MiVRy will remember previous attempts to identify the gesture and will produce more stable output.

(14) What is the “Update Head Position Policy” / “Compensate Head Motion” setting?

This setting decides whether the AI should consider changes in head position during the gesturing.
During gesturing, the current position of the VR headset can/will be updated via the “updateHeadPosition” procedure.
This data is saved together with the motion data.
However, in many cases it is not advisable to take head motions during gesturing into account, because people may watch their hands while gesturing.
Following the moving hands with the head would then eliminate the hand motion relative to the headset (the hands would always be “in front of the headset”).
However, in some cases it may be useful to use the changing head position, for example if the user might be walking during a gesture.
You can chose whether or not the the data provided via calls to “updateHeadPosition” functions will be used with the UpdateHeadPositionPolicy (or call to GestureRecognition.setUpdateHeadPositionPolicy().
“UseLatest” will cause MiVRy to use the changing head position, thus compensating the relative head motion during gesturing.
“UseInitial” will not consider changes in head motion during gesturing, but only the head position at the start of the gesture.
Note that if you use a GestureRecognition or GestureCombinations object directly, you also need to provide the changing head position via “updateHeadPosition()” for this to have any effect.
Also note that the data provided via “updateHeadPosition” is stored regardless of the policy, even when it is not used later.