OverviewGameplayCode FlowchartSetupConclusionReal world usageKey FeaturesImprovementsTakeaways

Bouncing Bally: A mixed reality game exploring the use of OpenCV in Unity for object and text detection

Unity | OpenCV — 2023
Project

Individual School Project

Timeline

7 days

Tools used

Unity, Figma

Overview
Project
Bouncing Bally is a mixed reality game that requires players to use real-world objects or their own hands to guide a virtual bouncing ball into a randomly positioned house. The game has multiple levels with increasing difficulty, and players must use their creativity and problem-solving skills to clear each level.
Gameplay
Exploring the game...When the game is started, the player enters the Main Scene, which acts as the main menu starting screen. Here, players can interact with balls continuously emitting from the top of the screen to become familiar with object interaction during gameplay. This can be considered a tutorial phase without guided instructions.

After warming up, players can trigger the game by writing PLAY anywhere in the webcam's view, bringing them to the gameplay stage. In the gameplay scene, the main player named Bally needs to be brought back home to a point on the screen.


Playing the game...The gameplay begins with a bouncing ball, aka the main player named Bally, being dropped from the top of the playview. The ball Bally bounces around randomly, and the player's objective is to guide it into its house that appears at a random location on the playview/surface. To do this, the player must use real-world objects (such as blocks, ramps, or even their own hands) to create paths for the ball to follow. They can also draw lines with a black marker to help guide the ball.
Level ProgressionAs the player progresses through the levels, the game becomes more challenging. For example, there may be multiple bouncing balls to guide, or the house may be positioned in a more difficult location. There may also be predetermined barriers in place that the player must work around.

The game can also incorporate conditions such as having different shapes of objects have different physical properties. For example, a curved object may cause the ball to bounce in a different direction, while a flat object may allow the ball to roll more smoothly.

Flowchart of code

Communication between Unity and OpenCV asset used
To make the virtual balls in Unity interact with the objects detected by OpenCV, I wrote a script(ContourFinder script) to send the data points of the contours of the detected objects to Unity, which maps the vector points to the polygon collider attached to the ContourFinder script (using Serialised Field).  

To detect text, I wrote a script(ScreenShot script) that takes a screenshot every 10 seconds and sends it to OpenCV for processing, with the output sent back to Unity. Using SceneManagement, the game play scene is triggered when the output is equal to PLAY.
Setup
To play Bouncing Bally, players will need a projector to display the virtual bouncing ball onto a flat surface for the best experience.
Conclusion
Bouncing Bally is a fun and engaging mixed reality game that challenges players to think creatively and use their problem-solving skills. With multiple levels and increasing difficulty, the game is sure to keep players entertained and engaged.

It can be designed as a mass group interactive gameThe game can be integrated with either a Kinect or Intel RealSense camera, along with projection mapping. People can participate in various ways based on their surroundings and available resources.
The game's simplicity and accessibility make it ideal for large group play, as it is easy for participants to understand and engage with.
ways to be utilised in the real world
1.   Physical Education
The game can be utilized in physical education classes to encourage students to engage in physical activity while simultaneously having fun. The game can also help students develop hand-eye coordination and spatial awareness.

2.   Cognitive Rehabilitation
The game can be used in cognitive rehabilitation therapy to help individuals recover from traumatic brain injuries or strokes. The game's various levels of difficulty and the need to move actual objects around can provide a cognitive challenge for patients, helping to improve their cognitive abilities.

3.   Entertainment
The game can provide a fun and interactive experience for individuals of all ages. The game's unique use of mixed reality technology can help immerse players in a dynamic and engaging environment.
what i achieved:
— Object detection and colliders on its contours
— Basic Text Recognition
— Particle trail effect on Bally the ball in gameplay (Start scene)
— Glow effect on falling blue balls in starting scene (Main Text draft scene) (not very obvious against bright/white backdrops)
— Gameplay sound effects on bounce/collision, Background music, etc
key features to be expanded on:
— Using object recognition to start (instead of text only)
— Virtual buttons rather than using the button UI on Unity
a. Once virtual button is covered using hand/tapped Start Scene is triggered
— Improving UI of falling balls and balls having a ripple effect after each bounce/collision
— Projection map game onto a surface
takeaways:
This was the first time experimenting with OpenCV though I have been learning Unity and C# for some time. So I encountered significant challenges while integrating the two.

Choose your resources wellI made the mistake of using the free trial version of OpenCVForUnity (downloaded fromhere)
Resulting in me spending a substantial amount of time simply trying to figure out how to get things to work the way it should in the asset package due to limited resources and tutorials online for fixing issues and guiding the integration process.

Plus some additional setbacks...— the label "OpenCVForUnity" was printed every time I used the example text recognition script, and I currently have no way of removing it.
— the free trial version does not support Webgl, which limited the ways in which I could use the software.

Given more time, I would explore how to incorporate OpenCV in Python into Unity instead of using an asset. This approach could have been a more flexible and efficient option, considering the abundance of tutorials and resources available online (which I found out a little too late into the project).