Project #1: Eat It24 Sep 2018
Eat It is an Augmented Reality app, developed using Vuforia and Unity.
Please watch the following video for a quick demonstration of the app and its features:
Table of contents
- Demo Video
- Table of contents
- How to use
- Tools Used
- Notes on Development
- Assets used
How to use
The Android version of this app can be downloaded from its GitHub releases page.
Alternatively, you can download the source code from GitHub:
- Download the zip file from master branch
- Unzip the contents and browse to the extracted folder
- Install Unity, if you don’t have it installed already
- Now browse to
Scenesand open the
MainScene.unityfile using Unity
- Switch on your webcam and hit the play button
You will also need to print the image targets to use with the app. These targets can be downloaded from here.
- 3 Multi-targets for cereal boxes, each with their own scenes
- 1 Cylindrical target for milk
- 2 Image targets – one acts like a placemat, and the other like a magazine
- Each scene makes use of its own lighting, as appropriate for the scene
- Placing objects on the placemat hides their scenes and displays nutritional information
- When not placed on the placemat, bringing any of the cereal boxes or the can closer, each prop exhibits a different reaction
- 3D magazine whose pages can be turned with a flip animation by using the virtual buttons
- Unity Game Engine
- Vuforia Augmented Reality SDK
- Blender 3D content creation tool
- Adobe Photoshop for Image Editing
- Audacity for Audio Editing
- VideoPad for Video Editing
Props to be used as targets for the app
3D Scene around the Placemat
Animated clouds in the Placemat Scene
3D Scene around the Drink target
3D Scene around the Cereal Box 1 Target
3D Scene around the Cereal Box 1 Target (Zoomed Out)
3D Scene with a Sci-Fi theme, around the Cereal Box 2 Target
Hovering Air Drone on top of the building
3D Scene around an Apple Jacks cereal box, as a real world example of this technology
3D Magazine whose pages can be flipped with an animation using the virtual buttons
Side by side – Cereal Box 1 and Drink Honey Bees coming out of their Bee-Hive and the telekinetic cow causing an earthquake, when the props are placed close enough
Side by side – Apple Jacks and Cereal Box 2
Apples falling from the apple tree and alien mushrooms grow when the two props are placed close to each other
Apple Jacks showing its nutritional information when placed on the placemat
Cereal Box 2 showing its nutritional information when placed on the placemat
Milk Bottle showing its nutritional information when placed on the placemat
Cereal Box 1 showing its nutritional information when placed on the placemat; Notice the Placemat’s mascot expressing his opinion whenever any of the drinks or boxes are placed on the placemat
Notes on Development
Stage 1 – The Noob
I found the project very challenging. It was the first time I was developing an AR application, but working with Vuforia and Unity made it very simple. With the help of Unity and Vuforia, you can easily display a scene on or around your real objects without wrtiting a single line of code!
Initially, I ended up spending most of my time to understand how to work with Unity and finding some good websites where I can find good quality (and free) assets to use in my scene. I found that SketchFab is a very good resource to search for any kind of 3D model. You will notice that most of my 3D models have been downloaded from this website. Most of the assets that I downloaded from other websites, didn’t work for me. Most of the times they will be missing the required
Materials, or they will be in a proprietary format, such as
.max (3DS Max) and
.maya (Maya), which can only be opened if I had access to their softwares (which I clearly didn’t have).
Upon importing an asset, most of the times they don’t work out of the box. You need to manually map the materials with the provided textures, and play with the material’s settings, to make it resemble the original object as depicted on the artist’s render of the object.
Stage 2 – Trying to create cool stuff using Blender
I couldn’t find any good, free resources for some of the objects that I wanted to put in my scenes, so I tried to create them on my own. At first, I tried SketchUp, but found the features of the free version to be very limited, and I didn’t like the idea of working inside a browser as well. Hence, I spent a fair share of my day learning Blender, which, at first, was very confusing to use – who uses right-click as a primary button!! But, I soon realized how powerful it was. I found the Introduction to Blender series of tutorials, by Brackeys very helpful to get started with Blender, and it’s probably the best tutorial I ever came across on YouTube.
Using Blender, I created a Milk Bottle for the Drink 1 Scene, and used Photoshop to create a simple texture to brand the milk bottle as Legendairy Milk. The image of the cow was taken from the 3D model I used for the cow.
Next, I created a 3D model for a Bee Hive. I followed this tutorial by pauljs75, to get started with creating a honeycomb pattern in blender, but I had a hard time trying to make it look like a bee hive. For the sake of my sanity, I finally had to give up trying to create a perfect bee hive and settle for something which I can pass off as one.
I also had to use Blender for fixing some intersecting polygons in the model that I used for Cheese, since Unity was happily not rendering half of the upper face of the cheese. For the building in the Cereal Box 2 scene (the Sci-Fi one), I had to chop off all the buildings that came with the 3D model that I downloaded, except the one that I wanted. Luckily, for both these models, the original artist had provided the asset in
Stage 3 – Adding Animation
Unity’s animation engine is very powerful, but lacks a very important feature – if you animate the properties of some objects in your scene, and then decide to change its hierarchy, to better organize your scene, you will end up breaking the references to those objects, and consequently, your animation.
Some of the models that I used came with an
animation clip, such as:
- the fidgeting cow
- bee flapping its wings
- the entire animation of the Buster Drone
- spinning of blades of the Air Drone
- swaying of the Green Tree and the Birch Tree
- mouth movement of the Chatter Teeth (Placemat’s mascot)
I just had to create an
Animator Controller for the above, and add these animation clips to the controller as the
All the other animations were done by me, and were orchestrated with the help of
parameters in the
Animator Controller. Here is a list of animations that I added to my scenes:
- The movement of clouds in the placemat scene – the animation is cyclic, i.e. when the clouds reach a certain distance on the left part of the scene, they reappear on the right side, so that the user always sees clouds in the middle of the scene
- The honey bee collecting nectar from the flowers and depositing them in the bee-hive – Notice that the speed of flapping wings changes dramatically when the bee is flying, vs. when it is not. Additionally, the buzzing stops whenever the bee is not flying.
- Swarm of bees that appear when any other object is kept close to Cereal Box 1
- The wings flapping and the hovering animation for The Wasp
- Hovering animation for the Air Drone
- Growing/Shrinking of mushrooms when any other object is kept close to Cereal Box 2
- Shaking of all the objects when any other object is kept close to Drink 1
- Falling of apples from the tree when any other object is kept close to Apple Jacks
- 3D page turn animation of the magazine
Stage 4 – Adding Interactivity and Encountering Bugs
Till now, all the scenes were just there, whenever an object came in the camera’s view. The next step was to add interactivity.
To detect when two objects were close to each other, I added
Colliders to all my scenes with the
Is Trigger option checked. I also added the
Rigid body component to these scenes so that my scripts may receive the
OnTriggerEnter and the
OnTriggerExit events whenever collisions are detected. The script will then check which object it collided with, and appropriately
trigger the animations. If the objects collide with the placemat, their corresponding scenes will be hidden and their nutritional information will be shown instead (all the nutritional information was sourced from nutritionix.com).
During testing, I found that my cereal boxes were going halfway inside the placemat, and when placed side by side, one would appear to be placed closer to the camera as compared to the other. I spent a couple of hours trying out different things to fix the issue, without any success. Finally, I decided to measure the physical size of all my props, and re-upload all the target images to Vuforia Target Manager and download the new database.
Updating my database with a new one, messed up all of my scenes, and I had to rescale and position everything again. I ended up wasting a full day due to this, but thankfully it solved the issue!
Stage 5 – Designing the Magazine and Lighting the scenes
First step was to create the animation and code the interactivity. I wanted to have a 3D page turn animation whenever the user touched (or rather, obstructed) the virtual buttons. After searching and trying out different models for a book, I came across a 3D model that had a nice page flip animation. After importing it in Unity, I found that the model actually consisted of several pages, each acting like a keyframe in the different stages of a page flip animation. Hence, to animate the page flip, I created an
animation clip that would hide and show the different pages. Now that the animation clip was ready, I wrote a script to appropriately change all the materials of these pages and trigger the animation whenever the virtual buttons were pressed.
The next step was to design the content. I decided to create a health magazine, consisting of a few healthy recipes for breakfast. I picked up a few recipes from Stacy Homemaker, Cooking Light, and Fitness Magazine, and designed the cover and the pages in Photoshop. Afterwards, it was just a simple matter of pointing my script to these images for the different pages.
For lighting my scenes, at first, I was using the default
Directional Light, but they looked terrible with the same lighting configuration for all of them, especially, the Sci-Fi scene around Cereal Box 2, which had
Emmisive materials for its 3D objects. So I removed the directional light and used 3
Point Lights for each scene, to effectively illuminate all parts of the scenes, and at the same time create a more warmer feel (as in the Honey themed Cereal Box 1 scene), or the more dark feel (as in the Sci Fi scene).
Stage 6 – The ill-fate of the Apple Jacks scene, and doing the submission
Due to shortage of time, I was not able to dedicate enough time to work on the additional requirements for Graduate students. So, I decided to use a box of Apple Jacks as my image target. I had initially bought the box to paste the cutouts for the two Cereal Box props (given to us by Professor Johnson), but the size didn’t match the prints and I wasn’t able to use them. Needless to say, the scene can use a lot of refinement as compared to the other scenes.
For the submission, I decided to build and run the app on my Android device, since I was experiencing some lag with my webcam and the video recording would be bad. I used AZ Screen Recorder app to record my phone’s screen, along with my narration while recording the video. I then looked for a decent video-editing software, and after trying out a few of them, I used VideoPad to put my video together, since none of the others were able to play the videos recorded from my phone. The last step was to upload the video on YouTube, and make this webpage.
This project uses a lot of 3D models, textures and sound effects, which are freely available for non-commercial use. Here is a detailed list of all the models used and where I got them from.
The list is organized scene wise, and then according to the type of asset (3D object, Sound, Texture). The numbers on the image for each scene, indicates the
S. No. of that object in the list.
Placemat 1 Scene
|1||Bread||SketchFab||andree||Creative Commons Attribution|
|2||Pancakes||SketchFab||soidev||Creative Commons Attribution|
|3||Plate||SketchFab||WinterPlayer||Creative Commons Attribution|
|4||Rug||SketchFab||Nicholas Record||Creative Commons Attribution|
|5||Clouds||SketchFab||PotatoBeenCrafted||Creative Commons Attribution|
|6||Fruit Basket||SketchFab||John Greenstone||Creative Commons Attribution|
|7||Chatter Teeth||SketchFab||Damco||Creative Commons Attribution|
|8||Umbrella||SketchFab||Le.Manh.Cuong||Creative Commons Attribution|
|9||Ground||SketchFab||Le.Manh.Cuong||Creative Commons Attribution|
|1||Grass||3D Textures||João Paulo||CCO 1.0 Universal - Public Domain Dedication|
|1||Nature Ambiance||FreeSound||blouhond||Creative Commons Attribution 3.0|
|4||Woo Hoo||PacDV||PacDV||Royalty Free|
|5||Ooh La La||FreeSound||blouhond||Creative Commons Attribution 3.0|
Cereal Box 1 Scene
|1||Bee Hive||-||Mayank K Rastogi||-|
|2||Bee||TurboSquid||can ocak||Royalty Free License - All Extended Uses|
|3||Green Tree||SketchFab||MaximilianY.||Creative Commons Attribution|
|4||Birch Tree||SketchFab||Alan Zimmerman||Creative Commons Attribution - Non-commercial|
|5||Colored Flowers||SketchFab||tojamerlin||Creative Commons Attribution|
|6||White Flowers||SketchFab||tojamerlin||Creative Commons Attribution|
|7||The Wasp||SketchFab||Hector Mora||Creative Commons Attribution|
|8||Ground||SketchFab||Le.Manh.Cuong||Creative Commons Attribution|
|1||Grass with Flowers||3D Textures||João Paulo||CCO 1.0 Universal - Public Domain Dedication|
|1||Bee Buzzing||SalamiSound||SalamiSound||Free for non-commercial use|
Cereal Box 2 Scene
|1||Air Drone||SketchFab||Jason.Arens||Creative Commons Attribution|
|2||Buster Drone||SketchFab||LaVADraGoN||Creative Commons Attribution - Non-commercial|
|3||Sci-Fi Plants||SketchFab||DarkLordFlash||Creative Commons Attribution|
|4||Building||SketchFab||d880||Creative Commons Attribution|
|5||Helipad||SketchFab||stefan89||Creative Commons Attribution|
|6||Ground||SketchFab||Le.Manh.Cuong||Creative Commons Attribution|
|1||Terrain||3D Textures||João Paulo||CCO 1.0 Universal - Public Domain Dedication|
|1||Sci-Fi Ambiance||SoundBible||Daniel Simion||Creative Commons Attribution 3.0|
Drink 1 Scene
|1||Milk Bottle||-||Mayank K Rastogi||-|
|2||Coffee Maker||SketchFab||amrmosho||Creative Commons Attribution|
|3||Plate||SketchFab||WinterPlayer||Creative Commons Attribution|
|4||Cheese||SketchFab||Christian Gentry||Creative Commons Attribution|
|5||Coffee Mug||SketchFab||French Baguette||Creative Commons Attribution|
|6||Teapot||SketchFab||WinterPlayer||Creative Commons Attribution - Non-Commercial|
|7||Cow||SketchFab||shakiller||Creative Commons Attribution|
|8||Counter-top||3D Textures||João Paulo||CCO 1.0 Universal - Public Domain Dedication|
|1||White Marble||3D Textures||João Paulo||CCO 1.0 Universal - Public Domain Dedication|
|1||Cow Moo||SoundBible||BuffBill84||Creative Commons Attribution 3.0|
Apple Jacks Scene
|1||Cereal Bowl||TurboSquid||cgs3d||Royalty Free License - All Extended Uses|
|2||Green Tree||SketchFab||MaximilianY.||Creative Commons Attribution|
|2||Yellow(ish) Tree||SketchFab||MaximilianY.||Creative Commons Attribution|
|4||Ground||SketchFab||Le.Manh.Cuong||Creative Commons Attribution|
|1||Grass||3D Textures||João Paulo||CCO 1.0 Universal - Public Domain Dedication|
|2||Apple||SketchFab||Le.Manh.Cuong||Creative Commons Attribution|
|1||Book||SketchFab||John Fino||Creative Commons Attribution|
|1||Page Flip||Sound Jay||Sound Jay||Royalty Free|
[Edit] Oct 3, 2018: Updated instructions to open the “MainScene.unity” scene after downloading the project; Added table of contents