Assignment #3: Google Translate Instant Camera — Real-time translation using your smartphone camera, and the future of synthetically-real objects in AR

Google Translate has been around for quite some time now. Today we are going to discuss about one of its features – the Instant camera.

The classic Google Translate allows you to input a text in a foreign language and outputs its translation in your native language (or the language of your choosing). However, if you want to translate a text which is not on your computer, you may use the Camera feature of Google Translate, which uses image recognition technology to extract text from the captured image, and output a transcript of the translation to the user. Taking it a step further, the instant mode not only recognizes and translates the text found on the image, it also replaces the original text with the translated version of it, and that too in real-time!

Have a look at the below images where I try to translate the features of a coffee maker mentioned on it’s box in Spanish: Features of Coffee Maker in Spanish (Original)

Features of Coffee Maker in Spanish (Original)


Features of Coffee Maker translated to English with varying degrees of success (01) Features of Coffee Maker translated to English with varying degrees of success (02) Features of Coffee Maker translated to English with varying degrees of success (03)

Features of Coffee Maker translated to English with varying degrees of success


The English translation is pretty impressive, although not 100% accurate, but it gets the job done. Here is an image of the features of the Coffee Maker in English (as printed on its box) for your reference: Features of Coffee Maker in English (Original)

Features of Coffee Maker in English (Original)


The instant camera is a good example of applications of augmented reality. This kind of capability can be very useful for tourists, or people travelling to a foreign country for work. Oftentimes, the street-signs are in a foreign language and the Google Translate app can act like a magic lens in these situations, through which they can read everything in their own native tongue.

Another audience for this kind of technology is people who are trying to learn a foreign language. Using the app, people can easily verify whether their translation of a foreign text was correct or not.

In the future, when we have easy access to lightweight AR glasses or AR contact lenses, you won’t even have to take out your smartphone whenever you need to translate something in a language foreign to you. Your AR gear will automatically detect and show you the translated text without you even realizing that you were looking at something written in a foreign language!

Some advantages of this approach, with respect to translation use case, could be:

AR gear capable of “removing” something real and “replacing” it with something synthetic will have far-reaching applications apart from translating foreign languages. Hate how sunny it is today? Your AR gear will act like sunglasses, or even render synthetic clouds to simulate a cloudy day! Are you on a diet and don’t want to get tempted with all the McDonald’s and pizza outlets down the street? Your AR gear can hide it from you by replacing them with innocuous buildings.

However, as Uncle Ben said, “with great power comes great responsibility”. With everyone wearing AR gear all the time, the marketing and advertisement companies will soon cash-in on the opportunity to show you ads everywhere. Literally everywhere! Surely your AR gear could filter them out for you – much like an Ad-blocker app on your web browser, but it is annoying nonetheless!

Let’s imagine another, somewhat interesting scenario. You are in a bar and looking for a potential companion to date. You plug-in a bunch of parameters – male/female, height, weight, sexual orientation, BMI, you name it – and bam, the individuals matching your criteria are lit up, while the rest become uninteresting silhouettes.

On the flip side, imagine you were driving down the mountains, enjoying the sunrise in the afternoon – because you were too lazy to get up early in the morning – and suddenly you find yourself falling down a cliff because you clearly saw a road in front of you! Turns out, much like computer viruses today, your AR gear might also be prone to vulnerabilities that a malicious, or a psychopath hacker, might want to take advantage of!

While the future of this tech might seem very lucrative, like any new technology, it will come with it’s own set of pros and cons, some of which we discussed above. While in some use cases – like the translation use case and the hiding of McDonald’s use case – the level of control the user has over the synthetic getting mapped over the real, is benign, there are other use cases – such as the filtering out of people use case – which will surely warrant a debate over its moral, ethical and legal implications.