Project #3: ARray of Things

The Array of Things is a collaborative effort among leading scientists, universities, local government, and communities to collect real-time data on urban environment, infrastructure, and activity for research and public use. Our project attempts to visualize some of this data in Augmented Reality.

Demo Video

Watch the following video on YouTube for a quick demo of the application

Download

Source code on GitHub

Android App

Table of contents

  1. Demo Video
  2. Download
  3. Table of contents
  4. Hardware Requirements
  5. How to run
  6. Features
  7. Libraries used
  8. Known Issues / Limitations
  9. Team
  10. Overview
  11. Application Description
    1. AoT Nodes Augmented
    2. Visualized Data
    3. Understanding time for which the data is shown
  12. Application Components
    1. Splash Screen
    2. Permissions and ARCore installation
    3. Main Screen
    4. User Actions
    5. Augmented Model
    6. Drawer Menu
    7. Toolbar
    8. Settings Menu
  13. Screenshots
  14. References

Hardware Requirements

  1. Android 8.0 (Oreo) or above
  2. ARCore supported device (List of supported devices)

How to run

  1. Download the android app from the GitHub releases page, on your ARCore supported Android device
  2. Tap Open or Install on the file – you’ll get a popup saying this app (or phone) doesn’t have permission to install apps from an unknown source. Tap Settings to grant it permission
  3. Now flip the switch to ON where it says “allow from this source”
  4. Hit back to restart the process, and select Install to get started
  5. Tap on Open to open the app once it is installed

Features

Libraries used

Known Issues / Limitations

  1. Various use cases of intermittent internet connection are not handled in the application
  2. When fetching data for all the nodes within a specific distance, if any of the API’s responds with an internal server error then the application does not continue to load the AR models and keeps waiting for the API to respond. This is because models are loaded only after all the nodes have returned the required data
  3. As the application depends on ARCore SceneForm, the the known issues of SceneForm are propagated into our application
  4. Due to difference in magnetic north and true north the phones compass is deflected by plus or minus 15 degrees resulting in geolocations of AoT sensor nodes being off by 15 degrees. This is handled to some extent by adjusting the bearing
  5. If the phone is locked and the app is opened in the backend. On unlocking the phone and opening the app again the app will not load models. Some lifecycle events are not handled when the app resumes after closing and opening the the already running app
  6. The data for the nodes is only available after 20th October 2018 from the Array of Things API
  7. ARCore seems to struggle with Anchors that are more than 1000 meters away, so for optimal performance anchors are redrawn every 5 seconds

Team

Overview

Array of Things is an urban sensor project aimed at changing the way we understand cities and urban life. This project is a collaborative effort by various entities to capture real-time data in various areas of a city to help gain a deeper understanding of urban infrastructure and environment for research and public use. As a part of the initiative, various sensor nodes are placed at multiple locations in a city to gather data on the environment, air quality and light at that particular locations. The data gathered by each sensor is available via an API. Currently, the project is undertaken by many cities like Chicago, New York, Seattle, Alto, Seattle etc. Chicago has over a hundred AOT sensor nodes.

Application Description

Project Proposal

Pen-paper Mockups

For our project, we have developed an AR application (ARray of Things) to visualize AOT sensor data. Our application is a native Android application which makes the use of AR core to visualize the AR components. The application augments the geolocation of AOT sensor nodes which lie within a certain distance of the users GPS location with an augmented model of different sensor data. The user has the option to change the day and hour of the day to view AoT node sensor data at that point of the day. The user also has an option to select the category to view data relating to a particular category. The augmented models allow the user to compare data of an augmented node with data of different AoT nodes at different days and time.

AoT Nodes Augmented

The AoT nodes whose geolocation is augmented by the application, are dependent on a distance parameter. Based on the distance parameter the application looks for all the nodes within the distance parameter value from the user’s GPS location and augments the geolocation of all the found nodes with augmented models. The user has an option to change the distance parameter from 0 meters to 5000 meters. By default, the distance parameter is set to 2000 meters when the application launches. The user can change the distance parameter from the settings menu.

Visualized Data

Now each AoT node captures various types of data from a variety of sensors on it. For our application, we show data from nine different data types which we have categorized into three categories namely weather data, air quality data and light data. The data for each of the nine different data types is captured by nine different sensors present on each AoT node.

Following are the types of data the user can view in each category:

# Category Measured Value Sensor Used Unit of Measurement (metric)
1 Weather Temperature PR103J2 °C
2 Weather Pressure BMP180 hPa
3 Weather Humidity HIH4030 %
4 Light Visible Light APDS 9006 020 lux
5 Light Infrared Light TSL260RD μW/cm2
6 Light Ultraviolet Light ML8511 μW/cm2
7 Air Quality CO Concentration Carbon Monoxide ppm
8 Air Quality SO2 Concentration Sulphur Dioxide ppm
9 Air Quality NO2 Concentration Nitrogen Dioxide ppm

The user can select a particular category from the drawer menu layout.

Understanding time for which the data is shown

The user can view the sensor data gathered by the AoT node at a particular time by changing the date and hour. The application aggregates all the values obtained by the sensor for the selected hour and displays the aggregated value for the selected hour to the user. As an example, if the user launches the application and the current time is 3.15 pm on the 1st of December 2018, then the application will fetch the required sensors data for each node from the array of things API for December 1, 2018. Since the time is 3.15 pm the application will aggregate the all the data from hour 3.00 pm to 3.15 pm for each sensor and display the aggregated hour data in the augmented model collected on Dec 1 2018. If the user selects the hour to 10, then the application will aggregate the data from hour 9 to hour 10 for the selected day and display it to the user. If no data is found for the selected hour then the application will display as NF (Not Found) to the user.

Application Components

Once the application is installed on an android phone the application shows up in the app drawer with the below logo.

ARray of Things - Logo

Splash Screen

On opening the application a splash screen is shown while the main application is loaded.

ARray of Things - Logo

Permissions and ARCore installation

If the application is loaded for the first time and ARCore is not installed on the phone then the application prompts the user to download ARCore, if the user accepts the download of ARCore the application downloads the ARCore application form the google store and proceeds with asking the user for permissions to run the application. The application asks the user for granting the following permissions to the application to run.

  1. Location: The location permission is required to get the GPS location of the user and the Geo Location of the AOT sensors so that the user’s distance to the AOT markers can be calculated
  2. Camera: Camera permission is required to allow the application to access the camera so that the augmented models can be placed on top of the camera view on the geo-locations of the AOT sensor nodes when the user moves the camera in the real world
  3. Internet: Internet permission is required to make a call to the array of things API to get the AOT sensor node data

When the user accepts all the permission requests, the application continues to launch the main application. If the permissions have already been granted to the application then, on future launches the permission is not requested and the application proceeds to run.

Main Screen

The main screen of the application contains a camera view on which the geolocation of the AoT nodes is augmented by AR models which contain AoT sensor node data. The data shown in the augmented models is based on the category of data selected and the date and hour or day. The augmented model shows data for the selected day and aggregates all the values recorded with the selected hour of the day.

On launching the application the GPS location of the mobile device is obtained and all the AoT nodes within a distance of 2000 meters from the GPS location are located. For each of the nodes and API calls are made to get the data for weather category for the current day. The default distance parameter is 2000 meters, the default category is weather category and the default time for which the data is shown in the current day and hour. By default, the data is aggregated for all the values in the current hour of the current time and shown.

User Actions

The user can perform the following actions in the application:

  1. Change the distance parameter to filter nodes based on distance from the GPS location of the device
  2. Change the data category to view data of a different category
  3. Change the hour of the day to view the data aggregated for the particular hour of the day of the selected day. If the user changes the data from hour 12 to hour 17, the application aggregates the data from all the observations gathered between hour 16 and hour 17 and shows it to the user
  4. Change the day to view the data on the selected day and selected hour
  5. Refresh GPS location to regather all the nodes and data for each node for current GPS location of the user
  6. Interact with the augmented model placed on the geolocation of each AoT node captured to view comparison between data, data trends with graphs and aggregations of data. This is explained in the augmented model’s section

Augmented Model

The augmented model is placed on top of the geolocation of each AoT sensor and show the data captured by the sensors based on the category of data selected and the day and hour of day selected.

The augmented model consists of four parts:

1. Inner Layout

The inner layout is augmented by default on the geolocation of each AoT sensor node detected by the application. The number of AoT sensor nodes detected depends on distance parameter and users GPS location. The inner layout shows the following data:

  1. Location Label: The location of the AoT sensor node
  2. Distance Label: Show the distance of the user from of the AoT sensor node. This distance is updated every frame
  3. Data Label 1: Shows Temperature if category selected is weather, shows Visible Light intensity if the category selected is Light, shows Carbon Monoxide value if the category selected is Air Quality
  4. Data Label 2: Shows Humidity if category selected is Weather, shows IR Light intensity if category selected is Light, shows Sulphur Dioxide value if the category selected is Air Quality
  5. Data Label 3: Shows Temperature if category selected is Weather, shows visible Light Intensity if category selected is Light, shows Nitrogen Dioxide value if the category selected is Air Quality

The data 1, data 2 and data 3 label is updated for the selected hour and selected day if the user decides to change the hour and day parameter.

2. Graph Layout

The graph layout appears when the user taps of any of the labels data1, data2 and data3 in the inner layout. Based on the shown by the label tapped a graph is constructed which shows the trend of all the values of that data type recorded by the sensor in the selected day. The graph layout shows a graph all recorded observations on the x-axis and value of each observation on the y-axis.

For example, if the user has selected category of weather, then data1 label will show temperature data recorded at the selected day and hour by that nodes temperature sensor. Now if the user taps on the data1 label of an inner model of an AoT node. A graph will appear showing all the recorded temperature values by the temperature recording sensor at that node for the selected day. The Y-Axis will show value magnitude and X-axis will show all the recorded values.

3. Comparison Layout

The comparison layout is used to compare an AoT nodes category values with other nodes category values. The comparison layout appears when a user long presses the inner layout of any node. Only one comparison layout is shown at one time and the comparison layout is fixed vertically on the left-hand side of the mobile screen. The layout does not disappear as it is not associated with the geolocation of the any AoT node.

On long press of any inner layout, the current data label values of the inner layout get copied to the comparison layout. Now the user can move the device to compare the values of a particular AoT node with other nodes.

4. Aggregate Layout

The aggregate layout also appears when the user clicks on any of the labels data1, data2 and data3 in the inner layout. It appears below the inner layout. The aggregate layout shows the median, maximum and minimum values of the data type the label is showing for the selected day.

Drawer Menu

The drawer menu is opened up by swiping from left to right on the left-hand side of the mobile screen or by tapping the hamburger menu icon in the toolbar. The drawer menu allows the user to select which category of data to show in the augmented model placed on the geolocation of each AOT node sensor. The user has the option to select from three categories

  1. Weather: The augmented model shows temperature, pressure and humidity data
  2. Light: The augmented model shows visible light, infrared light, and UV light data
  3. Pollution: The augmented model shows concentration levels for Carbon Monoxide (CO), Sulphur Dioxide (SO2), Nitrogen Dioxide (NO2)

Toolbar

On the top of the camera is a toolbar which shows the user selected date and time in the center. When the application has launched the date and time are set to the current date and time by default. The toolbar contains a hamburger menu of the left which opens up a drawer menu. The toolbar contains a settings menu its right and side which opens up the settings menu

  1. Hour Picker: The hour picker is used to change the selected hour for which the data is shown. The application will aggregate all the values gathered by a sensor for the selected hour on change of the hour by the user. If no data is found for the selected hour ‘Not Found’ is shown to the user
  2. Date Picker: The date picker is used to change the selected day for which to view the data. On the change of the day, the application will make an API call to gather all required sensor data for all the nodes for the selected day and refresh the layout.

Settings Menu

The settings menu opens up when the taps on the settings menu on the right of the toolbar. The user can change the following from the settings menu:

  1. Change the distance parameter: The user can change the distance parameter from 0 meters to 5000 meters. Based on the changed distance parameter the application filters out AoT sensor nodes with the value of the distance parameter set from the device GPS locations and fetches data for each node detected. The main screen is refreshed with the new augmented models placed at the geolocations of detected nodes
  2. Toggle between metric and imperial units: The user can opt to view the data values in either imperial units by turning on the toggle bar to show imperial units

Screenshots

Launch Screen Launch Screen

App requesting permissions Launch Screen

App determining user’s location Launch Screen

App fetching sensor data from nodes nearby the user Launch Screen

App busy rendering the markers on scene Launch Screen

Sensor data from AoT node at Racine & 18th Street Launch Screen

Default Settings Launch Screen

User changing the distance setting to 1400 meters Launch Screen

User switches to use imperial units Launch Screen

Weather data being shown in Fahrenheits, and inches of mercury after changing settings to use Imperial units Launch Screen

User switches to view light data Launch Screen

Light data being shown from sensor node at Racine & 18th Street Launch Screen

Changing the hours filter changes the data immediately Launch Screen

When changing hours filter to beyond current time, “No Data” is displayed Launch Screen

On tap of “Light” button, trend for the selected day is shown, along with min, median, and max of the day Launch Screen

Trend for pressure data being shown for the day Launch Screen

On long press of a node, the current data is pinned to the left for easy comparison with other nodes, and different time periods Launch Screen

On tapping the date, a date picker shows up Launch Screen

User changing the date to Nov 30, 2018 Launch Screen

Weather data on Nov 30, 2018 being compared against that of Dec 3, 2018 Launch Screen


References