Multimedia Sensing

• Purpose
• Main Functions
• MM-AQI
• MM-TrafficEvent
• Data loader
• Applications
• Configuration
• Operating environment
• Precautions
• Reference

Purpose

This information asset is derived from the Environmental Quality Short-Term Prediction^[1], which learns and predicts air quality (AQI) rank and incident occurrence from image data of the surrounding environment acquired with a portable camera.

Main Functions

Input features for air quality rank (AQI rank) prediction are extracted from image log data of the surrounding environment acquired by drive recorders and lifelog cameras. The LXMERT method used in VQA (Visual Question Answering) with images is extended to use a correlation search engine for image logs and environmental descriptions, and combined with Taxonomy, enables performance tuning specific to the target domain.

MM-AQI

Predicts air quality (AQI) ranks using Image-2-AQI^[2], a method for predicting environmental quality from image logs such as drive recorders and lifelog cameras.

MM-TrafficEvent

Incidents that could lead to accidents are predicted from image logs from in-vehicle cameras and other sources using the EfficientNet^[3] image recognition model.

Data loader

The environmental observation data to be used in MM-AQI is acquired in advance, and the data for analysis in event data format is extracted and registered in the database. The data for analysis in the form of event data is extracted and registered in a database.

Applications

• MM-AQI: Research and development of safe living environments, tourism support, traffic measures, etc.
• MM-TrafficEvent: AI dashcam video and more to find accidents and traffic obstacles

Configuration

This information asset consists of.

• Processing program（Python, R）
  • Transaction Generation Program
  • EfficientNet Predictive Model Generation Program
  • EfficientNet Predictive Model Learning Program
  • Pattern mining processing program
  • Air Quality (AQI) Prediction Processing
  • Incident (image) prediction program
  • Incident (video) prediction program
• Data model
  • EfficientNet Predictive Model Training Table (public format)
  • Transaction table (public format)
  • Pattern Mining Table (public format)
  • Air Quality (AQI) Prediction Table (public format)
• Prediction model
  • Image-2-AQI^[2] method.
  • EfficientNet^[3] method.

Operating environment

We confirmed the operation in the following environment.

• OS : Ubuntu 18.04 LTS
• RAM : 4GB Recommended
• DISK SPACE : 100GB or more
• Internet connection available
• Python : 3.x（Does not work with 2.x）
  • Python library
  • psycopg2
  • configparser
  • geojson
  • requests
• PostgreSQL : 9.2 or more

When using the xDataEdge^[4]environment, the following conditions are required in addition to the above.

• Services such as Apache, Nginx, etc. that have HTTP server functions (LISTEN to HTTP/HTTPS ports) must not be running on the OS.
• Have root privileges on the OS where xData Edge is installed

Precautions

• Restrictions
  • Functional Limitations
    • This information asset is not guaranteed to operate in environments other than those shown in the operating environment.
  • Data Restrictions
    • There are restrictions on the data to be registered in the database. For details, please refer to the Data Loader API Manual^[5].
• Disclaimer
  • While every effort has been made to ensure that the information on this page is as accurate as possible, we do not guarantee its accuracy or safety.
  • We are not responsible for any damages caused by the contents of this page or by what is provided.
  • All information provided on this page is sample only and we are not responsible for any predicted results.

Reference

• [1]Please refer to the catalog page for information assets for short-term environmental quality forecasts here.
• [2]Minh-Son Dao, Kazuki Tejima, Tuan-Vinh La, Rage Uday Kiran, Koji Zettsu.: "Improving the Awareness of Sustainable Smart Cities by Analyzing Lifelog Images and IoT Air Pollution Data".
• [3]Minh-Son Dao, Dinh-Duy Pham, Manh-Phu Nguyen, Thanh-Binh Nguyen, Koji Zettsu: "MM-trafficEvent: An Interactive Incident Retrieval System for First-view Travel-log Data".
• [4]xData Edge manual:https://www.xdata.nict.jp/docs/DataLoader/0.6/
• [5]Data Loader API manual:https://www.xdata.nict.jp/docs/Edge/1.0/