Submitting Event Data

Note

This document has been machine translated.

This section explains how to submit event data to EvWH in the xData Edge environment. The code and data used in this explanation can be downloaded from here.

1. Acquisition of data to be submitted

This time, we will use the data for Kanagawa Prefecture in October 2019 from Historical Weather Information published by the Japan Meteorological Agency. The downloaded data is 2019-10-kanagawa.csv.

2. Check the data

In order to register the downloaded data to EvWH as event data, time and space information is required.
Time is listed in the downloaded CSV, but since there is no spatial information, use the information from the List of Regional Weather Stations [PDF format] on the Japan Meteorological Agency's website, This information will be created later.
The latitude and longitude in the PDF file are in degree-minute format, but the event data uses decimal format, so the values will be converted when the data is submitted.

3. Creating a table to populate with data

Now that the data to be populated is ready, create a table in loaderwebapi.register_loader of the data loader WebAPI to create a table to populate with data.
The table to populate with event data must have columns start_datetime, end_datetime, and location. In addition to this, add a rainfall column for rainfall and create a table with the data name rain_jma in the schema event.
An example implementation of these is register_loader.py.

After running register_loader.py in a VM with Edge installed, login to EwWH(DB) with the following and confirm that the table event.rain_jma_tbl_0 is created in the database evwh. The connection destination (evwhdb01-67466c9dc5-fcpl2) depends on your environment, so run kubectl get pods and specify the pod that starts with evwhdb01. Also note that the table name (rain_jma_tbl_0) and data name (rain_jma) are different.

kubectl exec -it evwhdb01-67466c9dc5-fcpl2 -- psql -d evwh -U evwh_admin

4. Retrieve the template for data registration

Next, in order to register data according to the column definitions defined in the previous section when registering data, loaderwebapi.init_record of the data loader WebAPI to get the data format.
An example of this implementation is init_record.py. If you know the format of the data, you can skip this step.

If you run init_record.py on a VM with Edge installed, you will get the following result, and you can use that format to submit data in the next section.

{
    “id": 1,.
    “jsonrpc": ‘2.0’,.
    “result": {
        “features": [
            {
                “geometry": null, }
                “properties": {
                    “end_datetime": null, { ‘end_datetime’: null, ‘end_datetime’: null, ‘end_datetime’: null
                    “rainfall": null, ‘start_datetime’: null
                    “start_datetime": null
                }, }
                “type": ‘Feature’.
            }
        ],.
        “type": ‘FeatureCollection’.
    }
}

5. Data submission

Using the format of the data obtained in the previous section, loaderwebapi.add_record of the data loader WebAPI, execute the code add_record.py on the Edge VM to populate the data.
The following is an explanation of important points based on the example implementation of add_record.py.

Reading data files

The CSV downloaded in step 1 is implemented to be read as a DataFrame in pandas, the character encoding of the CSV is SHIFT-JIS, the header has three lines, and the first column with the time information is set as an index.

df = pd.read_csv('2019-10-kanagawa.csv', encoding='shift_jis', header=[1,2,3], index_col=0)

Correction of input data

The data downloaded in 1 does not include location information, so add the information retrieved in 2.

locations = [['三浦', 35, 10.7, 139, 37.8],
             ['辻堂', 35, 19.2, 139, 27],
             ['海老名', 35, 26, 139, 23.2],
             ['横浜', 35, 26.3, 139, 39.1],
             ['日吉', 35, 33.1, 139, 39],
             ['相模原中央', 35, 34.3, 139, 22.2],
             ['相模湖', 35, 36.8, 139, 11.6],
             ['丹沢湖', 35, 24.6, 139, 2.6],
             ['平塚', 35, 20.7, 139, 18.3],
             ['小田原', 35, 16.6, 139, 9.3],
             ['箱根', 35, 13.3, 139, 2.5]]

Also converts latitude and longitude from degrees to decimal degrees.

    lat = location[1]+location[2]/60
    lon = location[3]+location[4]/60

Creation of input data

Obtain time and precipitation(mm) for each location from the CSV imported as a DataFrame, and create the body part of the API request using the model obtained in step 4.
The location information is a geometry of type Point using the information in the previous section. Also, end_datetime should be one hour after start_datetime since this data is in one hour increments.

    for index, row in df[(location[0],'降水量(mm)')].iterrows():
        dt_str = index
        s_dt = dt.strptime(index, '%Y/%m/%d %H:%M:%S')
        e_dt = s_dt +  timedelta(hours=1)

        feature = {
            "type":"Feature", 
            "geometry": {"type": "Point", "coordinates": [lon, lat]},
            "properties":{
                "start_datetime": s_dt.strftime('%Y/%m/%d %H:%M:%S+09'), 
                "end_datetime": e_dt.strftime('%Y/%m/%d %H:%M:%S+09'), 
                "rainfall": row[0] 
            }
        }
        features.append(feature)

    body_dict = {
        "id": 1,
        "jsonrpc": "2.0",
        "method": "loaderwebapi.add_record",
        "params": {
            "data_name": "rain_jma",
            "options": {"if_exists": "override"},
            "record_set": features
        }
    }

6. Confirmation of input data

Log in to EwWH(DB) as described in step 3, and confirm that data is registered in table event.rain_jma_tbl_0.