Data Transfer

Passing data between the module script and xData FL

Note

This document has been machine translated.

Data transfer by file

Data transfer between the module script and xData FL is done by file.

Basic directory and file structure

When xData FL executes a module script, a folder hierarchy with the following structure is created in the current directory.

. /xxxxxxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxxxxxx/
   +- configuration.json
   +- input/
   +- output/
   +- work/
   +- session/
   +- application/

The top directory is named with a UUID, which is different each time the script is executed. The directories under the top directory are as follows.

directory/file name	summary
input	The directory that contains the files passed from xData FL to the module scripts. xData FL digs a directory and places files in this directory according to the script input definitions in definitions.json before the module script execution starts. The directories and files in this directory are destroyed after the script is finished.
output	The directory that contains the files returned from the module script to xData FL. The module script must dig the directory and place the output files in this directory according to the output definition of the script in definitions.json. After the script finishes, xData FL performs persistence and other operations on the files in this directory and then destroys them.
work	The working directory for the module script. xData FL only creates directories and does not create or read the files in them. Directories and files in this directory are destroyed after the script finishes. `
session	The contents of this directory are maintained for module API calls made from information assets to xData FL in the same session. xData FL only creates directories and does not create or read the files in them. The management of directories, files, etc. in this directory should be done by the module side. When the module API is called simultaneously for xData FL, the module scripts may be executed concurrently, so the module scripts should be aware of the exclusivity control for reading and writing files. Directories and files in this directory are destroyed after the session ends.
application	The contents of this directory are maintained for the execution of the module scripts. xData FL only creates directories and does not create or read the files in them. The management of directories, files, etc. in this directory should be done by the module side. When the module API is called simultaneously for xData FL, the module script may be executed concurrently, so the module script should be aware of the exclusivity control for reading and writing files. Directories and files in this directory are destroyed when xData FL is stopped.
configuration.json	API connection information to communicate with xData Edge arbitrarily. The data is placed at runtime according to the environment settings.

Structure of directories and files to be created under input / output

The directories and files under the input directory are created by xData FL based on the input definition in the module's definitions.json. The directories and files under the output directory are created by the module script according to the output definition in definitions.json.

An example of the correspondence with the input definitions in definitions.json is shown below.

Example 1

Outputting a single data file as output

The output definition under scripts in definitions.json is as follows.

"output": {
  "output_data_1": {
    "decription": "Output Example 1",
    "path": "output_1.csv",
    "paramtype": {
      "description": "Definition of the format of the output file",
      "definition": {
        "type": "CSV",
        "structure": {
          "lineseparator": "CR LF"
          "delimiter": ",",
          columns: [
            ...
          ]
        }
      }
    }
  }
}

In the above example, one file is generated directly in the output directory. The directory structure is as follows (UUID directories are omitted).

output/
   +- output_1.csv

Example 2

Output of one file of model metadata and one file of state_dict in a directory

The definition of output under scripts in definitions.json is as follows.

"output": {
  "output_model_1": {
    "decription": "Output Example 2",
    "path": "ouptut_model",
    "paramtype": {
      "description": "Model output by module script",
      "definition": {
        "type": "model",
        "form": "directory",
        "files": [
          {
            "name": "model_data",
            "path": "model.dat.xdata_meta",
            "form": "file",
            "format": {
              "type": "json",
              "structure": ""
            }
          },
          {
            "name": "model_file",
            "path": "model.dat",
            "form": "file",
            "format": {
              "type": "state_dict",
              "structure": ""
            }
          }
        ]
      }
    }
  }
}

In the above example, one file is generated directly in the output directory. The directory structure is as follows (UUID directories are omitted).

output/
   +- ouptut_model/
        +- model.dat.xdata_meta
        +- model.dat

Example 3

To provide a set of models by creating several directories for models in a directory and placing metadata and state_dict in the directories as input

The input definitions under scripts in definitions.json are as follows.

"input": {
  "input_model_1": {
    "decription": "Output Example 2",
    "path": "intut_models",
    "paramtype": {
      "type": "model_set",
      "form": "directory",
      "paramtype": {
        "description": "Models provided for module scripts",
        "definition": {
          "type": "model",
          "form": "directory",
          "files": [
            {
              "name": "model_data",
              "path": "model.dat.xdata_meta",
              "form": "file",
              "format": {
                "type": "json",
                "structure": ""
              }
            },
            {
              "name": "model_file",
              "path": "model.dat",
              "form": "file",
              "format": {
                "type": "state_dict",
                "structure": ""
              }
            }
          ]
        }
      }
    }
  }
}

In the above example, one file is generated directly in the input directory. The directory structure is as follows. If type is set to model_set, a directory with an arbitrary name (in the case of input, xData FL creates a directory with a UUID name) is created under it for the number of models.

input/
  +- intut_models/
       +- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
       |    +- model.dat.xdata_meta
       |    +- model.dat
       +- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
       |    +- model.dat.xdata_meta
       |    +- model.dat
       :
       +- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
            +- model.dat.xdata_meta
            +- model.dat

configuration.json

When a task is executed on a worker, xData's API connection information is placed. This API connection information can be used to communicate with xData at will.

Provided version: 3.0-alpha.1-mic

{
  "default": {
    "apikey": "xxx",
    "secret": "xxx"
  }
}

Provided version: 3.0-alpha.2-mic

{
  "default": {
    "proxies": {
      "no_proxy": "localhost,nginx,webapi-endpoint"
    }
  },
  "provenance": {
    "endpoint": "http://localhost/api/v1/provenance/jsonrpc",
    "apikey": "xxx",
    "secret": "xxx",
    "proxies": {
      "no_proxy": "localhost,nginx,webapi-endpoint"
    }
  },
  "fl": {
    "endpoint": "http://localhost/api/v1/fl/jsonrpc",
    "apikey": "xxx",
    "secret": "xxx",
    "proxies": {
      "no_proxy": "localhost,nginx,webapi-endpoint"
    }
  },
  "dataproc": {
    "endpoint": "http://localhost/api/v1/dataproc/jsonrpc",
    "apikey": "xxx",
    "secret": "xxx",
    "proxies": {
      "no_proxy": "localhost,nginx,webapi-endpoint"
    }
  }
}

Format of datetime type data in CSV file to be submitted to DDC

When submitting datetime type data to DDC via CSV file, it must be submitted in ISO format (YYYYY-MM-DDThh:mm:ss+09:00) with time zone.
The following is a sample CSV file containing datetime type data that can be submitted to DDC.

id,name,email,created_at
1,石塚 洋次,WzxKRGR6J@sample.com,2022-09-15T01:01:00+09:00
2,近藤 恭之,KefD6KU@test.net,2022-09-15T01:02:00+09:00
3,玉井 真帆,cTC6Gu7a@test.com,2022-09-15T01:03:00+09:00
4,井内 賢三,mIn0c2oY@sample.net,2022-09-15T01:04:00+09:00
5,澤田 洋一郎,SCZVtNh9f@sample.net,2022-09-15T01:05:00+09:00
6,佐田 真穂,CvWolLw_G@test.com,2022-09-15T01:06:00+09:00
7,米山 立哉,ADDwGnHX@sample.co.jp,2022-09-15T01:07:00+09:00
8,尾崎 素子,HUrn81dq@test.co.jp,2022-09-15T01:08:00+09:00