Skip to content

Data Transfer Between Module Scripts and xData FL

Note

This document has been machine translated.

File‑Based Exchange

Data transfer between module scripts and xData FL occurs via files.

Core Directory‑File Structure

When a module script is executed, xData FL creates the following folder hierarchy in its current directory:

./xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
   +- configuration.json
   +- input/
   +- output/
   +- work/
   +- session/
   +- application/

The top‑level directory name is a UUID that changes with each script execution.
Subdirectories are defined as follows:

Directory / File Description
input Stores files provided by xData FL to the module script. Before execution, xData FL creates directories and files in this folder according to the input definitions in definitions.json. All contents are deleted after script completion.
output Stores files returned by the module script to xData FL. The script must create directories and files in this folder according to the output definitions in definitions.json. After execution, xData FL persists and then deletes the contents.
work Working directory for the module script. The script may freely use subfolders here; xData FL only creates the directory and does not manage its contents. All contents are removed after script completion.
session Maintains state for API calls performed within the same session by information assets. The content persists across different scripts of the same module when the session is unchanged. xData FL only creates the directory; all file management is handled by the module. Concurrent API calls may cause parallel script execution, so the module must handle file‑level locking. Contents are deleted after session termination.
application Maintains state for the module’s script executions, persisting across different scripts of the same module. xData FL only creates the directory; all file management is handled by the module. Concurrent API calls may cause parallel script execution, so the module must handle file‑level locking. Contents are deleted when xData FL stops.
configuration.json Optional API connection information for communicating with xData Edge. Data appropriate to the runtime environment is placed here.

Directories and Files Created Under input / output

xData FL creates directories and files under the input directory according to the module’s definitions.json input definitions.
The module script creates corresponding structures under output based on the output definitions.

Example 1 – Single Output File

When a single data file is output, the output definition in definitions.json looks like:

"output": {
  "output_data_1": {
    "decription": "Example 1 output",
    "path": "output_1.csv",
    "paramtype": {
      "description": "Definition of the output file format",
      "definition": {
        "type": "CSV",
        "structure": {
          "lineseparator": "CR LF",
          "delimiter": ",",
          "columns": [ ... ]
        }
      }
    }
  }
}

The module script creates the file directly in output/:

output/
   +- output_1.csv

Example 2 – Model Metadata and State Dictionary

For a model consisting of metadata and a state_dict file, the output definition is:

"output": {
  "output_model_1": {
    "decription": "Example 2 output",
    "path": "ouptut_model",
    "paramtype": {
      "description": "Model produced by the module script",
      "definition": {
        "type": "model",
        "form": "directory",
        "files": [
          {
            "name": "model_data",
            "path": "model.dat.xdata_meta",
            "form": "file",
            "format": { "type": "json", "structure": "" }
          },
          {
            "name": "model_file",
            "path": "model.dat",
            "form": "file",
            "format": { "type": "state_dict", "structure": "" }
          }
        ]
      }
    }
  }
}

The resulting structure is:

output/
   +- ouptut_model/
        +- model.dat.xdata_meta
        +- model.dat

Example 3 – Multiple Models in a Set

When the input consists of multiple model directories, each containing metadata and a state_dict, the input definition is:

"input": {
  "input_model_1": {
    "decription": "Example 3 input",
    "path": "intut_models",
    "paramtype": {
      "type": "model_set",
      "form": "directory",
      "paramtype": {
        "description": "Set of models provided to the module script",
        "definition": {
          "type": "model",
          "form": "directory",
          "files": [
            {
              "name": "model_data",
              "path": "model.dat.xdata_meta",
              "form": "file",
              "format": { "type": "json", "structure": "" }
            },
            {
              "name": "model_file",
              "path": "model.dat",
              "form": "file",
              "format": { "type": "state_dict", "structure": "" }
            }
          ]
        }
      }
    }
  }
}

xData FL creates a UUID‑named directory for each model under input/intut_models/:

input/
  +- intut_models/
       +- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
           +- model.dat.xdata_meta
           +- model.dat
       +- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
           +- model.dat.xdata_meta
           +- model.dat
       : 
       +- xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
           +- model.dat.xdata_meta
           +- model.dat

configuration.json Example

When a task runs on the worker, xData API connection information is populated in configuration.json. The module can use this data to communicate with xData.

Provided Version 3.0‑alpha.1‑mic

{
  "default": {
    "apikey": "xxx",
    "secret": "xxx"
  }
}

Provided Version 3.0‑alpha.2‑mic

{
  "default": {
    "proxies": { "no_proxy": "localhost,nginx,webapi-endpoint" }
  },
  "provenance": {
    "endpoint": "http://localhost/api/v1/provenance/jsonrpc",
    "apikey": "xxx",
    "secret": "xxx",
    "proxies": { "no_proxy": "localhost,nginx,webapi-endpoint" }
  },
  "fl": {
    "endpoint": "http://localhost/api/v1/fl/jsonrpc",
    "apikey": "xxx",
    "secret": "xxx",
    "proxies": { "no_proxy": "localhost,nginx,webapi-endpoint" }
  },
  "dataproc": {
    "endpoint": "http://localhost/api/v1/dataproc/jsonrpc",
    "apikey": "xxx",
    "secret": "xxx",
    "proxies": { "no_proxy": "localhost,nginx,webapi-endpoint" }
  }
}

CSV File DateTime Format for DDC Ingestion

When ingesting datetime data into DDC via a CSV file, the value must be in timezone‑aware ISO format (YYYY-MM-DDThh:mm:ss+09:00).
The following is a sample CSV file containing datetime data acceptable to DDC:

id,name,email,created_at
1,石塚 洋次,WzxKRGR6J@sample.com,2022-09-15T01:01:00+09:00
2,近藤 恭之,KefD6KU@test.net,2022-09-15T01:02:00+09:00
3,玉井 真帆,cTC6Gu7a@test.com,2022-09-15T01:03:00+09:00
4,井内 賢三,mIn0c2oY@sample.net,2022-09-15T01:04:00+09:00
5,澤田 洋一郎,SCZVtNh9f@sample.net,2022-09-15T01:05:00+09:00
6,佐田 真穂,CvWolLw_G@test.com,2022-09-15T01:06:00+09:00
7,米山 立哉,ADDwGnHX@sample.co.jp,2022-09-15T01:07:00+09:00
8,尾崎 素子,HUrn81dq@test.co.jp,2022-09-15T01:08:00+09:00