Development Flow

Development Flow

Note

This document has been machine translated.

This section explains the module development workflow for “modularising the PyTorch Basic MNIST Example.”
MNIST is a dataset of handwritten digits 0–9. The Basic MNIST Example (see the link below) trains a model to predict the digit shown in an image.

• https://github.com/pytorch/examples/tree/main/mnist

Use this procedure as a reference when developing the required functionality as a module.
All sample files, including this document, can be downloaded from .

Specification Review and External Design

1. Decide Module Name

Choose a unique name that does not conflict with existing modules.
Here, the module will be named mnist_module.

2. Define Method Composition

Specify the methods that the module will expose.
The following two methods are required:

• train – Model training
• predict – Prediction using a trained model

3. Determine Method Input/Output

Define the inputs and outputs for each method.

Method	Input	Output
train	Labeled image data (training set)	Trained CNN model
predict	Labeled image data (prediction set) Trained CNN model	Prediction result data

4. Specify Data Schemas

Inputs and outputs are stored in EvWH tables or the model store.

Data Type	Schema
Labeled image data	image_id integer label integer image text (base64‑encoded binary)
Prediction result data	image_id integer label integer
Trained model (CNN)	No schema required (state_dict format)

Development Steps

1. Create Directory Structure

Create the module directory under the repository root.

mnist_module/
├─ docker-compose.yml
├─ Dockerfile_train
├─ Dockerfile_predict
├─ Dockerfile_get-definition
└─ src/
   ├─ train.sh
   ├─ predict.sh
   ├─ definitions/
   │  └─ definition.json
   └─ mnist/
      ├─ train.py
      ├─ predict.py
      ├─ requirements.txt
      └─ …

Files are described later; they need not be created at this moment.

2. Generate Skeleton Files According to Method Composition

Create minimal, no‑op train and predict methods that can be invoked via the Dataproc API.

Minimal Setup

• docker‑compose.yml – enables Docker Compose
• Dockerfile_* – builds containers and sets entrypoints (train.sh, predict.sh)
• definition.json – defines the module for Dataproc API

docker‑compose.yml (complete)

services:
  train:
    build:
      context: .
      dockerfile: Dockerfile_train
    entrypoint: ["/work/src/train.sh"]

  predict:
    build:
      context: .
      dockerfile: Dockerfile_predict
    entrypoint: ["/work/src/predict.sh"]

  get-definition:
    build:
      context: .
      dockerfile: Dockerfile_get-definition
    entrypoint: ["cat", "/work/src/definitions/definition.json"]

The get-definition service is mandatory for all Dataproc‑executable modules.

Dockerfile_train (incomplete)

FROM python:3

WORKDIR /work
COPY ./src /work/src

RUN chmod +x /work/src/train.sh

Dockerfile_predict (incomplete)

FROM python:3

WORKDIR /work
COPY ./src /work/src

RUN chmod +x /work/src/predict.sh

Dockerfile_get-definition (complete)

FROM alpine:latest

WORKDIR /work
COPY ./src /work/src

train.sh, predict.sh (incomplete)

#!/bin/sh

definition.json (incomplete, no I/O defined)

{
  "name": "module.mnist_module",
  "description": "MNIST module",
  "supports": "",
  "definitions": {},
  "scripts": {
    "train": {
      "description": "Train",
      "path": "train",
      "type": "training",
      "input": {},
      "output": {}
    },
    "predict": {
      "description": "Predict",
      "path": "predict",
      "type": "prediction",
      "input": {},
      "output": {}
    }
  }
}

Verify Operation

Run a build and, if successful, execute the sample client (mnist_module.zip/samples/sample_client_1.py).
Completion without errors indicates that the skeleton is ready.

3. Prepare Input/Output Definitions

The previous section omitted I/O definitions in definition.json.
Now, define them so that the module can retrieve input data from EvWH and store output data back to EvWH.
The train and predict scripts remain no‑ops.

Define Data Formats

Three data formats are required: labeled image dataset, trained model, and prediction result.

Format	Purpose	Storage
mnist_dataset	Labeled image data for training or prediction	CSV files in a directory (EvWH table)
mnist_model	Trained CNN model	Directory containing the state_dict file (mnist.pth) and metadata (mnist.xdata-meta.json) (EvWH model store)
mnist_predict_result	Prediction result data	Single CSV file (EvWH table)

CSV schemas include start_datetime, end_datetime, and location columns that are automatically added by the data loader; they are not used in MNIST.

Configure Scripts

Specify input and output for each script based on the defined formats.

train.input

• train_dataset (type: mnist_dataset) – Input dataset for training.
  The data will be placed at /work/runs/input/train_dataset/dataset.csv by the DDC.

train.output

• trained_model (type: mnist_model) – Output trained model.
  The model and metadata will be written to /work/runs/output/models/mnist.pth and
  /work/runs/output/models/mnist.xdata-meta.json, automatically stored in the model store.

predict.input

• predict_dataset (type: mnist_dataset) – Input dataset for prediction.
  The data will be placed at /work/runs/input/predict_dataset/dataset.csv.
  Labels are used only for accuracy calculation.
• trained_model (type: mnist_model) – Input trained model.
  The files will be placed at /work/runs/input/models/mnist.pth and
  /work/runs/input/models/mnist.xdata-meta.json.

predict.output

• predict_result (type: mnist_predict_result) – Output prediction results.
  The CSV will be written to /work/runs/output/predict_result/result.csv, then automatically stored in DDC.

Module Preparation Completed

The steps above have prepared the module with input and output.
From this point, we will verify the operation of the created module.

---

Preparatory Steps for Operation Verification: Creating Containers for Input and Output on the EvWH Side

The data defined in definition.json is stored either in a DDC or a model store.
To verify operation, the DDC and model store must be created in advance.

Execute the following curl commands to create the DDC and model store for prediction results.
In the subsequent description, we use the APIKey and Secret that are installed by default with xData Edge.
Adjust them to match your environment.

curl \
-H "Content-type:application/json" \
-H "APIKey: a4e16fab-9077-4a13-a7d5-0f1fda63cd16" \
-H "Secret: Do8feigevo8OhCi7OchoSan8zid7Fie3Aez4" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "fl.initialize_model_store",
  "params": {
    "model_store_ddc":"ddc:mnist_models"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/fl/jsonrpc

curl \
-H "Content-type:application/json" \
-H "APIKey: 0df23f0d-44cb-4085-8ac9-158e6ced3056" \
-H "Secret: 4AzbY3pvfes33aow49nhjn3aUphv9zFWctcT" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "prov.new_ddc",
  "params": {
    "ddc_label": "ddc:mnist_predict_result",
    "columns": [
      {"column_name":"image_id", "data_type":"int"},
      {"column_name":"predicted", "data_type":"int"}
    ],
    "key": "image_id"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/provenance/jsonrpc

Next, create a table and load labeled image data using the attached script
mnist_module.zip/tools/mnist_loader.py.

python mnist_loader.py ~/data_dir --kind train --end 6000
python mnist_loader.py ~/data_dir --kind test --end 1000

Finally, associate the DDC with the target tables.

curl \
-H "Content-type:application/json" \
-H "APIKey: 0df23f0d-44cb-4085-8ac9-158e6ced3056" \
-H "Secret: 4AzbY3pvfes33aow49nhjn3aUphv9zFWctcT" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "prov.set_ddc",
  "params": {
    "ddc_label": "ddc:mnist_train",
    "source": "event.mnist_train_tbl_0",
    "ddc_type":"link"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/provenance/jsonrpc

curl \
-H "Content-type:application/json" \
-H "APIKey: 0df23f0d-44cb-4085-8ac9-158e6ced3056" \
-H "Secret: 4AzbY3pvfes33aow49nhjn3aUphv9zFWctcT" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "prov.set_ddc",
  "params": {
    "ddc_label": "ddc:mnist_test",
    "source": "event.mnist_test_tbl_0",
    "ddc_type":"link"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/provenance/jsonrpc

---

Preparatory Steps for Operation Verification: Creating Dummy Output Data

At this stage, neither training nor prediction logic has been implemented.
Therefore, we create dummy output data for verification purposes.

train.sh (incomplete)

#!/bin/sh

mkdir "/work/runs/output/models"

MODEL_FILE="/work/runs/output/models/mnist.pth"
MODEL_META_FILE="/work/runs/output/models/mnist.xdata-meta.json"

echo 'dummy_model_data' > "$MODEL_FILE"
echo '{"model_kind": "mnist", "model_state": 10, "round": 1}' > "$MODEL_META_FILE"

predict.sh (incomplete)

#!/bin/sh

mkdir "/work/runs/output/predict_result"

RESULT_FILE="/work/runs/output/predict_result/result.csv"

echo 'image_id,label\n1,1' > "$RESULT_FILE"

---

Operation Verification

The preparation is complete.
Build the container and run the sample client.

docker compose build

If no build errors occur, execute mnist_module.zip/samples/sample_client_2.py.
After successful completion of the sample client, run:

curl \
-H "Content-type:application/json" \
-H "APIKey: a4e16fab-9077-4a13-a7d5-0f1fda63cd16" \
-H "Secret: Do8feigevo8OhCi7OchoSan8zid7Fie3Aez4" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "fl.get_models",
  "params": {
    "model_store_ddc":"ddc:mnist_models"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/fl/jsonrpc

curl \
-H "Content-type:application/json" \
-H "APIKey: 0df23f0d-44cb-4085-8ac9-158e6ced3056" \
-H "Secret: 4AzbY3pvfes33aow49nhjn3aUphv9zFWctcT" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "prov.get_ddc_records",
  "params": {
    "ddc_label": "ddc:mnist_predict_result"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/provenance/jsonrpc

You should observe that:

• A model has been added to the model store (output of training).
• Prediction records have been inserted into the result table (output of prediction).

When the module runs, it creates an execution folder (hash‑named) under the directory specified by WORKER_DATA_DIR (e.g., ~/work/runs/).
Two folders—one for training and one for prediction—should exist.
Verify that the input and output directories within each are correctly generated.

The following paths will be used for the next verification step:

• [training execution folder]/input/train_dataset/dataset.csv
• [prediction execution folder]/input/predict_dataset/dataset.csv

---

4. Developing Core Processing Logic

From this point, implement training and prediction for the MNIST dataset.
We base our implementation on the publicly available Basic MNIST Example (see link) and adapt input and output formats for the module.

• https://github.com/pytorch/examples/tree/main/mnist

Implementation of Training and Prediction (train.py, predict.py)

Create the following four files in src/mnist/:

• train.py
• predict.py
• common.py
• requirements.txt

Refer to the attached file mnist_module.zip/src/mnist/ for the final code.
Key points are described below.

Training Logic (train.py)

• Accept a dataset argument that specifies the CSV file path of labeled training images.
• The original implementation downloads data at runtime; modify it to read from the provided CSV file. Decode image data from base64 during loading.
• Use a --model argument to specify the output file path for the trained model.

Prediction Logic (predict.py)

• Accept a dataset argument that specifies the CSV file path of labeled prediction images.
• The original implementation downloads data at runtime; modify it to read from the provided CSV file. Decode image data from base64 during loading.
• Use a --model argument to specify the input file path of the trained model.
• Use a dataset argument to specify the output CSV file path for prediction results.

Creating Entry Points (train.sh, predict.sh)

Generate shell scripts to invoke train.py and predict.py.
Pass concrete file paths as arguments.

train.sh (complete)

#!/bin/sh

mkdir "/work/runs/output/models"

SCRIPT_FILE="/work/src/mnist/train.py"
MODEL_FILE="/work/runs/output/models/mnist.pth"
MODEL_META_FILE="/work/runs/output/models/mnist.xdata-meta.json"
DATASET_FILE="/work/runs/input/train_dataset/dataset.csv"

python "$SCRIPT_FILE" --model "$MODEL_FILE" --epoch 2 "$DATASET_FILE"

echo '{"model_kind": "mnist", "model_state": 10, "round": 1}' > "$MODEL_META_FILE"

predict.sh (complete)

#!/bin/sh

mkdir "/work/runs/output/predict_result"

SCRIPT_FILE="/work/src/mnist/predict.py"
MODEL_FILE="/work/runs/input/models/mnist.pth"
DATASET_FILE="/work/runs/input/predict_dataset/dataset.csv"
RESULT_FILE="/work/runs/output/predict_result/result.csv"

python "$SCRIPT_FILE" --model "$MODEL_FILE" "$DATASET_FILE" "$RESULT_FILE"

Local Operation Verification

Verify the training and prediction logic locally.

1. Install required packages:

pip install -r requirements.txt

(Use a virtual environment such as venv if desired.)

1. Training Verification
   Place the labeled training CSV file at the path specified by DATASET_FILE in train.sh.
   This can be the file generated earlier at [training execution folder]/input/train_dataset/dataset.csv.
   Run:

train.sh

If a model file appears at the location specified by MODEL_FILE, training succeeded.

1. Prediction Verification
   Place the labeled prediction CSV file at the path specified by DATASET_FILE in predict.sh.
   Use [prediction execution folder]/input/predict_dataset/dataset.csv from the earlier step.
   Move the trained model file to the path specified by MODEL_FILE in predict.sh.
   Run:

predict.sh

If prediction results appear at the location specified by RESULT_FILE, prediction succeeded.

---

5. Containerizing Core Processing

Next, package the training and prediction logic into Docker containers.

Add necessary package installation commands to Dockerfile_train and Dockerfile_predict.

Dockerfile for Training (Dockerfile_train, complete)

FROM python:3

COPY ./src/mnist/requirements.txt /tmp/

RUN pip install --upgrade pip
RUN pip install --upgrade setuptools
RUN pip install --no-cache-dir -r /tmp/requirements.txt

RUN rm -f /tmp/requirements.txt

WORKDIR /work
COPY ./src /work/src

RUN chmod +x /work/src/train.sh

Dockerfile for Prediction (Dockerfile_predict, complete)

FROM python:3

COPY ./src/mnist/requirements.txt /tmp/

RUN pip install --upgrade pip
RUN pip install --upgrade setuptools
RUN pip install --no-cache-dir -r /tmp/requirements.txt

RUN rm -f /tmp/requirements.txt

WORKDIR /work
COPY ./src /work/src

RUN chmod +x /work/src/predict.sh

Supplementary Note

The requirements.txt file is copied first and packages are installed separately.
This approach allows the build process to reuse cached layers when modifications are made only in the src directory, thereby skipping redundant package installations and reducing build time.

Next, run

docker compose build

to build the container. If no build errors occur, the module is ready for execution.

6. Verifying Module Operation via the Dataproc API

Finally, execute the implemented functionality as a module through the Dataproc API.
All prerequisites are already prepared from earlier steps; run the sample client (mnist_module.zip/samples/sample_client_2.py).
After successful completion, execute the following commands:

curl \
-H "Content-type:application/json" \
-H "APIKey: a4e16fab-9077-4a13-a7d5-0f1fda63cd16" \
-H "Secret: Do8feigevo8OhCi7OchoSan8zid7Fie3Aez4" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "fl.get_models",
  "params": {
    "model_store_ddc":"ddc:mnist_models"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/fl/jsonrpc

curl \
-H "Content-type:application/json" \
-H "APIKey: 0df23f0d-44cb-4085-8ac9-158e6ced3056" \
-H "Secret: 4AzbY3pvfes33aow49nhjn3aUphv9zFWctcT" \
-X POST -d \
'
{
  "jsonrpc": "2.0",
  "method": "prov.get_ddc_records",
  "params": {
    "ddc_label": "ddc:mnist_predict_result"
  },
  "id": "mnist_module"
}
' \
http://localhost/api/v1/provenance/jsonrpc

These requests confirm that:

• The model store contains the newly added model (output from train).
• A record of the prediction results has been appended (output from predict).

Unlike interim checks, these are genuine models and predictions rather than dummies.

The module development workflow based on the Basic MNIST Example is now complete.
By replacing the core implementation and configuring the input/output according to required features, you can develop additional modules.