Jobs & workflows

Jobs

Supported jobs are:

• Apache Spark
• Bash
• Ray
• Python
• PyTorch
• Dask
• Tensorflow (Distributed and Federated)
• Scikit-learn
• Spring (Batch and Cloud Data Flow)
• Apache Hadoop
• Apache Pulsar Function
• Distributed XGBoost
• SAS (soon)
• R (soon)

Resources

Specify resources for memory and cpu inside the job options:

"resources": {
    "requests": {
        "cpu": "3",
        "memory": "8G"
    },
    "limits": {
        "cpu": "3",
        "memory": "16G"
    }
}

Special characters inside parameters

Parameters are directly used inside the jobs. You have to escape special characters. Example:

"parameters": ["--path", "\"spècia!;ch@\""],

Use \" inside the JSON for this. If you want to escape from a CLI like PowerShell, use single quote for string with special characters

Schedule

Run once by letting an empty schedule or force a schedule of type once

"schedule": {
    "type": "once"
}

For cron job, specify a cron expression, for instance:

• Every day at 1h: 0 1 * * *
• Every 3 months at 0h: 0 0 * Jan,Apr,Jul,Oct *

inside a schedule type cron like this:

"schedule": {
    "type": "cron",
    "timezone": "Europe/Paris",
    "cron_expression": "0 1 * * *"
}