Jobs & workflows
Jobs
Supported jobs are:
- Apache Spark
- Bash
- Ray
- Python
- PyTorch
- Dask
- Tensorflow (Distributed and Federated)
- Scikit-learn
- Spring (Batch and Cloud Data Flow)
- Apache Hadoop
- Apache Pulsar Function
- Distributed XGBoost
- SAS (soon)
- R (soon)
Resources
Specify resources for memory and cpu inside the job options:
"resources": {
"requests": {
"cpu": "3",
"memory": "8G"
},
"limits": {
"cpu": "3",
"memory": "16G"
}
}
Special characters inside parameters
Parameters are directly used inside the jobs. You have to escape special characters. Example:
"parameters": ["--path", "\"spècia!;ch@\""],
Use \"
inside the JSON for this.
If you want to escape from a CLI like PowerShell, use single quote for string with special characters
Schedule
Run once by letting an empty schedule or force a schedule of type once
"schedule": {
"type": "once"
}
For cron job, specify a cron expression, for instance:
- Every day at 1h: 0 1 * * *
- Every 3 months at 0h: 0 0 * Jan,Apr,Jul,Oct *
inside a schedule type cron
like this:
"schedule": {
"type": "cron",
"timezone": "Europe/Paris",
"cron_expression": "0 1 * * *"
}