Using Graal Platform with Azure Data Factory
Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data.
You can for instance run a job and then wait for the completion via three activity:
- WebActivity: The Web Activity can be used to call a custom REST endpoint
- Wait: The Wait activity causes pipeline execution to pause for a specified period
- Until: The Until activity is a compound activity. It executes its child activities in a loop, until one of the below conditions is met:
- The condition it's associated with, evaluates to true
- Its timeout period elapses
Here is an example of a waiting loop for the completion of a un previously submitted:
{
"name": "DoUntilPipeline",
"properties": {
"activities": [
{
"type": "Until",
"typeProperties": {
"expression": {
"value": "@equals('Failed', coalesce(body('Get my job status')?.status, actions('Get my job status')?.status, 'null'))",
"type": "Expression"
},
"timeout": "00:05:00",
"activities": [
{
"name": "Get my job status",
"type": "WebActivity",
"typeProperties": {
"httpRequestTimeout": "00:00:05",
"method": "get",
"url": "https://api.graal.systems/api/v1/job/XXXXXXXX/runs/XXXX",
"headers": {
"Accept": "application/vnd.graal.systems.v1.jobrun+json;charset=UTF-8",
"X-Tenant": "XXXX"
},
"authentication":{
"type":"Basic",
"username":"XXXXXX",
"password":"XXXX API KEY"
}
},
"dependsOn": [
{
"activity": "WaitActivity",
"dependencyConditions": [ "Succeeded" ]
}
]
},
{
"type": "Wait",
"typeProperties": {
"waitTimeInSeconds": 1
},
"name": "WaitActivity"
}
]
},
"name": "MyUntilActivity"
}
]
}
}