Using Graal Platform with Azure Data Factory

Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data.

You can for instance run a job and then wait for the completion via three activity:

• WebActivity: The Web Activity can be used to call a custom REST endpoint
• Wait: The Wait activity causes pipeline execution to pause for a specified period
• Until: The Until activity is a compound activity. It executes its child activities in a loop, until one of the below conditions is met:
  • The condition it's associated with, evaluates to true
  • Its timeout period elapses

Here is an example of a waiting loop for the completion of a un previously submitted:

{
    "name": "DoUntilPipeline",
    "properties": {
        "activities": [
            {
                "type": "Until",
                "typeProperties": {
                    "expression": {
                        "value": "@equals('Failed', coalesce(body('Get my job status')?.status, actions('Get my job status')?.status, 'null'))",
                        "type": "Expression"
                    },
                    "timeout": "00:05:00",
                    "activities": [
                        {
                            "name": "Get my job status",
                            "type": "WebActivity",
                            "typeProperties": {
                                "httpRequestTimeout": "00:00:05",
                                "method": "get",
                                "url": "https://api.graal.systems/api/v1/job/XXXXXXXX/runs/XXXX",
                                "headers": {
                                    "Accept": "application/vnd.graal.systems.v1.jobrun+json;charset=UTF-8",
                                    "X-Tenant": "XXXX"
                                },
                                "authentication":{
                                   "type":"Basic",
                                   "username":"XXXXXX",
                                   "password":"XXXX API KEY"
                                }
                            },
                            "dependsOn": [
                                {
                                    "activity": "WaitActivity",
                                    "dependencyConditions": [ "Succeeded" ]
                                }
                            ]
                        },
                        {
                            "type": "Wait",
                            "typeProperties": {
                                "waitTimeInSeconds": 1
                            },
                            "name": "WaitActivity"
                        }
                    ]
                },
                "name": "MyUntilActivity"
            }
        ]
    }
}