Graal Platform Documentation

Graal Platform Documentation

  • Docs
  • Help

›How-to guides

Overview

  • What is Graal Platform?
  • Why use our platform?
  • How Graal Platform works?
  • Concepts
  • Jobs & workflows
  • Security

Quickstart

  • Quickstart

Tutorials

  • Get started with Python
  • Get started with Dask
  • Get started with XGBoost
  • Get started with Apache Spark and Maven
  • Get started with Apache PySpark
  • Get started with Apache Beam and Gradle
  • Use the API
  • Using the command line tool (graalctl)
  • Using secrets
  • Migration from Databricks
  • Get started with Tensorflow
  • Get started with Pytorch
  • Get started with Mxnet
  • Setting up the Hadoop bridge
  • Get started with Apache Flink and Maven
  • Get started with Dbt
  • Get started with Pulsar
  • Get started with Apache Spark Streaming Pulsar
  • Get started with Debezium
  • Get started with the SDK

How-to guides

  • Using Graal Platform with Azure Data Factory
  • Publishing your artefacts with Azure DevOps
  • Using Graal Platform with Apache Airflow
  • Publishing your artefacts with Jenkins
  • Spark
  • Network, VPN, gateway and firewall
  • Logs
  • Pricing

Security

  • Overview
  • Comply with requirements
  • Infrastructures under Graal Systems
  • Responsibilities

Troubleshoot & debug

  • Troubleshooting
  • Common issues
  • Debug jobs

Using Graal Platform with Azure Data Factory

Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data.

You can for instance run a job and then wait for the completion via three activity:

  • WebActivity: The Web Activity can be used to call a custom REST endpoint
  • Wait: The Wait activity causes pipeline execution to pause for a specified period
  • Until: The Until activity is a compound activity. It executes its child activities in a loop, until one of the below conditions is met:
    • The condition it's associated with, evaluates to true
    • Its timeout period elapses

Here is an example of a waiting loop for the completion of a un previously submitted:

{
    "name": "DoUntilPipeline",
    "properties": {
        "activities": [
            {
                "type": "Until",
                "typeProperties": {
                    "expression": {
                        "value": "@equals('Failed', coalesce(body('Get my job status')?.status, actions('Get my job status')?.status, 'null'))",
                        "type": "Expression"
                    },
                    "timeout": "00:05:00",
                    "activities": [
                        {
                            "name": "Get my job status",
                            "type": "WebActivity",
                            "typeProperties": {
                                "httpRequestTimeout": "00:00:05",
                                "method": "get",
                                "url": "https://api.graal.systems/api/v1/job/XXXXXXXX/runs/XXXX",
                                "headers": {
                                    "Accept": "application/vnd.graal.systems.v1.jobrun+json;charset=UTF-8",
                                    "X-Tenant": "XXXX"
                                },
                                "authentication":{
                                   "type":"Basic",
                                   "username":"XXXXXX",
                                   "password":"XXXX API KEY"
                                }
                            },
                            "dependsOn": [
                                {
                                    "activity": "WaitActivity",
                                    "dependencyConditions": [ "Succeeded" ]
                                }
                            ]
                        },
                        {
                            "type": "Wait",
                            "typeProperties": {
                                "waitTimeInSeconds": 1
                            },
                            "name": "WaitActivity"
                        }
                    ]
                },
                "name": "MyUntilActivity"
            }
        ]
    }
}
← Get started with the SDKPublishing your artefacts with Azure DevOps →
Graal Platform Documentation
Overview
What is Graal Platform?
Quickstart
Apache SparkApache FlinkApache BeamPythonTensorflowDaskDistributed XGBoost
Links
HomeConsoleCopyrights
Copyright © 2023 Graal Systems