ADF (Azure Data Factory) : An Introduction

In this blog post we will see an introduction of Azure Data Factory, but before moving further into ADF, there is a need to know you all about the ETL tool.

ETL stands for Extract, Transform and Load. This ETL Tool is a data pipeline used to Extract (collect) the data from different sources, Transform them into meaningful data according to business rules and Load them into the destination data store, e.g. Data warehouses, databases, etc.

What is Azure Data Factory?

• ADF is a heart of ETL in Azure.

• It is essential service in all data related activities in Azure.

• It is a cloud data integration service used to compose data storage, movement, and processing services into automated data pipelines.

• By using this ETL tool the user will define the data sets, create pipelines to transform the data and map them with various destinations.

• It is flexible and powerful Platform as a Service offering with multitude of connectors and integration capabilities.

• It is a highly scalable, increased agility, and cost-effective solution that provides code-free ETL as a service.

Components of Azure Data Factory.

• Pipelines: A pipeline is a logical grouping of activities that performs a unit of work. A single pipeline can perform different actions like ingesting data from the Storage Blob, Query the SQL Database, and more.

• Activities: Activity in a Pipeline represents a unit of work. An Activity is an action like copying a Storage Blob data to a Storage Table or transform JSON data in a Storage Blob into SQL Table records.

• Datasets: Datasets represent data structures within the data stores, which point to the data that the activities need to use as inputs or outputs.

• Triggers: Triggers are a way to execute a pipeline run. Triggers determine when a pipeline execution should start. Currently, Data Factory supports three types of triggers:

  • Schedule Trigger: A trigger that invokes a pipeline at a scheduled time.

  • Tumbling window trigger: A trigger that operates on a periodic interval.

  • Event-based trigger: A trigger that invokes a pipeline when there is an event.

• Integration Runtime: The Integration Runtime (IR) is the compute infrastructure used to provide data integration capabilities like Data Flow, Data Movement, Activity dispatch, and SSIS package execution. There are three types of Integration Runtimes available, they are.

  • Azure

  • Self-hosted

  • Azure SSIS

That's all in this post. If you liked this blog and interested in knowing more about Azure, Please Like, Follow, Share & Subscribeto www.ImJhaChandan.com.