Example of Azure Data Factory Connection

 Example of Azure Data Factory Connection                   

Scenario: 

Client A plans to send day-to-day order history through SFTP to the internal Reporting Team which will store in the Azure storage account.

Features: 

Azure Storage account, Azure Data Factory

Key Concept:

Azure Data Factory Pipelines and activities: pipeline is a logical grouping of activities that together perform a task. For example, a pipeline could contain a set of activities that ingest and clean log data, and then kick off a mapping data flow to analyze the log data. The pipeline allows you to manage the activities as a set instead of each one individually. You deploy and schedule the pipeline instead of the activities independently.

A picture containing timeline Description automatically generated

Azure Blob storage: Storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn't adhere to a particular data model or definition, such as text or binary data.


● Connecting Azure Data factory requires whitelisting/enabling all related ports on the firewall.

Workaround & Configuration

Creating Container and Brob storage

Graphical user interface, text, application Description automatically generated

Setting up Data factory 

Graphical user interface, text, application, email Description automatically generated


Graphical user interface, application, Teams Description automatically generated

Graphical user interface, application Description automatically generated

Graphical user interface, text, application, email Description automatically generated

By manually triggering the pipeline, you will be able to verify the data transaction. 


Comments

Post a Comment

Popular posts from this blog

To Be A CyberMan: Installing PfSense on a WatchGuard Firebox

Image
OX01 Preface My company recently upgraded its firewall to FortiGate Firewall.  They eliminated 3 of the licenses expired WatchGuard Firebox Firewalls. The WatchGuard FireBox models are XTM5 Series, M400, and M470. I was wondering if I can install the opensource Firewall application such as PfSense on FireBox to use as Testing Lab Environment. 0X02 Information      XTM5 Series comes with a 1GB CF card, free space is not enough for the lastest Pfsense,  need to add additional Hard Disk as PfSense system booting drive.     M400 comes with a 4 GB CF card, can use CF card as a PfSense system booting drive.     M470 comes with mSTAT SSD.  can SSD as a PfSense system booting drive. 0X03 Installing PfSense on M400 1. Preparation:           WatchGuard Firebox M400 Appliance          A USB Flash Drive          A CF Card USB reader          A co...
Read more

How I passed the CSX Fundamentals within one month

Image
  How I passed the CSX Fundamentals within one month Hi Guys, My name is Trevor Shi. I just passed the CSX within one month. I would like to share some experience maybe it will help you guys to prepare the exam. One of the funny things was, I used one month to learn it. But I spent the other one month to book the exam. I re-booked 5 times of exam as ISACA PSI online exam application has lots of issues.  I met every situation you can imagine. The first-time ISACA website down, the second-time they canceled my booked without excuse, the third-time application couldn't open, and the Fourth time was my fault, I just missed the exam. Finally, I got this certificate. We had a small group named CyberRookie, It just like an unorganized cyber knowledge self-studying group, we are trying to be more organized. So t he purpose of this article is as a guideline of the CyberRookie Project CSX certification learning method . A. The Cybersecurity Fundamentals Certific...
Read more

To Be A CyberMan: Set Up SFTP Server On Azure VM behind FortiGate Firewall

Image
1. Business request     S et up a file transfer server with public internet access.  2. security concern      The common ways are FTP, FTPS, or SFTP:     The FTPS requires applying the certificates to the FTP service,  So I deceived to go for SFTP.       SolarWinds SFTP & SCP Server is a Free SFTP server App. 3. Deployment     3.1  Set up SFTP on VM          Download the software to Server and install it                https://www.solarwinds.com/free-tools/free-sftp-server          Redirect the Root Directory           Create user to login           verify the SFTP service is running on localhost     3.2 Config Host firewall and  Azure NSG to allow SFTP service           Add Port 22 TCP into host firewall ...
Read more