Problem: During development of SSIS packages for data transferring between Azure Storage, you want it to be fully tested on your local first before any data transfers to your Azure account. You also want to avoid unnecessary charges on your Azure account for testing. How to achieve this?
Solution: Using Azure storage emulator
We know that Azure storage is charged based on volume of data stored every month, types of operations performed, and amount of data transfers involved base on Azure Storage pricing.
In order to avoid unnecessary charges during development and testing, we can leverage Azure storage emulator to test and debug Azure cloud services locally before real data transfer occurs between on-premise and Azure account.
First, you can download and install Azure storage emulator here. During installation, you need to take note of installation paths of Azure emulators. Here mine is installed by default at:
First, you can download and install Azure storage emulator here. During installation, you need to take note of installation paths of Azure emulators. Here mine is installed by default at:
C:\Program Files (x86)\Microsoft SDKs\Azure\Storage Emulator
After installation, start the storage emulator by searching "Azure Emulator" after clicking windows Start button.
If it is executed for the first time, it will do initialization first by creating a database in LocalDB and granting database access for current user.
You can also check the database created by Storage Emulator as shown below:
How to set connection to Azure storage emulator?
Here we use upload files to Azure Blob storage by SSIS as an example. (As for how to set up Azure environments for SSIS, please refer to the steps in "How to Upload Data Files to Azure Blob Storage". )
At connection manager pane in SSIS, choose "New Connection", then choose "AzureStorage" as shown below.
At connection manager pane in SSIS, choose "New Connection", then choose "AzureStorage" as shown below.
Next, choose "Use local developer account" instead of "Use Azure account". Then click "Test Connection" button to ensure connection with storage emulator is successful. Ensure that you have storage emulator running. You can check the status of emulator by running the following command on command console:
C:\Program Files (x86)\Microsoft SDKs\Azure\Storage Emulator>AzureStorageEmulator.exe status
C:\Program Files (x86)\Microsoft SDKs\Azure\Storage Emulator>AzureStorageEmulator.exe status
You can rename the connection to the name you want. Here I rename it to "azure-emulator" for future use.
To use it with Azure related task, the main change is the connection to be the Azure emulator one you just created. Below shows the configuration for Azure Blob Upload Task. Now you are good to go to debug and test your package on your local!
To use it with Azure related task, the main change is the connection to be the Azure emulator one you just created. Below shows the configuration for Azure Blob Upload Task. Now you are good to go to debug and test your package on your local!
Help Command in Storage Emulator
You can use help command to quickly locate commands for operations on storage emulator.
Note that storage emulator is an emulated environment running on a local SQL instance, there are Differences between the storage emulator and Azure Storage, in aspects of Blob Storage and table storage.
