Archive
Transactional Replication: Replicate On-Premise SQL Server databases to SQL Azure databases
Transactional replication to SQL Azure Database is now available as of Oct 23, 2017, This feature will help us in replicating/migrating data from On-Premise SQL Server databases to SQL Azure databases with minimal impact and downtime
In this post we will see the following
1. Setup new publisher and configure new distributor
2. Add SQL Azure Database as Subscriber
For this demo we are using
Tools/Servers Used
On-Premise SQL Server 2017 Developer Edition
SQL Azure Database
SQL Server Management Studio 2017 v 17.3
How to Setup new publisher and configure new distributor
Step 1: Please connect to the On-Premise SQL Server on which we are planning to publish the data, Right click on Replication and click on New –> Publication
Step 2: Click next to continue
Step 3: We are going to configure the publisher to act as Distributor as well, but if you prefer to use a different server, use the 2nd option
After selecting, Please click Next to continue
Step 4: By default SQL Server will create the ReplData folder under Data folder, but if you prefer to use a different folder, Please change it
Please make sure the Process account which you are planning to use and SQL Server service account has rights to Read/Write to this folder else you Replication Snapshot job will fail to generate scripts
Step 5: Please select the database from which we need to publish data, In this demo we are using database “Demo”
After selecting the database click “Next” to proceed
Step 6: Please select “Transactional Publication” and click “Next” to proceed
Step 7: Please select the required objects, you wish to replicate, we will use Customers table for this demo, after selecting the required objects, Please click “Next” to continue
Step 8: This is an optional step, if you want only filtered data to be replication, Please click on “Add…” add relevant filters
In this demo, we are going to replicate the entire table, so we will skip this and click “Next” to continue
Step 9: Please select the first option to create the initial snapshot and then use the 2nd option to setup the schedule in which the snapshot agent needs to run
Please click “Change…” to setup the schedule details
For this demo, we are setting it up to 24 hours/day for every 30 minutes
After configuring the schedule, Please click “Next” to proceed
Step 10: In this step we will be configuring the Snapshot and Log Reader Agent security settings
Click on “Security Settings…” to configure
If you want to use 2 different account for Snapshot agent and Log Reader agent, please unselect “Use the security settings from the Snapshot Agent” and use “Security Settings…” for Log Reader Agent
Please provide the Windows User account to run the Snapshot Agent
After providing the details, click “Ok” to close and then review the user name provided and click “Next” to continue
Step 11: By default “Create the publication” option will be selected, Please select “Generate a script file with steps to create the publication” if you prefer to run the script manually
If you selected the 2nd option, Please specify the path where you want the script to be created and click “Next” to continue
Step 12: Please provide the Publication Name and review the settings and click “Finish” to start the publisher/distributor configuration process
We can see the progress for each steps
After all steps are completed, we should see the Success message and click “Close” to continue
Step 13: To verify the created publication, Please refresh the Local Publications node under Replication and you should see the newly created Publication
We have successfully created the Publication now, you can verify the status of initial snapshot scripts creating by verifying the job status in SQL Server Agent – Job Activity Monitor
We can see the highlighted job as completed Successfully
How to setup SQL Azure Database as Subscriber
We will now see how to create a new SQL Azure Database subscriber
Step 1: Right click on the Replication –> Local Publications –> Publication Name and click on “New Subscriptions…”
Step 2: Click “Next” to continue
Step 3: Please review the selected Publisher and Publication and click “Next” to continue
Step 4: Please select “Run all agents at the Distributor” option and click “Next” to continue
Step 5: Click on “Add Subscriber”
Click on “Add SQL Server Subscriber”
Please provide the SQL Azure Database server name and select SQL Server Authentication and provide the user name and password and click “Connect” to continue
Step 6: Please select the Subscription Database and click “Next” to continue
Step 7: Please click “…” button to configure the Distribution Agent security
Please provide the windows user name to run the process account in Distributor Agent
For “Connect to the Subscriber”, Please provide the SQL Server Login account to connect
Please click “Ok” and then review the selected settings and click “Next” to continue
Step 8: Please specify the Agent Schedule and click “Next” to continue
Step 9: Please select “Initialize” and select “Immediately” for Initialize When, Please choose this accordingly if your snapshot agent job is completed or not
Click “Next” to continue
Step 10: By default “Create the subscription(s)” will be selected, if you prefer to create the subscription manually using script, Please select Generate Script option and click “Next” to continue
Please provide the script path and click “Next” to continue
Step 11: Please review the settings and click “Finish” to start the process
Once the process is complete, Please refresh the publication to see the Subscription
After the Replication Distribution job is run, Please connect to SQL Azure database to see the Table is created in SQL Azure database
In next post, we will see how to monitor the Replication, we will also make changes to local data and see the changes being replicated to SQL Azure
Hope you find this post helpful !!!
Share this:
How to import CSVs from Azure Blob Storage to SQL Server 2017 ?
SQL Server 2017 introduces option to import CSV files stored in Azure Blog Storage
In this article we will see how to implement that step by step
Step 1: Create Database Master Encryption Key
This step is optional, if you already have created your database master encryption key, Please skip this
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'PleaseUseStrongPasswordHere'
Step 2: Create Database Scoped Credential
CREATE DATABASE SCOPED CREDENTIAL DemoAzureCredential WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = 'INSERT YOUR SHARED ACCESS SIGNATURE HERE';
Please note when you are copying the Share Access Signature from Azure Portal, it will copy with ? as prefix, and you will need to remove it
Step 3: Create External Data Source
Please make sure to replace sqlxpertise with your own Blob account name and democsvs with your container name
CREATE EXTERNAL DATA SOURCE DemoAzureBlobCSVs
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://sqlxpertise.blob.core.windows.net/democsvs',
CREDENTIAL = DemoAzureCredential
);
Step 4: Test External Data source connection using OPENROWSET command
SELECT * FROM OPENROWSET( BULK 'DemoProducts.txt', DATA_SOURCE = 'DemoAzureBlobCSVs', SINGLE_CLOB) AS DataFile;
Sample output – This is based on the input file I used and it will vary based on your input file uploaded to Azure Blog Storage
Step 5: Import Data to table
CREATE TABLE tmpProducts
(
[productid] [int] IDENTITY(1,1) NOT NULL,
[productname] [nvarchar](40) NOT NULL,
[supplierid] [int] NOT NULL,
[categoryid] [int] NOT NULL,
[unitprice] [money] NOT NULL,
[discontinued] [bit] NOT NULL,
)
BULK INSERT tmpProducts
FROM 'DemoProducts.txt'
WITH
(
DATA_SOURCE = 'DemoAzureBlobCSVs',
FORMAT = 'CSV'
);
Output
Once data is successfully imported, let us review the data imported
SELECT productid,
productname,
supplierid,
categoryid,
unitprice,
discontinued FROM dbo.tmpProducts
Output
Note: BULK INSERT is working only for permanent tables and not temporary tables, You will get below error message if you try importing temporary tables
CREATE TABLE #tmpProducts ( [productid] [int] IDENTITY(1,1) NOT NULL, [productname] [nvarchar](40) NOT NULL, [supplierid] [int] NOT NULL, [categoryid] [int] NOT NULL, [unitprice] [money] NOT NULL, [discontinued] [bit] NOT NULL, ) BULK INSERT #tmpProducts FROM 'DemoProducts.txt' WITH (DATA_SOURCE = 'DemoAzureBlobCSVs', FORMAT = 'CSV');
Error Message
Msg 12703, Level 16, State 1, Line 57
Referenced external data source “DemoAzureBlobCSVs” not found.
Yes, the message is misleading, but I am sure, Microsoft SQL Server team will realize and change this.
Hope you find this post helpful !!!
Share this:
Temporal Tables – Part 5- Simple Guide to implement History Tables using SQL Server 2016/2017– Storing History data in Azure using Stretch Database
In previous posts, we saw
Part 1 – How to create Temporal Tables
Part 2 – Accessing History data from Temporal Tables
Part 3 – Modifying Temporal Tables, Disabling System versioning and why we need to disable ?
Part 4 – Managing Temporal Tables and History Data
In this post we will see how to store the history data in Azure using Stretch database
Step 1: Right click on Database name and click on Tasks –> Stretch –> Enable
Step 2: Select the tables which you would like to enable Stretch
Step 3: Click on “Entire Table” hyperlink in Migrate column to customize the criteria for Stretch, by default the Entire table will be stored in SQL Azure
Please specify the name of the Stretch Criteria, Please select the Column based on which you want to apply the filter criteria , select the Condition and then specify the value
After specify the criteria, click on “Check” to validate, once you “Success”, Please click “Done” to continue
Step 4: Please review the selected tables and click “Next” to continue
Step 5: Please sign in to your Azure account, if you are not already signed in, Please select the Subscription, Azure region
We can create a new SQL Azure server or use an existing server, in this demo I am using an existing server
After providing details, click “Next” to continue
Step 6: Please provide the Public IP Address range for which you need to enable access, If you have only one Public IP Address, Please specify that for both From and To and then click Next to proceed
FYI, If you created a new server, by default this step will enable the Firewall for Azure services
Step 7: Review the settings selected and also review the estimated Pricing, this will change based on the number of tables and size of data you are planning to store in SQL Azure
Please click “Finish” to start the Stretch enabling process
We can review the Stretch deployment process in below screenshot, based on your selecting, the steps shown below can change
Once all the steps are completed, we can see status as “Passed”, if you want to see the verbose details on what happened in each step, we can review the Logs, To access the logs, Please click on Hyperlink, This will be very helpful when there is any error occurred
How to monitor Stretch databases ?
We can now see the Icon for the database has changed
To monitor the progress of Stretch, Please right click on Database Name –> Tasks –> Stretch –> Monitor
In Monitor window, we can see the Source server, Target Azure server, Database and list of tables being stretched and number of rows uploaded
We can also connect to the SQL Azure server and access the history data as shown below
How to disable Stretch for a particular table / database ?
To disable Stretch for a table, right click on table name –> Stretch –> Disable
You have 2 options
1. Bring data back from Azure – This will download all the data from SQL Azure and update your local server
2. Leave data in Azure – This will leave the data as is in SQL Azure server and new data will be stored locally
Please choose the option based on your requirement
In this demo, we are using “Bring data back from Azure” option, Please click “Yes” to confirm the disable process
Depending on the size of the data, the process will run and you will see the below message
After disabling Stretch on all the tables, we can disable the Stretch at the database level by right clicking on Database name –> Tasks –> Stretch –> Disable
FYI, If you try disabling the Stretch in your database, before disabling Stretch in the tables, you will see this below validation message
Once the Stretch disable process is completed, we will see this below completed message
FYI, You can use the Stretch option not just for Temporal History tables, you can use this for any other tables where you want to store historical data or least accessed data remotely for reference or compliance purposes
Hope you find this post helpful !!!
Share this:
Simple Guide to provision SQL Server 2017 on Linux in Azure Virtual Machine
Hi Everyone, In this article, we will see how to provision SQL Server 2017 on Linux in Azure Virtual Machine
Step 1: Please login to azure portal and go to Virtual Machine blade
Step 2: Please search for SQL Server 2017
Step 3: Please select SQL Server 2017 Developer on Ubuntu Server, I have used Ubuntu linux for this demo but you can use Red Hat or Suse Linux as well
Step 4: Please review the legal terms, select deployment model, I prefer to use Resource Manager, so that management and cleanup is easier, Please also copy the highlighted command which will be helpful for configuring the sa password for SQL Server later
Step 5: Please provide the name of the VM, select Disk type, provide user name and select Authentication Type, in this case I am using Password
Please select the Azure Subscription
For Resource group, I am creating a new one named “SQLLinux” and using Azure data center location as “East US”, please change this according to your region
Step 6: In this step, we need to select the Size of our Linux VM, I am using B2S Standard, which has 2 vCPUs and 4 GB RAM, which is good for our testing, but you can choose the size based on your requirement, after selecting the size, Please click “Select” to proceed
Step 7: In this step, we will review some of the important settings
Virtual Network – For this demo, I am creating a new Virtual Network, but if you already have other VMs running in Azure and you need this SQL Server VM to be accessible for them, Please use the existing Virtual Network
Public IP Address – I have enabled Public IP Address for this demo, but this is optional, Please don’t enable this, if you are planning to use this server only within Azure, so that it will reduce the exposure
Network Security Group (Firewall) – We will see in next step, How to configure firewall to allow external access
Auto Shutdown – This is an useful optional setting for demo or test servers, so that you can configure it to shutdown automatically based on a timing, this will reduce billing of your VM
Time Zone – Please select the time zone based on your region
Step 8: Additional Settings Please enable Monitoring for both Boot diagnostics and Guest OS diagnostics, This will help you see the Boot log for any error or messages, after making required Settings changes, Click “Ok” to proceed
Step 9: To allow external access to SQL Server, we need to change Network Firewall setting by adding Inbound and Outbound security rules for MSSQL Service
Note: You can do this change after creating the Virtual machine
Step 10: Review the selection options and click “Purchase” to buy and create the virtual machine
Step 11: You can review the Virtual Machine deployment progress in the screen
If you closed it, you can see from the Top right corner Notifications area
You can also see the status under Virtual Machines blade as well
Once Virtual machine is provisioned and ready, you should see the status as “Running”, see below screenshot
Step 12: Please click on the Virtual Machine to see the settings and also to monitor the CPU/Network/Disk usage
Step 13: Configuring SQL Server in Azure Linux VM
Please connect to Azure Linux VM using any preferred SSH client, In this demo, I am using Bash Shell on Windows, since its built in Windows 10 now
Command to connect, Please type “Yes” to accept the authenticity of the host
Step 14: After logging in to VM, Please run the below command to check the status of SQL Server
sudo systemctl status mssql-server
Since we haven’t configured SA password, SQL Server is not running, so you seeing the status as “Inactive”
Step 15: Please run below command to configure SA password
sudo /opt/mssql/bin/mssql-conf setup
Please select your preferred edition of SQL Server, in this demo, we will use Developer edition, in next prompt, Please specify the SA password and press enter
Once password is setup, SQL Server 2017 will automatically run, Please run the status command again to review the SQL Server status
sudo systemctl status mssql-server
Step 16: Please run “SELECT @@VERSION” using SQLCMD utility to connect to the server to verify we are able to connect and see SQL Server version
Step 17: Now we can use SQL Server Management Studio to connect using the Public IP Address and SA password provisioned
Once connected, Please create your required databases and database objects and use as needed
SQL Server 2017 Developer Edition is up and running Azure Linux VM now in under 5 minutes
I believe you will find this post useful !!!
Share this:
SQL Azure Import/Export Service has been released to Production
SQL Azure Import/Export Service has been released to Production as of today!!!
SQL Azure Import/Export Service is available at free of charge for SQL Azure customers.
For more information visit the following blog post
Share this:
Manage SQL Azure server using PowerShell
Windows Azure Platform Powershell Cmdlets has been released and available for download at http://wappowershell.codeplex.com/releases/view/73463
It supports creating SQL Azure server, removing an existing SQL Azure server, managing the firewall and resetting the Administrator password.
For more detailed information, please visit
http://wappowershell.codeplex.com/documentation
Share this:
How to create Co-Administrator for SQL Azure Server ?
SQL Azure Co-Administrator option has been released in July 2011 Service Release
In this article, we will see How to add a Co-Administrator for your subscription
After logging in to Windows Azure Management Portal, click on “Database” tab
After navigating to Database section, click on “User Management” in Tool bar
In User Management screen, You can see the currently available Service Administrators and Co-Administrator for all available subscriptions
Click on “Add New Co-Admin”
In Add New Co-Administrator screen, specify the new users’ Windows Live ID and then select the subscription he can administer and then click “Ok”
Now we have successfully created a new Co-Administrator, Now he can login and create or drop servers, create or drop databases and new co-administrators as well
To remove a subscription or remove a Co-Administrator, click on Co-Administrator name and then click on “Manage Co-Admin”
If you have multiple subscriptions, you can remove the subscription which the co-admin should not manage, if you remove all subscriptions, the co-admin account will be removed automatically.
Now let us login using Co-Administrator login and see how the users are displayed
Since the Co-Administrator already has a subscription, he is displayed as Service Administrator for one of the account and Co-Administrator for other account.
Hope you all find this information helpful !!!
