Archive

Archive for the ‘SQL Azure’ Category

Transactional Replication: Replicate On-Premise SQL Server databases to SQL Azure databases

October 27, 2017 Leave a comment

Transactional replication to SQL Azure Database is now available as of Oct 23, 2017, This feature will help us in replicating/migrating data from On-Premise SQL Server databases to SQL Azure databases with minimal impact and downtime

In this post we will see the following

1. Setup new publisher and configure new distributor

2. Add SQL Azure Database as Subscriber

For this demo we are using

Tools/Servers Used

On-Premise SQL Server 2017 Developer Edition

SQL Azure Database

SQL Server Management Studio 2017 v 17.3

How to Setup new publisher and configure new distributor

 

Step 1: Please connect to the On-Premise SQL Server on which we are planning to publish the data, Right click on Replication and click on New –> Publication

capture20171026125922682

 

Step 2: Click next to continue

capture20171025093756535

Step 3: We are going to configure the publisher to act as Distributor as well, but if you prefer to use a different server, use the 2nd option

After selecting, Please click Next to continue

capture20171025093802802

Step 4: By default SQL Server will create the ReplData folder under Data folder, but if you prefer to use a different folder, Please change it

Please make sure the Process account which you are planning to use and SQL Server service account has rights to Read/Write to this folder else you Replication Snapshot job will fail to generate scripts

capture20171025093808736

Step 5: Please select the database from which we need to publish data, In this demo we are using database “Demo”

After selecting the database click “Next” to proceed

capture20171025093817202

Step 6:  Please select “Transactional Publication” and click “Next” to proceed

capture20171025093823858

Step 7: Please select the required objects, you wish to replicate, we will use Customers table for this demo, after selecting the required objects, Please click “Next” to continue

capture20171025094712912

Step 8: This is an optional step, if you want only filtered data to be replication, Please click on “Add…” add relevant filters

In this demo, we are going to replicate the entire table, so we will skip this and click “Next” to continue

capture20171025093902119

Step 9: Please select the first option to create the initial snapshot and then use the 2nd option to setup the schedule in which the snapshot agent needs to run

Please click “Change…” to setup the schedule details

capture20171025094004761

For this demo, we are setting it up to 24 hours/day for every 30 minutes

capture20171025093958343

After configuring the schedule, Please click “Next” to proceed

Step 10: In this step we will be configuring the Snapshot and Log Reader Agent security settings

Click on “Security Settings…” to configure

If you want to use 2 different account for Snapshot agent and Log Reader agent, please unselect “Use the security settings from the Snapshot Agent” and use “Security Settings…” for Log Reader Agent

capture20171025094552399

Please provide the Windows User account to run the Snapshot Agent

capture20171025094537848

After providing the details, click “Ok” to close and then review the user name provided and click “Next” to continue

Step 11: By default “Create the publication” option will be selected, Please select “Generate a script file with steps to create the publication” if you prefer to run the script manually

capture20171025094603896

If you selected the 2nd option, Please specify the path where you want the script to be created and click “Next” to continue

capture20171025094612146

Step 12: Please provide the Publication Name and review the settings and click “Finish” to start the publisher/distributor configuration process

capture20171025094629258

We can see the progress for each steps

capture20171025094635981

After all steps are completed, we should see the Success message and click “Close” to continue

capture20171025094745397

Step 13: To verify the created publication, Please refresh the Local Publications node under Replication and you should see the newly created Publication

capture20171026124457112

We have successfully created the Publication now, you can verify the status of initial snapshot scripts creating by verifying the job status in SQL Server Agent – Job Activity Monitor

We can see the highlighted job as completed Successfully

capture20171026132659759

How to setup SQL Azure Database as Subscriber

We will now see how to create a new SQL Azure Database subscriber

Step 1: Right click on the Replication –> Local Publications –> Publication Name and click on “New Subscriptions…”

 

capture20171026124759159

 

Step 2: Click “Next” to continue

capture20171025094823157

 

Step 3: Please review the selected Publisher and Publication and click “Next” to continue

capture20171025094828599

Step 4: Please select “Run all agents at the Distributor” option and click “Next” to continue

capture20171025094835370

Step 5: Click on “Add Subscriber

capture20171025094903713

Click on “Add SQL Server Subscriber”

capture20171025094910631

Please provide the SQL Azure Database server name and select SQL Server Authentication and provide the user name and password and click “Connect” to continue

capture20171025095231591

 

Step 6: Please select the Subscription Database and click “Next” to continue

capture20171025143325718

Step 7: Please click  “…” button to configure the Distribution Agent security

capture20171025143431898

Please provide the windows user name to run the process account in Distributor Agent

For “Connect to the Subscriber”, Please provide the SQL Server Login account to connect

capture20171025143419741

Please click “Ok” and then review the selected settings and click “Next” to continue

Step 8: Please specify the Agent Schedule and click “Next” to continue

capture20171025143445700

Step 9: Please select “Initialize” and select “Immediately” for Initialize When, Please choose this accordingly if your snapshot agent job is completed or not

Click “Next” to continue

capture20171025143505239

Step 10: By default “Create the subscription(s)” will be selected, if you prefer to create the subscription manually using script, Please select Generate Script option and click “Next” to continue

capture20171025143513271

Please provide the script path and click “Next” to continue

capture20171025143518278

Step 11: Please review the settings and click “Finish” to start the process

capture20171025143522741

Once the process is complete, Please refresh the publication to see the Subscription

capture20171025144459332

After the Replication Distribution job is run, Please connect to SQL Azure database to see the Table is created in SQL Azure database

capture20171025144542960

In next post, we will see how to monitor the Replication, we will also make changes to local data and see the changes being replicated to SQL Azure

Hope you find this post helpful !!!

How to import CSVs from Azure Blob Storage to SQL Server 2017 ?

October 26, 2017 Leave a comment

SQL Server 2017 introduces option to import CSV files stored in Azure Blog Storage

In this article we will see how to implement that step by step

Step 1: Create Database Master Encryption Key

This step is optional, if you already have created your database master encryption key, Please skip this

CREATE MASTER KEY ENCRYPTION
	BY PASSWORD = 'PleaseUseStrongPasswordHere'

Step 2: Create Database Scoped Credential

CREATE DATABASE SCOPED CREDENTIAL DemoAzureCredential
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET =
	'INSERT YOUR SHARED ACCESS SIGNATURE HERE';

Please note when you are copying the Share Access Signature from Azure Portal, it will copy with ?  as prefix, and you will need to remove it

Step 3: Create External Data Source

Please make sure to replace sqlxpertise  with your own Blob account name and democsvs with your container name

CREATE EXTERNAL DATA SOURCE DemoAzureBlobCSVs
    WITH  (
        TYPE = BLOB_STORAGE,
        LOCATION = 'https://sqlxpertise.blob.core.windows.net/democsvs',
        CREDENTIAL = DemoAzureCredential
    );

Step 4: Test External Data source connection using OPENROWSET command

SELECT * FROM OPENROWSET(
   BULK  'DemoProducts.txt',
   DATA_SOURCE = 'DemoAzureBlobCSVs',
   SINGLE_CLOB) AS DataFile;

Sample output – This is based on the input file I used and it will vary based on your input file uploaded to Azure Blog Storage

capture20171024154723249

Step 5: Import Data to table

CREATE TABLE tmpProducts
(
	[productid] [int] IDENTITY(1,1) NOT NULL,
	[productname] [nvarchar](40) NOT NULL,
	[supplierid] [int] NOT NULL,
	[categoryid] [int] NOT NULL,
	[unitprice] [money] NOT NULL,
	[discontinued] [bit] NOT NULL,
 ) 

 BULK INSERT tmpProducts
		FROM 'DemoProducts.txt'
	   WITH
       (
			              DATA_SOURCE = 'DemoAzureBlobCSVs',
                        FORMAT = 'CSV'
       );

Output

capture20171024155032516

Once data is successfully imported, let us review the data imported

 SELECT productid,
        productname,
        supplierid,
        categoryid,
        unitprice,
        discontinued FROM dbo.tmpProducts

Output

capture20171024155144082

Note: BULK INSERT is working only for permanent tables and not temporary tables, You will get below error message if you try importing temporary tables

CREATE TABLE #tmpProducts
(
	[productid] [int] IDENTITY(1,1) NOT NULL,
	[productname] [nvarchar](40) NOT NULL,
	[supplierid] [int] NOT NULL,
	[categoryid] [int] NOT NULL,
	[unitprice] [money] NOT NULL,
	[discontinued] [bit] NOT NULL,
 ) 

 BULK INSERT #tmpProducts
	FROM 'DemoProducts.txt'
	WITH (DATA_SOURCE = 'DemoAzureBlobCSVs', FORMAT = 'CSV');

Error Message

Msg 12703, Level 16, State 1, Line 57
Referenced external data source “DemoAzureBlobCSVs” not found.

Yes, the message is misleading, but I am sure, Microsoft SQL Server team will realize and change this.

Hope you find this post helpful !!!

Temporal Tables – Part 5- Simple Guide to implement History Tables using SQL Server 2016/2017– Storing History data in Azure using Stretch Database

October 25, 2017 Leave a comment

In previous posts, we saw

Part 1 – How to create Temporal Tables

Part 2 – Accessing History data from Temporal Tables

Part 3 – Modifying Temporal Tables, Disabling System versioning and why we need to disable ?

Part 4 – Managing Temporal Tables and History Data

In this post we will see how to store the history data in Azure using Stretch database

Step 1: Right click on Database name and click on Tasks –> Stretch –> Enable

capture20171023145111550

Step 2: Select the tables which you would like to enable Stretch

capture20171023145132469

Step 3: Click on “Entire Table” hyperlink in Migrate column to customize the criteria for Stretch, by default the Entire table will be stored in SQL Azure

Please specify the name of the Stretch Criteria, Please select the Column based on which you want to apply the filter criteria , select the Condition and then specify the value

capture20171023145208563

capture20171023143520236

After specify the criteria, click on “Check” to validate, once you “Success”, Please click “Done” to continue

Step 4:  Please review the selected tables and click “Next” to continue

capture20171023145217186

Step 5: Please sign in to your Azure account, if you are not already signed in, Please select the Subscription, Azure region

We can create a new SQL Azure server or use an existing server, in this demo I am using an existing server

After providing details, click “Next” to continue

capture20171023145308669

Step 6: Please provide the Public IP Address range for which you need to enable access, If you have only one Public IP Address, Please specify that for both From and To and then click Next to proceed

FYI, If you created a new server, by default this step will enable the Firewall for Azure services

capture20171023145338248

Step 7: Review the settings selected and also review the estimated Pricing, this will change based on the number of tables and size of data you are planning to store in SQL Azure

Please click “Finish” to start the Stretch enabling process

capture20171023145344372

We can review the Stretch deployment process in below screenshot, based on your selecting, the steps shown below can change

capture20171023145350517

Once all the steps are completed, we can see status as “Passed”, if you want to see the verbose details on what happened in each step, we can review the Logs, To access the logs, Please click on Hyperlink, This will be very helpful when there is any error occurred

capture20171023145653549

How to monitor Stretch databases ?

We can now see the Icon for the database has changed

capture20171023145727577

To monitor the progress of Stretch, Please right click on Database Name –> Tasks –> Stretch –> Monitor

capture20171023145943961

In Monitor window, we can see the Source server, Target Azure server, Database and list of tables being stretched and number of rows uploaded

capture20171023145858728

We can also connect to the SQL Azure server and access the history data as shown below

capture20171023150934383

capture20171023150943585

capture20171023151039526

How to disable Stretch for a particular table / database ?

To disable Stretch for a table, right click on table name –> Stretch –> Disable

You have 2 options

1. Bring data back from Azure – This will download all the data from SQL Azure and update your local server

2. Leave data in Azure – This will leave the data as is in SQL Azure server and new data will be stored locally

Please choose the option based on your requirement

capture20171023150056965

In this demo, we are using “Bring data back from Azure” option, Please click “Yes” to confirm the disable process

capture20171023151116898

Depending on the size of the data, the process will run and you will see the below message

capture20171023151124884

After disabling Stretch on all the tables, we can disable the Stretch at the database level by right clicking on Database name –> Tasks –> Stretch –> Disable

capture20171023151619546

FYI, If you try disabling the Stretch in your database, before disabling Stretch in the tables, you will see this below validation message

capture20171023150002544

Once the Stretch disable process is completed, we will see this below completed message

capture20171023151629941

FYI, You can use the Stretch option not just for Temporal History tables, you can use this for any other tables where you want to store historical data or least accessed data remotely for reference or compliance purposes

Hope you find this post helpful !!!

Simple Guide to provision SQL Server 2017 on Linux in Azure Virtual Machine

October 12, 2017 1 comment

Hi Everyone, In this article, we will see how to provision SQL Server 2017 on Linux in Azure Virtual Machine

Step 1: Please login to azure portal and go to Virtual Machine blade

capture20171006153829221

Step 2: Please search for SQL Server 2017

capture20171006153917048

Step 3: Please select SQL Server 2017 Developer on Ubuntu Server, I have used Ubuntu linux for this demo but you can use Red Hat or Suse Linux as well

capture20171006153950231

Step 4: Please review the legal terms, select deployment model, I prefer to use Resource Manager, so that management and cleanup is easier, Please also copy the highlighted command which will be helpful for configuring the sa password for SQL Server later

capture20171006154030026

Step 5: Please provide the name of the VM, select Disk type, provide user name and select Authentication Type, in this case I am using Password

Please select the Azure Subscription

For Resource group, I am creating a new one named “SQLLinux” and using Azure data center location as “East US”, please change this according to your region

capture20171006155312325

Step 6: In this step, we need to select the Size of our Linux VM, I am using B2S Standard, which has 2 vCPUs and 4 GB RAM, which is good for our testing, but you can choose the size based on your requirement, after selecting the size, Please click “Select” to proceed

capture20171006155634830

Step 7: In this step, we will review some of the important settings

Virtual Network – For this demo, I am creating a new Virtual Network, but if you already have other VMs running in Azure and you need this SQL Server VM to be accessible for them, Please use the existing Virtual Network

Public IP Address – I have enabled Public IP Address for this demo, but this is optional, Please don’t enable this, if you are planning to use this server only within Azure, so that it will reduce the exposure

Network Security Group (Firewall) – We will see in next step, How to configure firewall to allow external access

Auto Shutdown – This is an useful optional setting for demo or test servers, so that you can configure it to shutdown automatically based on a timing, this will reduce billing of your VM

Time Zone – Please select the time zone based on your region

capture20171006155902614

Step 8: Additional Settings Please enable Monitoring for both Boot diagnostics and Guest OS diagnostics, This will help you see the Boot log for any error or messages, after making required Settings changes, Click “Ok” to proceed

capture20171006155918920

Step 9:  To allow external access to SQL Server,  we need to change Network Firewall setting by adding Inbound and Outbound security rules for MSSQL Service

Note: You can do this change after creating the Virtual machine

capture20171006161838234

capture20171006161922839

Step 10: Review the selection options and click “Purchase” to buy and create the virtual machine

capture20171006160025569

Step 11: You can review the Virtual Machine deployment progress in the screen

capture20171006160058057

If you closed it, you can see from the Top right corner Notifications area

capture20171006160124574

You can also see the status under Virtual Machines blade as well

capture20171006160259721

Once Virtual machine is provisioned and ready, you should see the status as “Running”, see below screenshot

capture20171006160508333

Step 12:  Please click on the Virtual Machine to see the settings and also to monitor the CPU/Network/Disk usage

capture20171006160608869

Step 13: Configuring SQL Server in Azure Linux VM

Please connect to Azure Linux VM using any preferred SSH client, In this demo, I am using Bash Shell on Windows, since its built in Windows 10 now

Command to connect, Please type “Yes” to accept the authenticity of the host

ssh username@serveripaddress

capture20171006160654951

Step 14: After logging in to VM, Please run the below command to check the status of SQL Server

sudo systemctl status mssql-server

Since we haven’t configured SA password, SQL Server is not running, so you seeing the status as “Inactive”

capture20171006160938531

Step 15: Please run below command to configure SA password

sudo /opt/mssql/bin/mssql-conf setup

Please select your preferred edition of SQL Server, in this demo, we will use Developer edition, in next prompt, Please specify the SA password and press enter

capture20171006161516727

Once password is setup, SQL Server 2017 will automatically run, Please run the status command again to review the SQL Server status

sudo systemctl status mssql-server

capture20171006161118193

Step 16: Please run “SELECT @@VERSION” using SQLCMD utility to connect to the server to verify we are able to connect and see SQL Server version

capture20171006161645028

Step 17: Now we can use SQL Server Management Studio to connect using the Public IP Address and SA password provisioned

capture20171006162510369

Once connected, Please create your required databases and database objects and use as needed

capture20171006162600552

capture20171006162708830

SQL Server 2017 Developer Edition is up and running Azure Linux VM now in under 5 minutes

I believe you will find this post useful !!!

SQL Azure Import/Export Service has been released to Production

January 24, 2012 Leave a comment

SQL Azure Import/Export Service has been released to Production as of today!!!

SQL Azure Import/Export Service is available at free of charge for SQL Azure customers.

For more information visit the following blog post

Manage SQL Azure server using PowerShell

September 21, 2011 Leave a comment

Windows Azure Platform Powershell Cmdlets has been released and available for download at http://wappowershell.codeplex.com/releases/view/73463

It supports creating SQL Azure server, removing an existing SQL Azure server, managing the firewall and resetting the Administrator password.

For more detailed information, please visit

http://wappowershell.codeplex.com/documentation

How to create Co-Administrator for SQL Azure Server ?

July 15, 2011 Leave a comment

SQL Azure Co-Administrator option has been released in July 2011 Service Release

In this article, we will see How to add a Co-Administrator for your subscription

After logging in to Windows Azure Management Portal, click on “Database” tab

image

After navigating to Database section, click on “User Management” in Tool bar

image

In User Management screen, You can see the currently available Service Administrators and Co-Administrator for all available subscriptions

 image

Click on “Add New Co-Admin”

image

In Add New Co-Administrator screen, specify the new users’ Windows Live ID and then select the subscription he can administer and then click “Ok”

image

Now we have successfully created a new Co-Administrator, Now he can login and create or drop servers, create or drop databases and new co-administrators as well

image

To remove a subscription or remove a Co-Administrator, click on Co-Administrator name and then click on “Manage Co-Admin”

image

If you have multiple subscriptions, you can remove the subscription which the co-admin should not manage, if you remove all subscriptions, the co-admin account will be removed automatically.

Now let us login using Co-Administrator login and see how the users are displayed

image

Since the Co-Administrator already has a subscription, he is displayed as Service Administrator for one of the account and Co-Administrator for other account.

Hope you all find this information helpful !!!

%d bloggers like this: