Archive

Archive for the ‘SQL Server’ Category

How to import CSVs from Azure Blob Storage to SQL Server 2017 ?

October 26, 2017 Leave a comment

SQL Server 2017 introduces option to import CSV files stored in Azure Blog Storage

In this article we will see how to implement that step by step

Step 1: Create Database Master Encryption Key

This step is optional, if you already have created your database master encryption key, Please skip this

CREATE MASTER KEY ENCRYPTION
	BY PASSWORD = 'PleaseUseStrongPasswordHere'

Step 2: Create Database Scoped Credential

CREATE DATABASE SCOPED CREDENTIAL DemoAzureCredential
WITH IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET =
	'INSERT YOUR SHARED ACCESS SIGNATURE HERE';

Please note when you are copying the Share Access Signature from Azure Portal, it will copy with ?  as prefix, and you will need to remove it

Step 3: Create External Data Source

Please make sure to replace sqlxpertise  with your own Blob account name and democsvs with your container name

CREATE EXTERNAL DATA SOURCE DemoAzureBlobCSVs
    WITH  (
        TYPE = BLOB_STORAGE,
        LOCATION = 'https://sqlxpertise.blob.core.windows.net/democsvs',
        CREDENTIAL = DemoAzureCredential
    );

Step 4: Test External Data source connection using OPENROWSET command

SELECT * FROM OPENROWSET(
   BULK  'DemoProducts.txt',
   DATA_SOURCE = 'DemoAzureBlobCSVs',
   SINGLE_CLOB) AS DataFile;

Sample output – This is based on the input file I used and it will vary based on your input file uploaded to Azure Blog Storage

capture20171024154723249

Step 5: Import Data to table

CREATE TABLE tmpProducts
(
	[productid] [int] IDENTITY(1,1) NOT NULL,
	[productname] [nvarchar](40) NOT NULL,
	[supplierid] [int] NOT NULL,
	[categoryid] [int] NOT NULL,
	[unitprice] [money] NOT NULL,
	[discontinued] [bit] NOT NULL,
 ) 

 BULK INSERT tmpProducts
		FROM 'DemoProducts.txt'
	   WITH
       (
			              DATA_SOURCE = 'DemoAzureBlobCSVs',
                        FORMAT = 'CSV'
       );

Output

capture20171024155032516

Once data is successfully imported, let us review the data imported

 SELECT productid,
        productname,
        supplierid,
        categoryid,
        unitprice,
        discontinued FROM dbo.tmpProducts

Output

capture20171024155144082

Note: BULK INSERT is working only for permanent tables and not temporary tables, You will get below error message if you try importing temporary tables

CREATE TABLE #tmpProducts
(
	[productid] [int] IDENTITY(1,1) NOT NULL,
	[productname] [nvarchar](40) NOT NULL,
	[supplierid] [int] NOT NULL,
	[categoryid] [int] NOT NULL,
	[unitprice] [money] NOT NULL,
	[discontinued] [bit] NOT NULL,
 ) 

 BULK INSERT #tmpProducts
	FROM 'DemoProducts.txt'
	WITH (DATA_SOURCE = 'DemoAzureBlobCSVs', FORMAT = 'CSV');

Error Message

Msg 12703, Level 16, State 1, Line 57
Referenced external data source “DemoAzureBlobCSVs” not found.

Yes, the message is misleading, but I am sure, Microsoft SQL Server team will realize and change this.

Hope you find this post helpful !!!

Advertisements

Temporal Tables – Part 5- Simple Guide to implement History Tables using SQL Server 2016/2017– Storing History data in Azure using Stretch Database

October 25, 2017 Leave a comment

In previous posts, we saw

Part 1 – How to create Temporal Tables

Part 2 – Accessing History data from Temporal Tables

Part 3 – Modifying Temporal Tables, Disabling System versioning and why we need to disable ?

Part 4 – Managing Temporal Tables and History Data

In this post we will see how to store the history data in Azure using Stretch database

Step 1: Right click on Database name and click on Tasks –> Stretch –> Enable

capture20171023145111550

Step 2: Select the tables which you would like to enable Stretch

capture20171023145132469

Step 3: Click on “Entire Table” hyperlink in Migrate column to customize the criteria for Stretch, by default the Entire table will be stored in SQL Azure

Please specify the name of the Stretch Criteria, Please select the Column based on which you want to apply the filter criteria , select the Condition and then specify the value

capture20171023145208563

capture20171023143520236

After specify the criteria, click on “Check” to validate, once you “Success”, Please click “Done” to continue

Step 4:  Please review the selected tables and click “Next” to continue

capture20171023145217186

Step 5: Please sign in to your Azure account, if you are not already signed in, Please select the Subscription, Azure region

We can create a new SQL Azure server or use an existing server, in this demo I am using an existing server

After providing details, click “Next” to continue

capture20171023145308669

Step 6: Please provide the Public IP Address range for which you need to enable access, If you have only one Public IP Address, Please specify that for both From and To and then click Next to proceed

FYI, If you created a new server, by default this step will enable the Firewall for Azure services

capture20171023145338248

Step 7: Review the settings selected and also review the estimated Pricing, this will change based on the number of tables and size of data you are planning to store in SQL Azure

Please click “Finish” to start the Stretch enabling process

capture20171023145344372

We can review the Stretch deployment process in below screenshot, based on your selecting, the steps shown below can change

capture20171023145350517

Once all the steps are completed, we can see status as “Passed”, if you want to see the verbose details on what happened in each step, we can review the Logs, To access the logs, Please click on Hyperlink, This will be very helpful when there is any error occurred

capture20171023145653549

How to monitor Stretch databases ?

We can now see the Icon for the database has changed

capture20171023145727577

To monitor the progress of Stretch, Please right click on Database Name –> Tasks –> Stretch –> Monitor

capture20171023145943961

In Monitor window, we can see the Source server, Target Azure server, Database and list of tables being stretched and number of rows uploaded

capture20171023145858728

We can also connect to the SQL Azure server and access the history data as shown below

capture20171023150934383

capture20171023150943585

capture20171023151039526

How to disable Stretch for a particular table / database ?

To disable Stretch for a table, right click on table name –> Stretch –> Disable

You have 2 options

1. Bring data back from Azure – This will download all the data from SQL Azure and update your local server

2. Leave data in Azure – This will leave the data as is in SQL Azure server and new data will be stored locally

Please choose the option based on your requirement

capture20171023150056965

In this demo, we are using “Bring data back from Azure” option, Please click “Yes” to confirm the disable process

capture20171023151116898

Depending on the size of the data, the process will run and you will see the below message

capture20171023151124884

After disabling Stretch on all the tables, we can disable the Stretch at the database level by right clicking on Database name –> Tasks –> Stretch –> Disable

capture20171023151619546

FYI, If you try disabling the Stretch in your database, before disabling Stretch in the tables, you will see this below validation message

capture20171023150002544

Once the Stretch disable process is completed, we will see this below completed message

capture20171023151629941

FYI, You can use the Stretch option not just for Temporal History tables, you can use this for any other tables where you want to store historical data or least accessed data remotely for reference or compliance purposes

Hope you find this post helpful !!!

Temporal Tables–Simple Guide to implement History Tables using SQL Server 2016/2017–Part 4–Managing Temporal Tables and History Data

October 24, 2017 1 comment

In previous posts, we saw

Part 1 – How to create Temporal Tables

Part 2 – Accessing History data from Temporal Tables

Part 3 – Modifying Temporal Tables, Disabling System versioning and why we need to disable ?

In this post we will see how to manage Temporal tables and History

To get list of Temporal Tables – Snippet 1

SELECT name,
       object_id,
       temporal_type,
       temporal_type_desc,
       history_table_id,
       is_remote_data_archive_enabled,
       is_external,
       history_retention_period,
       history_retention_period_unit,
       history_retention_period_unit_desc
FROM sys.tables
WHERE temporal_type > 0
ORDER BY name;

 

Output

capture20171023102432733

If we see the output, we can identify the following

1. Type of Temporal Table – System Versioned or History table using “temporal_type” and “temporal_type_desc” columns

2. For System Versioned tables, we can also see the History Table ID

3. History Retention Period can be determined using history_retention_period, history_retention_period_unit and history_retention_period_unit_desc

history_retention_period = -1  = History is kept forever

Positive Integer = Refers to the number of Days/Weeks/Years

history_retention_period_unit will have one of these values

-1 = INFINITE

3 = DAY(S)

4 = WEEK(S)

5 = MONTH(S)

6 = YEAR(S)

By default, when we create temporal tables, if we don’t specify Retention period, it will be created with Infinite period of retention

Modifying Retention Period

Let us see how we can change Retention Period for an existing Temporal Table

Code to alter table to set Retention Period for 2 days – Snippet 2

ALTER TABLE dbo.Demo
	SET (SYSTEM_VERSIONING = ON
			(
				HISTORY_TABLE = dbo.DemoHistory,
				HISTORY_RETENTION_PERIOD = 2 DAYS
			)
		)

If we run query from Snippet 1, we will see below output

capture20171023103757653

To get list of Databases with Retention Enabled – Snippet 3

Once retention period is enabled for any table in the database, SQL Server will set the is_temporal_history_retention_enabled flag = 1 for the database by default and also create background jobs to cleanup the history data

SELECT name,
       database_id,
       is_temporal_history_retention_enabled
FROM sys.databases
WHERE is_temporal_history_retention_enabled = 1
ORDER BY name;

Output

capture20171023104055758

If you want to disable automatic cleanup task, Please use below command

Query to disable cleanup task – Snippet 4

ALTER DATABASE SQL2017_DEMO
     SET TEMPORAL_HISTORY_RETENTION OFF

If we run the query in Snippet 3, we can see the changes

capture20171023104605524

How to cleanup History tables manually ?

If you prefer to cleanup the History tables manually rather than automated, Please follow below steps

1. Make your application offline

2. Run query to disable System versioning on required tables

3. Run query to delete data from History using SysEndTime as Filter criteria

4. Run query to enable System versioning for tables disabled in Step 2

5. Make your application online

If your application has to be on 24/7, better option is to use the in-built automated cleanup task

Hope you find this post helpful !!!

Temporal Tables – Simple Guide to implement History Tables using SQL Server 2016/2017–Part 3–Modifying Temporal Tables, Disabling System Versioning and why we need to disable

October 23, 2017 2 comments

Part 1 – How to create Temporal tables

Part 2 – Accessing History data from Temporal Tables

In this post we will how to modify Temporal Tables, How to disable System Versioning in Temporal tables to update large set data of without affecting history or truncate tables

Modifying Temporal Tables

For adding new columns to Temporal tables, Please use below command

ALTER TABLE dbo.Demo ADD DemoDesc VARCHAR(255) NULL

Screenshot before adding the new column

capture20171022142508566

Screenshot after adding the new column – If we notice, adding the column in the main table automatically modifies the history table as well

capture20171022142736382

Similarly we can modify the tables to drop column or increasing the size of the field, but some of the changes to table are not supported ex. Adding Identify column or changing the data type of the field which is not supported using implicit conversion, in that scenario we need to disable the system versioning to make changes

Disable System Versioning

Screenshot before disabling system versioning

capture20171022143805111

Script to disable system versioning

ALTER TABLE dbo.Demo	SET (SYSTEM_VERSIONING = OFF);

Screenshot after disabling system versioning – If we notice the System versioning is disabled and both Main and history table is displayed as regular tables now

capture20171022143922452

There are other scenarios we need to disable System versioning, if we need to do any of the following operations

1. TRUNCATE TABLE on Main or History table

Msg 13545, Level 16, State 1, Line 1
Truncate failed on table ‘SQL2017_DEMO.dbo.Demo’ because it is not a supported operation on system-versioned tables.

2. Deleting rows from History table

Msg 13560, Level 16, State 1, Line 4
Cannot delete rows from a temporal history table ‘SQL2017_DEMO.dbo.DemoHistory’.

3. Dropping Main or History table

Msg 13552, Level 16, State 1, Line 6
Drop table operation failed on table ‘SQL2017_DEMO.dbo.Demo’ because it is not a supported operation on system-versioned temporal tables.

Hope you find this post helpful !!!

Temporal Tables–Simple Guide to implement History Tables using SQL Server 2016/2017–Part 2–Accessing History data

October 21, 2017 3 comments

In previous post, we saw how to create Temporal tables or enable System Versioning on an existing table

In this post, we will review how to access History data

Using below code, we will create simple Temporal table and inserted 5 records and querying both the Main and History table

CREATE TABLE dbo.Demo 
 (    
      DemoID int NOT NULL PRIMARY KEY CLUSTERED  IDENTITY 
    , DemoName varchar(50) NOT NULL  
    , SysStartTime datetime2 GENERATED ALWAYS AS ROW START NOT NULL  
    , SysEndTime datetime2 GENERATED ALWAYS AS ROW END NOT NULL  
    , PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)     
 )    
 WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.DemoHistory)) 
 ;

INSERT INTO dbo.Demo (DemoName) VALUES ('Demo1') 
 INSERT INTO dbo.Demo (DemoName) VALUES ('Demo2') 
 INSERT INTO dbo.Demo (DemoName) VALUES ('Demo3') 
 INSERT INTO dbo.Demo (DemoName) VALUES ('Demo4') 
 INSERT INTO dbo.Demo (DemoName) VALUES ('Demo5')

SELECT DemoID, 
        DemoName, 
        SysStartTime, 
        SysEndTime FROM dbo.Demo

SELECT DemoID, 
        DemoName, 
        SysStartTime, 
        SysEndTime FROM dbo.DemoHistory

 

Output: Main table has 5 rows and and History has no data, because no changes were made yet to the data, Also if you notice the SysEndTime, all of them are set to 12/31/9999 23:59:59.9999999, which refer to them as current record

capture20171020104246809

Let us modify some data in 2 minute intervals

First Change – Update 1 row and delete 1 row

UPDATE dbo.Demo 
    SET DemoName='Demo11' 
    WHERE DemoID=1

DELETE FROM dbo.Demo WHERE DemoID=4

Output: Now we can see that Main table has 4 rows and History has 2 rows , the green highlighted is the record we updated as you can history table contains the previous version of record and also SysEndTime in History table indicates when it was updated

Similarly Red highlighted record in history is the deleted with Deleted time

capture20171020104303651

Second Update: Update 1 record and Delete 1 record

UPDATE dbo.Demo 
    SET DemoName='Demo12' 
    WHERE DemoID=2

DELETE FROM dbo.Demo WHERE DemoID=3

Output:

capture20171020121832704

Let us now review how we can access the History data

Method 1: AS OF

We can use FOR SYSTEM_TIME AS OF to get records in Demo as of the specific time, This will be helpful for scenarios when you want to see How much Inventory was in hand as of a particular date and time or What products were in hand as of particular date and time

In below example we are using 3 different time with 2 minute intervals

SELECT DemoID, 
       DemoName, 
       SysStartTime, 
        SysEndTime FROM dbo.Demo 
    FOR SYSTEM_TIME AS OF '2017-10-20 14:42:34' 
    ORDER BY DemoID

SELECT DemoID, 
       DemoName, 
       SysStartTime, 
       SysEndTime FROM dbo.Demo 
    FOR SYSTEM_TIME AS OF '2017-10-20 14:44:34' 
    ORDER BY DemoID

SELECT DemoID, 
       DemoName, 
        SysStartTime, 
       SysEndTime FROM dbo.Demo 
    FOR SYSTEM_TIME AS OF '2017-10-20 14:46:34' 
    ORDER BY DemoID

Output: As we can see in first output we are able to see all records as how it was inserted and in 2nd put, we can see 1st record was updated and Demo ID 4 record is missing and in output 3 23 can seem Demo ID 1 and 2 are updated and 3 and 4 are deleted

capture20171020122217154

Method 2: FROM…TO and  BETWEEN…AND option

We can use FROM…TO and BETWEEN…AND option to get records changed between 2 specified date times

SELECT DemoID, 
       DemoName, 
       SysStartTime, 
       SysEndTime 
    FROM dbo.Demo 
    FOR SYSTEM_TIME FROM '2017-10-20 14:42:33' TO '2017-10-20 14:46:55' 
    ORDER BY DemoID, SysStartTime

SELECT DemoID, 
        DemoName, 
       SysStartTime, 
       SysEndTime 
    FROM dbo.Demo 
    FOR SYSTEM_TIME BETWEEN '2017-10-20 14:42:33' AND '2017-10-20 14:46:55' 
    ORDER BY DemoID, SysStartTime

 

Output: In below output you can see all changes made between the specified time intervals, but if you notice it, its providing us both version of the records, so based on your application need, you will need to use this or else, it might provide more records for display than you need

capture20171020123046024

Method 3: CONTAINED IN

When we use CONTAINED IN option, it provides data from history table only for the specified datetime(s)

SELECT DemoID, 
        DemoName, 
        SysStartTime, 
       SysEndTime 
    FROM dbo.Demo 
    FOR SYSTEM_TIME CONTAINED IN ('2017-10-20 14:42:33', '2017-10-20 14:45:56') 
    ORDER BY DemoID, SysStartTime

 

Output: As we can see the below output doesn’t contain the current version of record or other records from Main table

capture20171020123430862

Method 4: ALL

When we use ALL option, it provides data from both Main and History table for all the changes made so far

SELECT DemoID, 
        DemoName, 
        SysStartTime, 
       SysEndTime FROM dbo.Demo 
    FOR SYSTEM_TIME ALL 
    ORDER BY DemoID, SysStartTime

Output

We can see all records from main and history table combined, Please be cautious using this option when running without Filter criteria for larger tables

capture20171020123659353

We can create Views using Temporal Tables and those views supports these clauses as well

Hope you find this post helpful !!!

Temporal Tables–Simple Guide to implement History Tables using SQL Server 2016/2017

October 20, 2017 4 comments

Temporal Tables is a new feature introduced in SQL Server 2016, which makes database/application developers life easy to implement History feature in their application without having to write Triggers or any extra logic in stored procedures

Method 1: Creating Temporal Tables for New Table without explicitly specifying History table name

Please add these 3 columns to your table definition and use WITH (SYSTEM_VERSIONING = ON)

CREATE TABLE dbo.Demo
(
DemoID int NOT NULL PRIMARY KEY CLUSTERED
, DemoName varchar(50) NOT NULL
, SysStartTime datetime2 GENERATED ALWAYS AS ROW START NOT NULL
, SysEndTime datetime2 GENERATED ALWAYS AS ROW END NOT NULL
, PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)

)
WITH (SYSTEM_VERSIONING = ON);

In below screenshot, you can see that Demo table is created with History table, but History table name is generated by SQL Server itself, the default naming conversion is dbo.MSSQL_TemporalHistoryFor<OBJECTID of Table>

capture20171019184815324

Method 2: Creating Temporal Tables for New Table with explicit History table name

CREATE TABLE dbo.Demo2
(
DemoID int NOT NULL PRIMARY KEY CLUSTERED
, DemoName varchar(50) NOT NULL
, SysStartTime datetime2 GENERATED ALWAYS AS ROW START NOT NULL
, SysEndTime datetime2 GENERATED ALWAYS AS ROW END NOT NULL
, PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)

)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.Demo2History));

In the below screenshot, you can see the dbo.Demo2 table is created with History table name

capture20171019193451893

Method 3: Altering an existing table to implement Temporal table

The below table dbo.Demo3 already exists

capture20171019194304832

ALTER TABLE dbo.Demo3 ADD SysStartTime datetime2 GENERATED ALWAYS AS ROW START NOT NULL
, SysEndTime datetime2 GENERATED ALWAYS AS ROW END NOT NULL
, PERIOD FOR SYSTEM_TIME (SysStartTime,SysEndTime)

ALTER TABLE dbo.Demo3
SET (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.Demo3History))

In below screenshot you can see Demo3 table is modified to include

capture20171019194022657

In next post, we will see how INSERT/UPDATE/DELETE operation ran on main table generates History data and how we can access it

Hope you find this post helpful !!!

sys.dm_db_log_stats–New DMF in SQL Server 2017 to keep track of your Log backup and Log changes

October 19, 2017 Leave a comment

SQL Server 2017 introduces a new Dynamic Management Function “sys.dm_db_log_stats” to track information on log files, the most interesting information it provides is when was the last log backup completed and size of the log growth since last backup

This will be very helpful for high transaction system when log backup has to be done on regular basis, you can create your monitoring jobs to track using functions whether log backup is running or not using “log_backup_time

Other option we can design the log backup job is to run it based on specific size of growth since last backup using “log_since_last_log_backup_mb” instead of running it on regular schedule, that will reduce the number of log backups as well on times when there are less or no changes

Sample Usage

DECLARE @DBID INT = Db_id()

SELECT
       database_id,
       recovery_model,
       total_log_size_mb,
       active_log_size_mb,
       log_truncation_holdup_reason,
       log_backup_time,
       log_since_last_log_backup_mb
FROM   sys.Dm_db_log_stats(@DBID)

Output

capture20171018120433316

Hope you find this post helpful !!!

%d bloggers like this: