Monday, February 9, 2015

Usage of NOLOCK hint and READ UNCOMMITTED.

I've been hit again by wide usage of NOLOCK hint in a client's code.

To prove for the client and for myself necessity of using that hint I've made an experiment, which want to post here.

At first, will go to TestDB and create three tables: tbl_Test with test data, tbl_Counts - there will be inserted counts of records from tbl_Test table, tbl_Flag - communication table to transfer a signal between two SSMS windows.

As a result of that script table tbl_Test has to be populated by 10,000 "Empty" records.



-- # Script 1 - Window 1 - Preparation
use TestDB
GO
-- Drop Test Table if exists
IF EXISTS ( SELECT TOP 1 1 FROM sys.tables WHERE name = 'tbl_Test')
       DROP TABLE tbl_Test;
GO
-- Drop Test Table if exists
IF EXISTS ( SELECT TOP 1 1 FROM sys.tables WHERE name = 'tbl_Counts')
       DROP TABLE tbl_Counts;
GO
-- Drop Test Table if exists
IF EXISTS ( SELECT TOP 1 1 FROM sys.tables WHERE name = 'tbl_Flag')
       DROP TABLE tbl_Flag;
GO
-- Create Test Table
CREATE TABLE tbl_Test(
       ID INT IDENTITY (1,1) CONSTRAINT PK_Test PRIMARY KEY,
       TestText VARCHAR(100) NULL,
);
GO
CREATE TABLE tbl_Counts (Counted INT)
GO
CREATE TABLE tbl_Flag(ID INT IDENTITY(1,1))
GO
-- Fill Test records
INSERT INTO tbl_Test(TestText)
SELECT TOP 10000 NULL
FROM sys.messages;
GO

After table tbl_Test has been populated I run the second script.
That script is waiting for a signal to collect the data and collects it until another signal.

Data collection is a simple counting of records from the tbl_Test table.
If the number of records won't match 10,000 it will be reported to tbl_Counts table.


-- # Script 2 - Window 1 - Collecting the data
SET NOCOUNT ON
GO
DECLARE @i INT = 0;
WHILE @i = 0
 SELECT @i = COUNT(*) FROM tbl_Flag;

WHILE (SELECT COUNT(*) FROM tbl_Flag) > 0
BEGIN
       SELECT @i = COUNT(*) FROM tbl_Test (NOLOCK);
       IF @i != 10000
              INSERT INTO tbl_Counts(Counted) VALUES(@i);
END
GO
SELECT Counted, 10000 - Counted as Diff
FROM tbl_Counts;
GO

While the second script is running I open another SSMS tab an start third script.
That script inserts one record into tbl_Flag table to activate data collection and then just does an update of tbl_Test. After update is finished it sends signal back to stop collecting the data.


-- # Script 3 - Window 2 - Changing test data
INSERT INTO tbl_Flag DEFAULT VALUES;
GO
UPDATE tbl_FF_Test
SET TestText = REPLICATE('A',100);
GO
TRUNCATE TABLE tbl_Flag;
GO

After that script has finished I switched to the first tab and got my expected results:


That shows that during "UPDATE" operation my second query captured some situations when number of records is not equal 10,000 records.

That is easy proof how NOLOCK hint can produce incorrect results.

Microsoft recommends to use "READ UNCOMMITTED" transaction isolation level instead of NOLOCK.
Look how how it performs.
At first, run script #1 again to recreate our tables.

Then instead of script #2 run it's modification: #4


-- # Script 4 - Window 1 - Collecting the data
SET NOCOUNT ON
GO
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
GO
DECLARE @i INT = 0;
WHILE @i = 0
 SELECT @i = COUNT(*) FROM tbl_Flag;

WHILE (SELECT COUNT(*) FROM tbl_Flag) > 0
BEGIN
       SELECT @i = COUNT(*) FROM tbl_Test;
       IF @i != 10000
              INSERT INTO tbl_Counts(Counted) VALUES(@i);
END
GO
SELECT Counted, 10000 - Counted as Diff
FROM tbl_Counts;
GO

Then in another SSMS tab run script #3 and go back for the results. They are the same.
There are several captured records which do not match count 10,000.
That means "READ UNCOMMITTED" works in the same manner as NOLOCK hint and may generate misleading data.

Now will talk about the difference.
You can not use "READ UNCOMMITTED" within a function, but you can use NOLOCK hint.


CREATE FUNCTION dbo.fn_Test()
RETURNS INT AS
BEGIN
  RETURN (SELECT COUNT(*) FROM tbl_Test (NOLOCK));
END

You have to apply "NOLOCK" hint individually to every table you want.
"READ UNCOMMITTED" will be applied to the whole statement until isolation level will be changed or until closing of connection.

Now is the most important question: Do you have to use "NOLOCK" at all? Is it "silver bullet" or is it evil?

SQL developers often use that hint to avoid unnecessary locking  of tables/pages/records during heavy transactional activity to avoid possible deadlocks.
However, as you could see from my examples you can possibly get inaccurate data as a result.
If the result is not important to you then it is OK, but if that result is used in following calculations or important financial reporting then I'd say it is not OK.

I'll try to define some ONLY cases when it would be OK to use "NOLOCK" hint:
1. For development/troubleshooting purposes, when results are not very important;
2. In extremely big and heavy transactional systems with thousands of requests per second SOME user reporting queries can be hinted with "NOLOCK". For instance: it is not big deal if Amazon store you get a duplicate or missing item in a suggestion items line.
3. For a query, which does clustered index seek or scan for historical data, which can't be changed or deleted during the request.


Generally speaking, if you have "NOLOCK" in your code means that you have big problem with performance and you are at risk of having even bigger problem of having incorrect data.

Monday, January 5, 2015

Building Sky Map in SSMS using GeoSpatial methods.


<= Drawing a simple picture in SSMS


Drawing an Ellipse in SSMS =>


I always try to link my hobbies. This time again I put together SQL and Astronomy.

In this post I will show how to build a Sky Map using GeoSpatial methods.

Will do it by steps:

1. At first we need to get database of stars. You can get it here: "www.astronexus.com/hyg". I've used the latest database "HYG 3.0", but assume there is no problem to use earlier and smaller versions.
If, by any chance, that resource is closed I loaded the data file into my Google drive and you can get it here.
After downloading extract comma separated data file "hygdata_v3.csv".

2. After you get a file upload it to a new SQL Server database on your test server using "SQL Server 2014 Import and Export Data". If you do not know where to find it search for "DTSWizard.exe". It is supposed to be in your SQL "DTS\Binn" folder.
a) Use "Flat File Source" to read the file and check check-box "Column names in the first data row".

b) For the destination create a new database on your test SQL Server:

c)Rename target table, which will be created in the new database:

d) End in the End just run the package to upload the data. Hopefully you won't have any surprises.

3. After you loaded data into SQL you can start working with it in SSMS.
Just go to your new database and run following script:


DECLARE @SpatialTable TABLE (

       Object varchar(50),

       Constellation varchar(50),

       Latitude Float,

       Longitude Float,

       Magnitude Float,

       g geography);



INSERT INTO @SpatialTable(Object, Constellation, Latitude, Longitude, Magnitude, g)

SELECT Proper, Con, CAST(ra as FLOAT)*(-15), [dec], CAST(mag as FLOAT),

       geography::STGeomFromText('POINT('

              + CONVERT(VARCHAR,CAST(ra as FLOAT)*-15)

              + ' ' + CONVERT(VARCHAR,[dec])

              + ')',104001).STBuffer((CAST(mag as FLOAT)*(-1)+5) * 0.002)

FROM dbo.tbl_Stellar_List

WHERE CAST(mag as FLOAT) <= 4.5 and Con != '';



INSERT INTO @SpatialTable(Object, g)

SELECT 'North Pole', geography::STGeomFromText('POINT(0 90)',104001)

UNION ALL

SELECT 'South Pole', geography::STGeomFromText('POINT(0 -90)',104001)

UNION ALL

SELECT 'Map Center', geography::STGeomFromText('POINT(0 0)',104001)

UNION ALL

SELECT 'Map Left', geography::STGeomFromText('POINT(-179.999 0)',104001)

UNION ALL

SELECT 'Map Right', geography::STGeomFromText('POINT(179.999 0)',104001)



SELECT * FROM @SpatialTable;

Then you can get beautiful pictures of Winter Sky constellations.
Here is "Orion":

And that is "Cassiopeia":

The problem with these pictures is that some stars look like ovals. That is because Geometrically they are projected on a Sphere and are curved a little.


Now let me explain the script I use piece by piece:
1. At first I create temporary table variable to include Star Name (Object),  Constellation name, Object's Latitude and Longitude, Magnitude of a Star (Star brightness), and Geography coordinates, understandable by SQL Server.

2. Insert list of stars into the Table Variable. I insert only stars, which are part of any constellation and have magnitude less than 4.5 (that will include only stars visible by naked eye). Astronomical magnitude value is little bit tricky: lover magnitude object is brighter than object with higher magnitude.

3. To build Stars' Geography coordinates I've used following statement:


geography::STGeomFromText('POINT('
    + CONVERT(VARCHAR,CAST(ra as FLOAT)*-15)
    + ' ' + CONVERT(VARCHAR,[dec])
    + ')',104001).STBuffer((CAST(mag as FLOAT)*(-1)+5) * 0.002)


a) At first, I created a "Point" on a sphere using "geography::STGeomFromText" function.
That function requires two parameters: Geometry coordinates and SRID (Spatial Reference Identifier)
Coordinate for a point is defined as 'POINT(1.2345 67.8901)' (Point is defined by two numbers separated by space).
For SRID I used number 104001, which is ID for Microsoft Spheroid (check system table SYS.SPATIAL_REFERENCE_SYSTEMS for the reference)

b) After defining Geography point I used a function "STBuffer()" to transform a dot into a 2D figure. The size of that figure is determined by Star's magnitude by formula: "(CAST(mag as FLOAT)*(-1)+5) * 0.002".

c) To generate a Geography point I had to transform object's Latitude from hours to degrees, to represent it in SQL. I also had to negate the Latitude value, because it has to be mirrored from it's Sky coordinates to Sphere. As a result I've got following formula: "CONVERT(VARCHAR,CAST(ra as FLOAT)*-15)".

4. After loading Stars' coordinates into Table Variable I also inserted additional map coordinates to represent Sky Map correctly. That is not necessary for the full skies, but very crucial when you try to build map of only one constellation. Points are: North, South, Center, Left and Right.

5. The last step is to select data from Table Variable, choose correct projection, Zoom and grid lines if necessary.


Now I want to show how to play with a new Sky Map.

With little modification in the WHERE clause I try to extract only Ursa Major (Great Bear) constellation, also known as Big Dipper:


DECLARE @SpatialTable TABLE (

       Object varchar(50),

       Constellation varchar(50),

       Latitude Float,

       Longitude Float,

       Magnitude Float,

       g geography);



INSERT INTO @SpatialTable(Object, Constellation, Latitude, Longitude, Magnitude, g)

SELECT Proper, Con, CAST(ra as FLOAT)*(-15), [dec], CAST(mag as FLOAT),

       geography::STGeomFromText('POINT('

              + CONVERT(VARCHAR,CAST(ra as FLOAT)*-15)

              + ' ' + CONVERT(VARCHAR,[dec])

              + ')',104001).STBuffer((CAST(mag as FLOAT)*(-1)+5) * 0.002)

FROM dbo.tbl_Stellar_List

WHERE CAST(mag as FLOAT) <= 4.5 and Con = 'Uma';



INSERT INTO @SpatialTable(Object, g)

SELECT 'North Pole', geography::STGeomFromText('POINT(0 90)',104001)

UNION ALL

SELECT 'South Pole', geography::STGeomFromText('POINT(0 -90)',104001)

UNION ALL

SELECT 'Map Center', geography::STGeomFromText('POINT(0 0)',104001)

UNION ALL

SELECT 'Map Left', geography::STGeomFromText('POINT(-179.999 0)',104001)

UNION ALL

SELECT 'Map Right', geography::STGeomFromText('POINT(179.999 0)',104001)



SELECT * FROM @SpatialTable;

As you can see, picture is not very good because constellation has been split on both sides of a map.
In order to fix this problem you have to shift the sphere 180 degrees.
Just add 180 degrees to Latitude value:


DECLARE @SpatialTable TABLE (

       Object varchar(50),

       Constellation varchar(50),

       Latitude Float,

       Longitude Float,

       Magnitude Float,

       g geography);



INSERT INTO @SpatialTable(Object, Constellation, Latitude, Longitude, Magnitude, g)

SELECT Proper, Con, CAST(ra as FLOAT)*(-15), [dec], CAST(mag as FLOAT),

       geography::STGeomFromText('POINT('

              + CONVERT(VARCHAR,CAST(ra as FLOAT)*-15+180)

              + ' ' + CONVERT(VARCHAR,[dec])

              + ')',104001).STBuffer((CAST(mag as FLOAT)*(-1)+5) * 0.002)

FROM dbo.tbl_Stellar_List

WHERE CAST(mag as FLOAT) <= 4.5 and Con = 'Uma';



INSERT INTO @SpatialTable(Object, g)

SELECT 'North Pole', geography::STGeomFromText('POINT(0 90)',104001)

UNION ALL

SELECT 'South Pole', geography::STGeomFromText('POINT(0 -90)',104001)

UNION ALL

SELECT 'Map Center', geography::STGeomFromText('POINT(0 0)',104001)

UNION ALL

SELECT 'Map Left', geography::STGeomFromText('POINT(-179.999 0)',104001)

UNION ALL

SELECT 'Map Right', geography::STGeomFromText('POINT(179.999 0)',104001)



SELECT * FROM @SpatialTable;

Then we have a very nice picture of Big Dipper where you can even notice a double star Mizar-Alkor (the second one on the handle):


Hopefully, I did a detailed enough explanation on how to build Sky Map in SSMS, but if there are any gaps or misunderstandings, please let me know.