Showing posts with label Administration. Show all posts
Showing posts with label Administration. Show all posts

Wednesday, February 12, 2020

Use of Uniqueidentifier in Persisted calculated column.

This post is for you in case you decide to use Uniqueidentifier column in your table and then you think about including it into a Persisted calculated column.
You also might see it useful if you like weird or funny SQL Server behavior.

At first, here is the SQL Server version I have that problem on.
After a while Microsoft might fix that bug.
Microsoft SQL Server 2016 (SP2-CU11) (KB4527378) - 13.0.5598.27 (X64) 

To demonstrate the problem in the most simple way you can run following script:
DROP TABLE IF EXISTS #tbl_Error_Test;
GO
CREATE TABLE #tbl_Error_Test(
       GuidOne UNIQUEIDENTIFIER,
       GuidTwo as CAST(GuidOne as CHAR(36)) PERSISTED
  );
GO
INSERT INTO #tbl_Error_Test(GuidOne) VALUES (NewID()), (NewID());
GO
SELECT * FROM #tbl_Error_Test;

GO

It will return something like this:

As you can see, columns GuidOne and GuidTwo are different.
Moreover, if you run following command you will get very unpleasant error:
DBCC CHECKTABLE('#tbl_Error_Test') WITH NO_INFOMSGS, EXTENDED_LOGICAL_CHECKS;

Msg 2537, Level 16, State 106, Line 152
Table error: object ID -1485216446, index ID 0, partition ID 6052840780930088960, alloc unit ID 2738196559872000000 (type In-row data), page (4:1079768), row 0. The record check (valid computed column) failed. The values are 2 and 0.
Msg 2537, Level 16, State 106, Line 152
Table error: object ID -1485216446, index ID 0, partition ID 6052840780930088960, alloc unit ID 2738196559872000000 (type In-row data), page (4:1079768), row 1. The record check (valid computed column) failed. The values are 2 and 0.
CHECKTABLE found 0 allocation errors and 2 consistency errors in table '#tbl_Error_Test_____________________________________________________________________________________________________000000001926' (object ID -1485216446).

repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKTABLE (tempdb.dbo.#tbl_Error_Test_____________________________________________________________________________________________________000000001926).

If you try to check what the correct value should be, it won't make any good:
SELECT GuidOne, GuidThree = CAST(GuidOne as CHAR(36))
FROM #tbl_Error_Test;


The SQL Server is smart enough to recognize same function and instead of calculating the value it will return you the value stored in the table.

You can only do a trick if you change the function and convert Guid to 37 chars, instead of 36:
SELECT GuidOne, GuidFour = CAST(GuidOne as CHAR(37))

FROM #tbl_Error_Test;

Now you got the right result, but how to fix it in the table?

It is not easy, but possible. You just have to replace Guid column by itself, but from another table:
UPDATE G1 SET GuidOne = G2.GuidOne
FROM #tbl_Error_Test as G1
INNER JOIN #tbl_Error_Test as G2
       ON G1.GuidOne = G2.GuidOne;
GO
SELECT * FROM #tbl_Error_Test;

GO


It is fixed now and it also fixes DBCC CHECKDB error.

If you see the same behavior on your SQL Server you can vote for that bug to be fixed at Microsoft site:
https://feedback.azure.com/forums/908035-sql-server/suggestions/39694663-use-of-guid-column-in-persisted-calculated-column

Friday, December 20, 2019

Recovering from Severe Database corruption error 9100

In my previous post I outlined very dangerous SQL Server problem, caused by usually not very harmful commands "PARSE" and "TRY_PARSE": Having "NaN" value for REAL and FLOAT producing severe error.

This post will be about a major issue it causes and on how to fight it.

Will start from generating the problem.

Attention: Do not run that in production!!!
Here is a script to generate the issue:
DROP TABLE IF EXISTS tbl_Real_Test;
GO
CREATE TABLE tbl_Real_Test (ID UNIQUEIDENTIFIER, R REAL, F FLOAT);
GO
INSERT INTO tbl_Real_Test (ID, R, F)
SELECT NewID(), TRY_PARSE(r as REAL), TRY_PARSE(f as FLOAT) 
FROM (VALUES
       ('0','434534776544')
       ,('0.0000323', 'NaN')
       ,('3.3881317890172E+17','')
       ,('','000000000000x000000000000')
       ,('NULL','ABCDEFG')
       ,('Null','--------------------')
       ,('null','!!!!!!!!!!!!!!!!!!!!')
       ,('ABC','345435467655665676545')
       ,('NaN','5.3881317890172E+47')
       ,('Nan','0.0000000000345')
       ,('nan','434.34543543')
) as R(r,f);
GO
SELECT * FROM tbl_Real_Test
WHERE R Is not Null ;

GO

That script will generate following error:
Msg 9100, Level 23, State 2, Line 57
Possible index corruption detected. Run DBCC CHECKDB.

Troubleshooting:

You might try to run "DBCC CHECKDB" against that database like this:
DBCC CHECKDB('Slava_Temp_Test');

DBCC CHECKDB resulted couple of errors and besides of that it made an error Dump into an ERRORLOG file and created four other DUMP files:
Msg 2570, Level 16, State 3, Line 62
Page (1:16200), slot 1 in object ID 2002106173, index ID 0, partition ID 72057594042843136, alloc unit ID 72057594048479232 (type "In-row data"). Column "F" value is out of range for data type "float".  Update column to a legal value.
Msg 2570, Level 16, State 3, Line 62
Page (1:16200), slot 8 in object ID 2002106173, index ID 0, partition ID 72057594042843136, alloc unit ID 72057594048479232 (type "In-row data"). Column "R" value is out of range for data type "real".  Update column to a legal value.
DBCC results for 'tbl_Real_Test'.
There are 9 rows in 1 pages for object "tbl_Real_Test".

CHECKDB found 0 allocation errors and 2 consistency errors in table 'tbl_Real_Test' (object ID 2002106173).

Fixing attempt:

To fix the issue you might try following commands
DBCC CHECKDB ('Slava_Temp_Test' , REPAIR_REBUILD) WITH ALL_ERRORMSGS;
GO
DBCC CHECKDB ('Slava_Temp_Test' , REPAIR_ALLOW_DATA_LOSS) WITH ALL_ERRORMSGS;

GO

However, in order to perform the fix DBCC CHECKDB requires the database to be in a single user mode:
Msg 7919, Level 16, State 3, Line 141
Repair statement not processed. Database needs to be in single user mode.

The problem is that it might be not possible to switch database into a single user mode, because it is part of an Availability Group and experiencing very heavy user activity. The use of conventional SQL Server tool would cause disruption in Production for who knows for how much time.

Solution:

To solve that problem we do not have to switch database in single mode or restore it from a previous backup to eliminate the issue. You just run an update script over your problematic table and columns to replace bogus values, which SQL Server can't interpret correctly, by NULLs:
UPDATE tbl_Real_Test
SET R = CASE CAST(R as BINARY(4))
       WHEN CAST( PARSE('NaN' as REAL) as BINARY(4))
       THEN Null Else R END
       , F = CASE CAST(F as BINARY(8))
       WHEN CAST( PARSE('NaN' as FLOAT) as BINARY(8))
       THEN Null Else F END;
GO
SELECT * FROM tbl_Real_Test
WHERE R Is not Null OR F Is not Null;

GO

The result of this query should be like this:

Disclaimer and Advertisement:

There are probably a lot of other cases, which might cause "Database corruption error 9100" and if that particular solution does not apply to your environment you can contact me for the resolution.


Tuesday, November 26, 2019

ERRORLOG flooded with "Service Broker endpoint" messages.

During system cleanup I hit an issue when SQL Server ERRORLOG file with a size of about 10 GB was 99.9% flooded by messages like this:
The Service Broker endpoint is in disabled or stopped state.

There were millions of them and every 15 minutes were adding the new ones.

I've searched the internet and got a not proven suggestion to create a Service Broker Endpoint.
Fortunately enough, the organization I'm servicing right now has Microsoft Service support hours assigned to a SQL DBA group. I've opened a ticket and got an answer from Microsoft.
MS representative referred me an online article: Service Broker with Always On Availability Groups (SQL Server) That article clearly states that all machines with Availability Groups must have
the Service Broker endpoint exists and is correctly configured.

So, the solution to stop that avalanche of useless messages will be very simple: just apply following script on a SQL Server, where you have Availability Groups and experiencing the issue:
CREATE ENDPOINT [SSBEndpoint] 
    STATE = STARTED 
    AS TCP  (LISTENER_PORT = 4022, LISTENER_IP = ALL ) 

    FOR SERVICE_BROKER (AUTHENTICATION = WINDOWS);

To prove that action is successful you can run following script:
SELECT name, protocol_desc, type_desc, state_desc, port
FROM sys.tcp_endpoints

WHERE type_desc = 'SERVICE_BROKER';


Also you can search in the ERRORLOG file that your Service Broker is Listening and does not produce garbage messages anymore:
EXEC sys.xp_readerrorlog 0, 1, N'listening';



Friday, November 22, 2019

Deleting a SQL Agent Job originated from an MSX server

Hit very weird situation when tried to remove jobs running on remote-target servers in multi-server managed environment.

First what I've tried is to unassign target servers from multi-server job on MSX server, however, it did not delete jobs on some targets.

When I've tried to delete them manually via SSMS I've got following error:

TITLE: Microsoft SQL Server Management Studio
------------------------------
Drop failed for Job 'CPU_Usage_By_Database'.  (Microsoft.SqlServer.Smo)
For help, click: https://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=15.0.18131.0+((SSMS_Rel).190606-1032)&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Drop+Job&LinkId=20476
------------------------------
ADDITIONAL INFORMATION:
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
------------------------------
Cannot add, update, or delete a job (or its steps or schedules) that originated from an MSX server. (Microsoft SQL Server, Error: 14274)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&ProdVer=12.00.5223&EvtSrc=MSSQLServer&EvtID=14274&LinkId=20476
------------------------------

When I tried to follow help links, provided by newest SSMS, they led me to Microsoft's advertisement portal.

The second I've tried was manual deletion of that job by a script:

EXEC msdb.dbo.sp_delete_job @job_name=N'My_MSX_Created_Job', @delete_unused_schedule=1;

It was not successful and gave me following short error without Microsoft adds:
Msg 14274, Level 16, State 1, Procedure sp_delete_job, Line 91 [Batch Start Line 59]

Cannot add, update, or delete a job (or its steps or schedules) that originated from an MSX server.

I've looked in "sysjobs" table and saw there a column "originating_server_id", which is referencing to "sysoriginatingservers_view" view, which has "originating_server_id" value always to be zero for the local server.
So the fixing script is as easy as this:
UPDATE msdb..sysjobs
SET originating_server_id = 0
WHERE name=N'<Your Job Name>';
GO
EXEC msdb.dbo.sp_delete_job
       @job_name=N'<Your Job Name>',
       @delete_unused_schedule=1;

GO


Wednesday, September 25, 2019

SQL Server CEIP service

Very recently, by monitoring SQL Server activity found that some service creates Extended Events and then drop them.
The user, who run these traces is "SQLTELEMETRY" and an application is "SQL Server CEIP service (MSSQLSERVER)"

Usually I missed that, because I look at running tasks only when there is a problem and if any running task does not generates any extra CPU activity and does not cause any current issues I just ignore it while hunting for the real problem.
But recently I've started monitoring DDL events activity and one of the servers showed me that "CEIP service for Sql server" produces a lot of DDL event on a regular base.

I do not like any extra noise in my environment and started to hunt that issue.
That Microsoft article describes how "How the Customer Experience Improvement Program (CEIP) helps Microsoft identify ways to make our software better." 

In other words that service collects information within your SQL Server and sends it to Microsoft. Even though Microsoft claims that it does not collect and send passwords and certificates it still admits that some Customer Content can be sent to Microsoft.
That means if you have any PII - Personal Identifiable Information, which can be Social Security Numbers, Dates of Birth, Names, addresses, Credit Card numbers, financial information can easily be transferred outside of your server and being unprotected.

How have you got that scary service? 

Whoever installed SQL Server on that box once allowed that service to exist and send data to Microsoft.

How to stop/disable that service?

The Books Online article describes how you can turn sending that information to Microsoft off, but as paranoid DBA I'd prefer to disable the service completely.
You can find it in the list of services on the server:

Just open it, stop and select "Disabled" in the "Startup Type"

And keep your customers' data protected.



Friday, September 6, 2019

SSMS Feature generates Severity 20 Error in the Errorlog.

Yesterday I've needed to use Dedicated Administrator Connection (DAC) once in a while, and because I have all kinds of notifications in my system, I immediately got an "Severity 20" alert.

As you probably know, Severity 20 Errors "Indicate system problems and are fatal errors" (See books online: https://docs.microsoft.com/en-us/sql/relational-databases/errors-events/database-engine-error-severities?view=sql-server-2017)

Even though "Severity 20" does not indicate any problems with data and belong only to a user process it is still worth to investigate the problem.

I looked at my Log file and found following record:
Error: 17810, Severity: 20, State: 2. 
Could not connect because the maximum number of '1' dedicated administrator connections already exists. 
Before a new connection can be made, the existing dedicated administrator connection must be dropped, either by logging off or ending the process. [CLIENT: 192.168.234.54]

I knew that I'm using the DAC on that server right now, but it was a surprise for me that "somebody else" is also trying to connect to the same SQL server via DAC.

I've set the simplest extended event session (script is below) to monitor the cause of the problem.
CREATE EVENT SESSION [Tracking DAC Errors] ON SERVER
ADD EVENT sqlserver.error_reported(
    WHERE ([category]=(4) AND [error_number]=(17810) AND [severity]=(20))),
ADD EVENT sqlserver.errorlog_written
ADD TARGET package0.ring_buffer
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=OFF)

GO

And by doing that I discovered a little "Aero" SSMS window, which is driven by IntelliSense, and usually appears for a fraction of a second when you connect to a new server or a database and then disappears. It reads the schema of current database to provide you suggestions. However, I noticed this time that this small popup window did not disappear and it also had an error.

I've tried to hit a "Restart" button and immediately got an error in my extended event's "Live View" window:

As you can see from the screenshot IntelliSense was trying to read the schema of my current database using the same connection string I used to connect to the server, but because I've used DAC, which allows you to have only one that kind of connection IntelliSense got an error.

I've tried to turn IntelliSense off, however it hasn't fixed the issue. Behind the seen SSMS still tried to pull some information from a SQL Server using the same connection string to as my connection in a tab.

At first, my thought was that it might be a bug, but after little thinking I realized that SSMS is not so smart and can't determine correct connection string to the server you want and for simplicity just uses same connection string, which was already provided.
It would be nice though, if SSMS realized that I'm using DAC and did not try to connet to the Server itself and not producing unnecessary error.

What can we learn from that?
If you are at the point when you have to use DAC to connect to your server, you are on your own and SSMS interface become useless. All its reports and GUI features are unavailable and only pure T-SQL can help you.
I'm glad I'm using my own set of scripts embedded into SSMS shortcuts, if you want to check it out, here is the link to a downloadable archive: https://drive.google.com/open?id=0B5yWoyX1eEWqZ3FJUnNHZm80bzQ