error found in 8.1.0.148 when reading in scripts

yzguy

RedGate Compare Engine info:
Version = 8.1.0.148
Runtime Version = v2.0.50727
(I am using the newer one, because I have to read Redgate 8.1 snapshots)

I would attach a zip file with the view script and DB info xml file I am using, but I don't think I can attach files. The text of the view is (I think it has something to do with the comments):

SET QUOTED_IDENTIFIER Off
GO
SET ANSI_NULLS ON
GO

CREATE VIEW v_focus_lists
AS
select * from symbol b
where 1=1 -- list_type_code = 'F' and end_date is null LG: 10/23/06
/*comment*/

GO

I am using this code (I get the error on the db.Register line). If I try to compare a scripts location with this view in it, I get an error.

Database db = new Database();
Options _projectOptions = Options.Default;

db.Register(@C:\DBScripts, null, _projectOptions);

This is the stack trace:


at RedGate.SQLCompare.ASTParser.StructureLexer.nextToken()
at antlr.TokenBuffer.fill(Int32 amount)
at antlr.TokenBuffer.LA(Int32 i)
at antlr.LLkParser.LA(Int32 i)
at RedGate.SQLCompare.ASTParser.StructureParser.bracketedSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.collateSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.unarySubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.bitwiseSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.divideOrMultiplySubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.addOrSubtractSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.expressionSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.expression()
at RedGate.SQLCompare.ASTParser.StructureParser.comparisonSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.isInSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.notSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.andSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.orSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.searchSubclause()
at RedGate.SQLCompare.ASTParser.StructureParser.searchCondition()
at RedGate.SQLCompare.ASTParser.StructureParser.wherePart()
at RedGate.SQLCompare.ASTParser.StructureParser.simpleSelectExpression()
at RedGate.SQLCompare.ASTParser.StructureParser.subSelectExpression()
at RedGate.SQLCompare.ASTParser.StructureParser.selectExpression()
at RedGate.SQLCompare.ASTParser.StructureParser.selectStatement()
at RedGate.SQLCompare.ASTParser.StructureParser.createViewStatement()
at RedGate.SQLCompare.ASTParser.StructureParser.createStatement()
at RedGate.SQLCompare.ASTParser.StructureParser.statementFirstInBatch()
at RedGate.SQLCompare.ASTParser.StructureParser.statementBatch()
at RedGate.SQLCompare.ASTParser.StructureParser.expr()
at RedGate.SQLCompare.Engine.DatabaseFromScriptObjectResolver.ColumnsInObject(ISqlObject obj)
at RedGate.SQLCompare.Engine.DatabaseFromScriptObjectResolver.a(SchemaObject )
at cY.a()
at RedGate.SQLCompare.Engine.Database.Register(String path, ScriptDatabaseInformation dbinfo, Options options)[/quote]

Brian Donahue

Hi Mark,

Thanks for your post. Can you please let us know the exception type, eg do you get a "NullReferenceException"? Off the top of my head, I would suggest that there is a problem in the script containing the "symbol" object that is preventing it from being registered.

You could try running the SQL Compare UI and checking for warnings when trying to compare the scripts folder. You could access the list of warnings programatically, but this may take a bit more time.

yzguy

The error is :

antlr.TokenStreamRecognitionException was unhandled
Message="unexpected char: 0xFFFF"
Source="RedGate.SQLCompare.ASTParser"

It has nothing to do with the objects that are actually in the script, the error is in the reading in of the script. This is not my actual view, I just deleted all of the non essential stuff so that the error was easily reproduceable.

I have removed even more from the script and am still able to reproduce it with this script:

SET QUOTED_IDENTIFIER Off
GO
SET ANSI_NULLS ON
GO

CREATE VIEW v_focus_lists AS
select * from s -- list_type_code = 'F' and end_date is null LG: 10/23/06
/**/

GO

Except this time I went further to determine that if I remove the /**/ line, or if I remove one of the spaces between the 'F' and the 'and' (there are 2 spaces there, which don't show up in message boards), or if I remove the date in the comment, it does not error. There is some cobination here that is causing the parser to throw an error (unexpected char: 0xFFFF)


I can not easilly run the sql compare UI on this, as comparing to a script folder is only an option in the pro version, and I do not have the pro version of SQL compare. I expect this to be a bug in the parser, and if you look at the stack trace, you can see which function is actually generating the error.

Brian Donahue

There must be something about the file, for instance, corruption in the raw bytes that suggest a Unicode continuation followed by invalid data.

Can you copy and paste the script into a brand-new file and see if you have the same problem?

yzguy

I have tried that (using notepad), and get the same result. It is something about this string of chars.

if you want I can email you the file. I'd post it, but I don't think this board allows attachments.

yzguy

Ok I can give exact steps to reproduce.

Using RegGate.SqlCompare.Engine
Version 8.1.0.148
Runtuime Version v2.0.50727
(downloaded from this forum)

Copy this sql:

SET QUOTED_IDENTIFIER Off
GO
SET ANSI_NULLS ON
GO

CREATE VIEW v_focus_lists AS
select * from s -- list_type_code = 'F' and end_date is null LG: 10/23/06
/**/

GO

Create these directories:
C:\DBScripts\Views

Create a new text file in the Views dir, and name it "dbo.v_focus_lists.sql"

Open in notepad and paste in code from above.
Add an extra space (so 2 total) between the "'F'" and the "and"
Add an extra space (2 total) between the "s" and the "--"

Save the file

Then run this code:

using RedGate.SQLCompare.Engine;
using RedGate.SQLCompare.Engine.ReadFromFolder;


private void test()
{

Database db = new Database();
string scriptLocation = "c:\\DBScripts\\";

ScriptDatabaseInformation info = new ScriptDatabaseInformation();
FolderDataSource folder;
folder = new FolderDataSource(scriptLocation);
// Read the schema from scripts folder
db.Register(scriptLocation, folder.DatabaseInformation, Options.Default);



}

Brian Donahue

Can you try getting SQL Compare from the latest cumulative patch and referencing your libraries from there? This works for me. RedGate.SQLCompare.Engine.dll version 8.1.2.29.

http://www.red-gate.com/messageboard/vi ... php?t=9860

yzguy

that link updated my RedGate.SQLCompare.Engine.dll to:
Version 8.1.2.37

and is still doing the same thing.

I don't know if it matters but the other redgate dll's I have included are:

RedGate.Shared.SQL version 8.0.0.21
RedGate.Shared.Utils version 8.0.0.21

Brian Donahue

I'm sorry, but your reproduction doesn't work for me. The database registers fine. What am I missing?

yzguy

I started from scratch.

New project using VS Team System 2008 Database Edition Version 9.0.30729.1
Visual C#
Windows
Windows Forms Application


Add reference to:
C:\Program Files\Red Gate\SQL Comparison SDK 8\Assemblies\SQL Compare\RedGate.Shared.SQL.dll (8.0.0.21)
C:\Program Files\Red Gate\SQL Comparison SDK 8\Assemblies\SQL Compare\RedGate.Shared.Utils.dll (8.0.0.21)
C:\Program Files\Red Gate\SQL Compare 8\RedGate.SQLCompare.Engine.dll (8.1.2.37)

I noticed that I have this copy of the same dll but this one is an older version: (I'm not using this one, I just noticed it)
C:\Program Files\Red Gate\SQL Comparison SDK 8\Assemblies\SQL Compare\RedGate.SQLCompare.Engine.dll


In form1 add using statements

using RedGate.SQLCompare.Engine;
using RedGate.SQLCompare.Engine.ReadFromFolder;

In form load put this code:

Database db = new Database();
string scriptLocation = "c:\\DBScripts\\";

ScriptDatabaseInformation info = new ScriptDatabaseInformation();
FolderDataSource folder;
folder = new FolderDataSource(scriptLocation);
// Read the schema from scripts folder
db.Register(scriptLocation, folder.DatabaseInformation, Options.Default);

Copy this text:

SET QUOTED_IDENTIFIER Off
GO
SET ANSI_NULLS ON
GO

CREATE VIEW v_focus_lists
AS
select * from symbol b
where 1=1 -- list_type_code = 'F' and end_date is null LG: 10/23/06
/*comment*/

GO

Create the directory c:\DBScripts\views then create a new text file (I used notepad) named dbo.v_focus_lists.sql in the views dir and in it paste the above text, then add an extra space between the "'F'" and the "and" and between the "s" and the "--"

Save the file, and run the app.

I still get this error on the db.Register line:

"unexpected char: 0xFFFF"


I also moved these files locally to the project, changed the reference to the local copy, and zipped up the project and the script so that it can just be unzipped and run (assuming the pc has a license for the redgate products). Just unzip it to your C drive and it will create a RedgateTestApp dir with the script, dlls and the .net project in it.

I don't have a way to post an attachment here, and I don't have an easy way to post something publicly available from work, so when I get home tonight I'll upload the zip and post a link to it here.

edit-
here is a link to the zip file containing the project and the script:
http://www.1bad69.com/redgatetestapp.zip

Brian Donahue

Thanks for the files. I see that there is a problem, but I'm still looking for the cause. I think you have caused some odd condition where the SQL Parser thinks there is a character not valid for a UTF-8 file. This is a weird one!

yzguy

yeah, I'm pretty sure it is a specific string of chars that is throwing the parser off. I know I can fix this one example, but I don't know the exact pattern that is causing this (in order to look for it in the future), and the view that we happened to find this on, is code from a vendor that we really don't update.

Brian Donahue

Hi,

I think I am coming to the conclusion that I can't work out what the problem is and I'll have to get another pair of eyes on it.

• delete space from the comment and the comment identifier (-- and l), the database registers.
• put the space back in, sitll registers. It definitely should not, considering I put the file back the way it was!

I'd still say that there must be some rubbish in the file at this point, but probably need an expert at text file formats to help work it out.

yzguy

I don't think the file is corrupted, as I have repeated this many times, using a whole new file and pasting in text to notepad.

that is surprising that you can remove, then put back the space, and see it register correctly. I have done that, and mine goes back to not registering.

Chris Auckland

I'm pleased to inform you that we now have a patch version of SQL Compare that fixes this issue.

If you would like to try the patch, you can download it from here

Let me know if it fixes the issue or not.

yzguy

I had to add references to
RedGate.SQLCompare.Rewriter.dll
and
RedGate.SQLCompare.ASTParser.dll

but, now it works!

Thanks,
Mark