Get the SSDT projects to compile with parameters from MSBuild

The problem is that with the new .sqlproj and the SSDT\Microsoft.Data.Tools.Schema.SqlTasks.targets there’s no built-in way to pass the database name or even a connection string when building the project via MSBuild — which is a critical part of our continuous integration builds. The project I’m working on these days has 7 or 8 teams working in up to twice that many branches, and all those branches need CI builds, every one of which deploys the database project and validates it. Since the branches change at least once a week, it’s way too much work to run around modifying publish.xml files to change database names every time we create a new branch (which we need to do to avoid the builds deploying over the top of each other).

With the old .dbproj format, there was a SQLDeploy task called in the TeamData\Microsoft.Data.Schema.SqlTasks.targets build target file which included a whole bunch of variables that could be overridden on the command line, so we could pass TargetDatabase and TargetConnectionString as MSBuild arguments, and then, to be able to compile the whole solution and still call the Deploy target, we added this to the project file:

In our workflow, we redefine MSBuildArguments in the workflow, and now we can msbuild the whole solution and the database will be deployed:

But that doesn’t work with the new SSDT project type.

First of all, they don’t deploy with all their dependencies, instead, they have to be published. It’s basically the same thing, with a different name. But the SqlPublish task requires all the parameters to be in an xml file, and there’s no build properties we can override, because the properties are hiding in that publish.xml file that doesn’t get tokenized

I’ve spent the last couple of days figuring out a work around, so I figured I should blog it up here and help the next guy. The process is not simple. The bottom line is that I haven’t found a way to get the SQLPublish task to take it’s values from anywhere except a publish xml file, so the solution I came up with was to rewrite the publish file using the XDT transform tasks defined for Web.config transforms.

Web.Config Transforms, on random XML files

The cool thing is, there’s actually a ParameterizeTransformXml task which allows you to define your transform as a string in the build file.

In web projects, that task is used to replace connection strings (to hide them in web packages), but we can use it to replace the database name in our publish.xml. In fact, I can actually add the same properties to the build that we used to have with the old project format (and which we’ll define on the command-line, exactly the way we did before). We put some default properties for TargetDatabaseName and TargetConnectionString in our Debug.publish.xml and our CI.publish.xml and then we just replace them during the build.

It’s a lot more complicated than what we had to do previously, partly because we need to define the SqlPublishProfilePath for the Publish task, but we have to use CallTarget to call the Publish target (not Deploy this time), which doesn’t support passing properties, nor does the target you call inherit properties that are defined in your scope. This means we need to define the SqlPublishProfilePath property in the BeforePublish target which the publish target depends on (the “dependson” relationship inherits defined properties).

But the real work is actually setting up the TransformPublishXml property with the right XML to replace the nodes with the properties from the command-line arguments, and then actually calling the task. Since we imported the task before, we just need a property group to define our variables with default values, and then a BeforeBuild target to actually call the ParameterizeTransformXml:

So all you have to do is put those three blocks of XML at the bottom of your .sqlproj file, and then call msbuild with /p:TargetDatabase=DBName;TargetConnectionString="Data Source=DBServer;User ID=sa;Password=password";DBDeployOnbuild=True to get the database project to build and deploy to the database you want.

If you’ve got questions, post ‘em — I’m writing this at 1:30 in the morning so I’m not at my most lucid

I’m currently trying to figure out what database tables I need to be concerned with, and I just have to vent, because this system is the worst mess I’ve ever seen.

• There’s one database supporting five or more applications…
• There are 87 tables (with names like tblAQ_DcSs, tblAQ_SwNw, tblSFM, and tblSW_PWSOS, tblTestCaseTestLayout )
• There are three duplicate user tables: tblPeopleLookup, tblUser, and tblUser3 — tblUser2 is a view onto an external user database which is what is supposedly being used … and apparently, tblPeopleLookup is some sort of mapping from tblUser2 to tblUser3 … and tblUser1 is the original user table. I don’t know why these are all still here — I can only hope none of these others are still being used.
• There are 144 stored procedures (with names like sp_Fix, sp_Fix2, sp_Fix3, spLeftToTestMulti, spLeftToTestMulti2, spLeftToTestMulti3, sp_Whatever, and the awesome spTestCaseTestLayoutTestsUpdate, spTestCaseTestLayoutTestsSelect, etc.)

• There are no Foreign Keys. Yeah. None.

Technically, there are lots of foreign keys — it’s just that none of them are declared as such, so there’s no referential integrity (did I mention that there’s an access database floating around out there with linked tables and a hard-coded login which the end-users pass around to each other so they can insert data into some of the tables by hand because the original developers didn’t get around to writing this management app that I’ve been asked to write now?)

You can tell that some of the columns should be Foreign Keys, because obviously a column in a “tblReq_Tag” table named “Feature_ID” must be an external lookup of some sort … but there’s 86 other tables … and at least two of them have Primary Keys called “Feature_ID” ...

So, I’m spending a lot of time searching the source code and the 144 stored procedures … An astonishing number of these stored procedures involve cursors and multiple nested case statements. I just picked one at random which I thought sounded simple: spEnterGroupResults ... it’s about 150 lines of SQL, and it uses a single cursor variable “crsUnit” which it redefines three separate times onto three different queries which it iterates over. Each of these queries involves joins onto nested subqueries, and I count myself lucky because the tricky part is actually enclosed in a transaction, and at least this one isn’t doing all of that just to dynamically generate a further SQL query to execute.

So yeah, I’m literally looking through source code to try to understand the database design. The problem is that there are more than five different applications, each using slightly different technologies.

• One of them which has never been migrated from classic ASP ... with the business logic written entirely in Javascript, and the data handling performed entirely by sending huge XML files back and forth to a “do all” webservice.
• One of them was written in VB.NET in VS 2003, and has never been upgraded.

• The rest are in C# — with most in VS 2005, and at least one in VS 2008 and C# 3.0 — some are Asp.Net, some are rich client …

The tables I’m most concerned with right now (for this app) have some ahem ... impressive design decisions of their own. Of the 8 tables that I’m looking at directly (I think these are the only ones I need to modify as part of this app), five of them have multi-column primary keys that involve more than half the columns in the table, including columns which are, in fact, unconstrained foreign keys. And there are so far 5 foreign key looking columns which I haven’t been able to find the primary key column for …

Edit: Oh yeah, and half of these tables have columns like [Enabled] [char](1) NULL … That’s a boolean value folks, stored in the database as a y or a n … and it’s nullable even though a null (or any value other than y or n, really) will most likely blow up some code somewhere. And no, there’s no script constraint or trigger to ensure that this doesn’t happen (I checked). For extra fun, the other half of the tables use ‘bit’ columns for things like this — because they were written after I started working with this team (on a different project) and happened across one of these char columns during our one and only code review ever and wondered aloud why we needed to pretend it was still 1992. Why they just switched, without changing the others, I’ll never know…