Showing posts with label Database. Show all posts
Showing posts with label Database. Show all posts

Things I Keep In My Toolbox

This thread is my ongoing attempt to remind myself of the tools I keep somewhere in my toolbox.

IDE Tools

Web

Cryptography

Design Tools

Development Other

Windows General Tools

Mac General Tools

Running Database Project Deploy Script In SQL Management Studio

Visual Studio Database Projects is a great way to manage and maintain your SQL server database schema.  The projects offer compile time checking just like any other code type project would.  When you build build the project you have the options of either creating a deployment script or a script plus deploy it to a given database instance.  If you create the script, there is one caveat that I find myself forgetting the first time I run it via SQL Management Studio, setting the SQL command mode.  This is necessary to resolve the lines that start with “:”, such as “:setvar”.  To enable the script to run successfully, open the menu Tools ^ Options.  Click the “Query Execution” in the list.  Check the “By default, open new queries in SQLCMD mode”.  Open a new query window and run your script.

temp

Executing Stored Procedure With NHibernate

There are two reasons why you might want to execute a stored procedure using NHibernate.  The first would be for your normal CRUD activities which can be found here, Populating Entities From Stored Procedures With NHibernate.  I’d like to focus on the other scenario you might find yourself needing to execute a procedure, like running a complex report or some query that is better performed in a procedure.  The real benefit of NHibernate is that you’re not coding to an implementation but to a set of interfaces.  This buys you the blissful ignorance of the underlying database vendor, see also my post on NHibernate And Paging Results.

Executing a procedure requires two main items, a connection and a command object.  The connection comes from the session, and the command comes from the connection’s method CreateCommand.  Below shows a very straightforward way of executing a procedure.  Notice that the connection and command objects are just interfaces from System.Data namespace.

List<SomeDto> list = new List<SomeDto>();

using (ISession session = NHSessionManager.OpenSession())
{
// use the ISession to get the database connection
IDbConnection conn = session.Connection;

// use the connection object to get the database command object
IDbCommand cmd = conn.CreateCommand();

// set the command text to the procedure's name and set the command type
cmd.CommandText = "dbo.MyAwesomeProcdure";
cmd.CommandType = CommandType.StoredProcedure;

// using the command object you can create a parameter, setting it's properties accordingly
IDbDataParameter p;
p = cmd.CreateParameter();
p.DbType = DbType.Int32;
p.ParameterName = "@someParam";
p.Value = someParamValue;
cmd.Parameters.Add(p);

// data read to capture results
IDataReader reader = null;
try
{
reader = cmd.ExecuteReader(CommandBehavior.SingleResult);
while (reader.Read())
list.Add(new SomeDto
{
Name = reader.GetString(0),
Age = reader.GetInt32(1),
...
});
}
finally
{
if (reader != null)
reader.Close();
}
}

return list;

NHibernate and Paging Results

One of the most common operations in web sites that list data such as inventory items, blog posts, pictures, etc. is data paging.  If you happen to be using NHibernate for you project and want a really clean way to page and sort through your items, then read on. 

Problem:  The other day I was trying  to optimize some data paging that was using Linq on a large record set instead of paging the records at the data level.  I knew that NHibernate had the methods SetMaxResults and SetFirstResult for limiting results, but what I didn’t know is how to execute all the data retrieval, including total pages count, all in on query, especially using all the filters that the List<> query was using.  I knew where I wanted to get but was missing a few puzzle pieces.  I can’t take all the credit for the solution below, but I will consolidate 2 great ideas into one post.*

Solution:  Using NHibernate, we will utilize the following concepts/methods:

  • CreateMulitCriteria() – a way to execute more than one ICriteria in one database call
  • CriteriaTransformer.Clone() – clones the a given criteria object
  • SetMaxResults – limits result set
  • SetFirstResult – takes results from a starting location

Since paging is pretty common we can make use of Generics to alleviate repetitive coding, and Extensions to append functionality to the existing ISession.  If you don’t want to use Extension, you can write a static helper class.

Define a wrapper for our paged results:

public class PagedResult<TModel>
{
public IList<TModel> Items { get; protected set; }

public long TotalItems { get; protected set; }

public PagedResult()
{
this.Items = new List<TModel>();
}
public PagedResult(IList<TModel> items, long totalItems)
{
Items = items;
TotalItems = totalItems;
}
}
Define the Extension method to do the work.
public static class NHibernateExtensions
{
public static PagedResult<TModel> CreatePagedResult<TModel>(this ISession session, ICriteria criteria, int startIndex, int pageLength)
{
// clone the criteria for page count
ICriteria countCriteria = CriteriaTransformer.Clone(criteria)
.SetProjection(NHibernate.Criterion.Projections.RowCountInt64());
countCriteria.ClearOrders(); // clear order by

// set paging on base criteria
criteria.SetFirstResult(startIndex)
.SetMaxResults(pageLength);

// get all data in one execution
var multiCriteria = session.CreateMultiCriteria()
.Add(criteria)
.Add(countCriteria);
// execute both queries
IList results = multiCriteria.List();

var pagedResult = new PagedResult<TModel>(
((IList<TModel>)results[0]), // get paged items
(long)((IList)results[1])[0]); // get total records

return pagedResult;
}
}
First we clone the ICriteria which represents our parent query to get our list data.  The parent ICriteria holds all the filter conditions, grouping, etc. that is going to be applied in determining the results.  After cloning this, we apply a projection that generates the count query, and we also clear the ordering clause since it’s not needed.  Next we take advantage of the paging capabilities in the ICriteria interface, SetFirstResult and SetMaxResults.  This is similar to using TSQL’s ROW_NUMBER function, or MySql LIMIT.  In order to execute the queries in one go, we use the ISession.CreateMulitCriteria() method which prepares us for the 2 query execution.  Executing this will return our paged items and secondly our total records count.  In the case of our total records we’re only concerned with the result in the first row hence the “((IList)results[1])[0]”.
 
Here is a capture from a NHibernate Profiler.  I had to remove a lot of the columns so the important pieces are noticeable, and highlighted some of the key items.  Notice the top select is for the list of items, and the bottom select is for the total records.  Since this is SQL server 2005+ it’s making use of the ROW_NUMBER function mentioned above.
nhibernate_profile_capture

* See: stackoverflow post’s comment by Geir-Tore Lindsve, Stefan Sedich’s Blog post

How To Deal With Unix Files LF And BULK INSERT

Preface: I was trying to import the GeoNames data into SQL Server and was going crazy trying to do it.  The most frustrating part was the BULK INSERT and the EOF.  I found 2 relatively good posts on how-to’s, but neither of them got me all the way.  When you download the countries from GeoNames, it has unicode data and Unix file format which uses a end-of-line character line feed (LF or \n).  In order to import it into SQL Server, you first need to convert it to utf16 encoding.  A good post by Johannes Kebeck uses NAnt to do this conversion and works very well.  The only issue with his post is that I couldn’t get the import wizard to work at all.  Meanwhile I found another good post by Ed Katibah using a different approach taking advantage EditPad (like Notepad++) and BULK INSERT.  He used EditPad to convert the file, but every time I tried the application would fail 75% of the way through with memory issues; I had a little better luck with Notepad++ but instead of creating errors it never processed the whole file.  I found that using NAnt to convert the file was the fastest and most systematic approach.  This leads use to the issue of BULK INSERT, the topic of the post.

Problem: I have this Unix data file where the data fields are separated by tabs (\t) and the EOF is a line feed (LF).  BULK INSERT can be the most finicky operation.  So for this file, I attempted the following command.

BULK
INSERT dbo.GeoNames
FROM 'F:\db\Import Data\Converted\allCountries.txt'
WITH
(
DATAFILETYPE = 'widechar', -- for unicode
FIELDTERMINATOR = '\t', -- tab
ROWTERMINATOR = '\n' -- LF
)

When I ran this I got the error shown below.  At this point I started searching for answers because it doesn’t get any more simple than this.  One post said I had to convert to Windows file format which uses CR + LF (carriage return + line feed or \r\n).  I kind of got this working but I couldn’t systematically convert my whole file.

Msg 4864, Level 16, State 1, Line 1
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 19 (modification_date).

Solution:  The online documentation for BULK INSERT had a snippet of code at the bottom of the page which talked about Unix file format and importing.  Below is basically that snippet of code.  The row terminator is set to CHAR(10) which is LF (\n), the exact same row terminator definition as the above insert statement.  For some reason this works, but the above doesn’t, go figure.

DECLARE @bulk_cmd varchar(1000)
SET @bulk_cmd = 'BULK INSERT dbo.GeoNames
FROM '
'F:\db\Import Data\Converted\allCountries.txt''
WITH (ROWTERMINATOR = '
'' + CHAR(10) + ''')'
EXEC(@bulk_cmd)

Reporting Services Alternating Row Colors

Reporting Services is a easy way to produce good reports with little effort.  A common tablature based report style is to have rows of alternating color.  This eases the readers eyes when it comes to visually scanning and extracting data row by row.  In order to alternate the row color, you need to give the BackgroudColor property of the row a formula to determine which color to apply.  Here is a basic formula to switch colors between white and gray.

=iif(RowNumber(Nothing) Mod 2, "White", "Gainsboro")

The UI property window would look like the image shown below, where the red arrow points to the tables highlighted row.

rs_row_alt_color

Using master dbschema File To Resolve Login References

If you’re using MS SQL Server, Visual Studio’s database projects are a good way to manage your database objects.  After importing your database schema, if you working with existing databases, or creating from scratch, you will typically need a reference to the master.dbschema.  A typical error due to an unresolved server login reference will say “Error TSD03006: User: [someuser] has an unresolved reference to Login [someuser].”.

The schema file is just a xml file of the server object meta data, this is how it can resolve references when referenced by other database projects in your solution.  The VS Database Project comes with master.dbschema files that you can copy and modified if needed.  The different version of SQL server schema files can be found in “C:\Program Files\Microsoft Visual Studio 9.0\VSTSDB\Extensions\SqlServer\”, if “C:\Program Files\” is your installed path of VS.  I would copy the master.dbschema file that you need to another location and rename with an appropriate name so others know what it represents.  If you have server logins that you don’t want to manage through you build and deploy, adding them to this copied master schema file is the way to resolve the build references.  Below shows the an example of the element that would need to be added to the dbschema file to resolve a particular login, the added part is italicized and bold.

<DataSchemaModel FileFormatVersion="1.0" SchemaVersion="1.0" DspName="Microsoft.Data.Schema.Sql.SqlDsp.Sql100DatabaseSchemaProvider" CollationLcid="1033" CollationCaseSensitive="False">
<Header>
<CustomData Category="ModelCapability">
<Metadata Name="ModelCapability" Value="Default" />
</CustomData>
<CustomData Category="DBSchema">
<Metadata Name="DatabaseType" Value="master" />
<Metadata Name="SqlServerVersion" Value="10.00" />
<Metadata Name="Author" Value="Microsoft Corp." />
</CustomData>
<CustomData Category="DatabaseVariableLiteralValue">
<Metadata Name="Name" Value="master" />
</CustomData>
</Header>
<Model>
<Element Type="ISql90Login" Name="[someuser]">
<Property Name="DefaultLanguage" Value="us_english" />
<Property Name="IsCheckPolicyOn" Value="False" />
<Annotation Type="GloballyScoped" />
</Element>

</Model>
</DataSchemaModel>
 
Once this is added, the login in reference error will resolve correctly.  Notice that I only have the one <Element> under the <Model> node.  If your projects don’t require usage of most of the master server objects, like extended procedures, you can removed them from the <Model> node.  Doing this trimming is worth the effort, especially in the cases where your using another projects dbschema file of which only a few items are needed to resolve references.  A good article of dbchema file trimming can be found here Right sizing the master.dbschema file for better design time performance.  Only including the necessary element in the model will reduce build time for projects depending on the schema file.

VS Database Project Error TSD04105, Build Issues With Multiple Projects

Problem: You have multiple database projects in a solution, one references another, and when you go to build you get some or a lots of TSD04105 or TSD03006 errors because you have the same named item it both of your projects.  For example, Project_1.myschema.SomeDbObject and Project_2.myschema.SomeDbObject and Project_2 has a db reference to Project_1.

Solution:  I googled this for while and it appears there are/were some bugs around this issue.  Supposedly they were fixed in the GDR release but not that I can tell, see this bug article.  What you need to do is uniquely identify the objects that have the same name which reside in both of the projects.  You can give a name to you data project that is reference either when you are adding the reference or afterwards by going to the References tab in the project properties.

When adding a project reference, you define database variable which can either be a literal value or a variable name such as $(MyVariable).

add_reference

If you do it after adding the database reference, you can accomplish the above task by going to the References tab in the properties of the project.  Below shows what that screen will look like from VS2008.  To setup a database variable, enter in a name in the column “Database Variable Name” or “Database Variable Value” column.  If you check the Literal checkbox, you can only enter in the “Database Variable Value” column.

references_tab

Now the project should build since the same named objects will be uniquely identified by “Database Variable Value”.myschema.SomeDbObject.

Using Database Project Cross-Database References

Visual studio an excellent project template for database management; currently it supports project templates for 2000, 2005, and 2008 SQL Server.  Depending on the version of Visual Studio you are using, you may need to install the latest for 2008 templates here.  In this post I will describe a scenario where you might be using linkservers to reference a database that you possibly aren’t managing or do not want to add to the database solution you are working on.  Below describes the scenario I came up against and used the cross-database reference to solve it.

Scenario:  You have a database solution with various database projects.  In one of these database projects, you have some code, say in a stored procedure, that has a line “SELECT … FROM MyRemoteServer.MyRemoteDb.dbo.TableAbc A …”.  When you go to build your database project, it fails and says “… has an unresolved reference to object …”.  So, we needed a way to have a reference to this report database schema without creating a project within our solution.

Fix: A way to fix this scenario is to use a cross-database references.  This msdn article outlines some of the reasons you might want to do this http://msdn.microsoft.com/en-us/library/bb386242%28VS.80%29.aspx.  In our case we didn’t want to create a project for this remote database schema that didn’t change.  So in order to fix the references that cause the build errors we had to import the remote database into a .dbschema file using the VSDBCMD tool.  This can be found here http://msdn.microsoft.com/en-us/library/dd172135.aspx.  Once the .dbschema file is create you can add it to you solution.  Once it’s in the solution, you can go to your project, right click References and do a “Add Database Reference…”.  Choose “Database project schema” for which you will browse to the .dbschema file that was just added to solution.  Once you have added the reference, go to the project properties and view the references (see picture below).  You will need to associate a server and database variable name for this reference.  Now that you have these variables the problematic build code described in the Scenario will need to be changed to this “SELECT … FROM [$(ServerVarName)].[$(DatabaseVarName)].dbo.TableAbc A …”.  With this in place the build will succeed because the project can now resolve the remote database objects through the addition of the .dbschema file.

proj_refs

NHibernate : Object Relational Mapping Coolness

I haven’t been around every block, but I have seen many ways people have cooked up mappings between a persistent layer, typically a database, and some sort of object model, whether a business oriented domain or just a bunch of objects.  Probably the worst I’ve seen has been data classes with methods like Select(), From(), Where(), Group() and bits and pieces of dynamic sql spilled all over.  It was impossible to follow or comprehend.  I’ve seen other approaches, including myself, that is ordered, neat, and works well, but isn’t something I’d call great.  The concept of object to relation database mapping isn’t new, so when I started looking at some of the frameworks out there, I said to myself “why haven’t I been using this sooner?” One ORM in particular that caught my eye was NHibernate, which is a port from a ORM in java called Hibernate.  It’s got a bit of a steep initial learning curve but once you got this engine firing on all cylinders, damn this stuff is cool.  One of the core concepts that really caught my attention was that fact that NHibernate stresses domain driven design.  In a nut shell, forget about the storage of your data objects and focus on what you business model is and how it works; deal with the persistence later because that’s what the “mapping” part is all about.  Of course there is a bit of give and take because you might be trying to use this ORM with a legacy database system that doesn’t easily fit into a nice domain model.  Also, NHibernate doesn’t solve all the database mapping scenarios, although I would say it could follow the 80/20 rule: it will solve 80% of the most common scenarios, leaving only 20% to some other type of technological solution.

Anyway, I’m not going to give a full demo of how it works, there will be links at the end of the post that can get you started.  The following 4 items are all that is needed to get up a running:

  1. Some domain model with a mapping to a persistent layer (i.e. a database).  The mapping can be done a few different ways, but the concept is that you indicate how your objects will map to given database tables/columns.
  2. Enter the correct app.config settings.  The configuration settings tell NHibernate information like what type of database it’s talking to and how (i.e. SQL2005, MySql, Oracle, etc).
  3. Generate the session factory in you app, preferably only once such as a singleton.  The SessionFactory is in charge of generating proxy classes for you mappings.  If you get stuck by why properties have to be virtual on some of your mapped classes read this.
  4. Use the session object to interact with the persistence layer.  The Session is the item used the most, it’s kind of like a database connection/command in that you have to open it and commit stuff and close it when your done, but that’s as far that analogy goes.

It’s amazing how much time one can save on the mundane tasks of persisting your hard work you put into your domain model.  This is the biggest selling point for me, as importance of your enterprise application isn’t general how you going to save data object but how do these data objects relate to each other.

Good book: NHibernate in Action

Example sites:

Using NHibernate with ASP.NET 2.0 site

Simple Example Using Test Driven Approach