SharePoint Lessons Learned – Clearly Define a Deployment Baseline (Part 1)

One of the major lessons that I’ve learned so far with SharePoint development is how important it is to clearly define your Deployment Baseline from very early stages in the development lifecycle. In part 1 of this blog post I will describe the concepts behind this Deployment Baseline and in part 2 I will describe how in Collaboris we apply them to the development of SharePoint applications.

A deployment baseline is a clear definition of the artefacts that are going to be built and deployed when creating a new application. These artefacts should be catalogued and grouped based on their purpose, functionality and on how we plan to deploy and maintain them once the application is live. This baseline will also help to clearly define the different areas of responsibility within the team.

Imagine that you are developing a new Web Content Managed application where the development team is responsible for the creation of all the technical artefacts (ASP, HTML, CSS etc) and the business team is responsible for the creation of content (pages and documents). In this scenario it will make sense to create two separate packages; one for the technical artefacts the other for the content artefacts. These packages can then be maintained, managed and version controlled separately. To do a release you would just need to release the package with the technical artefacts followed by the content package (you can also have more than one content package but more on that later).

In SharePoint development, the Features and Solution Framework gives you the right tools to define and create this Deployment Baseline and I think that it is a worthy investment to get familiar with it before starting to develop any SharePoint application (strongly recommend Andrew Connell’s book to learn this). Unfortunately this framework also has a lot of limitations that are not immediately obvious when you start to use it so be prepared to get your hands dirty and do some coding to overcome some of them (or have a look at SAF the free tool that Collaboris released on CodePlex).

In part 2 of this post I will go through the steps on how in Collaboris we organise our deployment artefacts to create a deployment baseline for our projects. Stay tuned.

SharePoint Lessons Learned – Don’t forget to use proven design patterns and practices

SharePoint should be seen as another layer in the technology stack that your code will interact with. But just because you are using this layer you should not forget to follow good and proven design practices. The following points describe some of the principles that sometimes seem to be forgotten when using SharePoint:

1. Create a data access layer to access your lists.

   After all you would be doing this if you were programming against a database table so why not do the same for a SharePoint list. The last thing you want is to have several components of your application accessing the same list in several different ways. If the list schema, name or location changes then you would need to change the code in every component that uses that list (this could be acceptable for small proof of concept applications but not for enterprise applications). In a future post I'll be giving an example of designing and coding a simple SharePoint data access layer.

2. Use SharePoint lists to store configuration data. Create a configuration site, on your site collection, containing all the configuration lists

   Before SharePoint all configuration needed by you applications resided in a data store (normally a database). You then needed to build a UI to allow power users and/or administrators to change that information. You also had to implement security mechanisms to make sure that only the users with the correct permissions were allowed to change or access that data. With SharePoint you can accomplish all of that with out of the box functionality without writing a single line of code.

   You can also adopt the same principle for structured content. For example let's say that you want to create a very simple top navigation menu (like the one on the top right corner of this page) and allow your users to add new links or edit existing ones. You could create a TopNavs list on a "/config" site under your site collection with "Name" and "URL" columns. You could then create the menu using a simple ASP custom control and bind it to the TopNavs list. I'll be showing how to do exactly that in a future post.

3. Do not forget about databases

   Just because you are using SharePoint and have all of those nice lists that you can easily create and configure it doesn't mean that you should stop using databases (disregarding the fact that SharePoint is actually storing all of the list data on a database). You should always ask yourself the reasons why you should use a SharePoint list instead of a database table. Remember - SharePoint is a service layer - so if you are not using the services provided by that layer then perhaps you should be using a database table instead. SharePoint lists also have greater limitations than database tables (see this Microsoft document for more information on those limitations).

4. Plan the deployment process early and be very careful with tools like SharePoint Designer

   Don’t assume, that just because you can use the server that sits under your desk to develop fantastic and feature rich SharePoint sites using SharePoint Designer, that you will be able to easily transform them into enterprise class applications that will run on your corporate servers. Sites developed using SharePoint designer are normally difficult to move between environments (development, test and production) and can bypass all the version control mechanisms your organisation already has in place to develop other types of applications (ex. TFS or VSS). So get familiar with the SharePoint Designer limitations before start using it (don’t make everything into a nail just because you have a new shinny hammer).

The SharePoint Maturity Model (SMM)

MOSS has been the most successful server product Microsoft ever released. Sales are growing much faster than Microsoft ever expected and apparently the UK is outstripping worldwide growth (for more see this). Unfortunately this quick growth is also highlighting one of the major problems that everyone seems to be struggling with - deployment. I'm a SharePoint consultant and I've been working with MOSS since Beta 2. I've worked across several companies in the UK and I have also debated this issue with other colleagues and we are all in agreement: deployment is the biggest pain on any SharePoint project. It's one of the areas that will give you more problems and cost you more money. What is curious is that all companies adopting SharePoint seem to go through the same evolution path. Finding a way to measure where my customers are on this path gives me a good idea on the challenges that I will be facing when moving their projects forward. The kind of measure that I'm talking about it's called a Maturity Model so I called it the SharePoint Maturity Model (SMM).

Like the Capability Maturity Model (CMM) I divided the SMM is divided into 5 levels (except for level 4, I have used the same names has the CMM because they are a good match):

1. Level 1 – Ad hoc (Chaos): Companies in level one normally do all development directly in the production system. There is no development or system test environment because no one knows how to move the site to a different server. These companies are normally just starting using MOSS and they do not have experienced MOSS resources.
   
2. Level 2 – Repeatable (Backup/Restore): Level 2 companies have found that SharePoint content and configuration can be moved across environments using backup and restore. As such they have a repeatable deployment process. Unfortunately these companies soon realise that using backup and restore will cause a series of problems that are not immediately noticeable (like the publishing pages still pointing to page layouts on the old environment or exceptions being raised when trying to edit publishing page settings). I was once called to a customer that was having a series of problems with their live environment; it was running too slow with frequent crashes, also they could not understand why the problem seemed to become worse when they turned off their development server (which was on the same network as live). It turned out to be the typical backup\restore syndrome that had to be fixed by copying the content of the publishing pages to new pages created in the live environment (we gave them a small C# script to do that).
   
3. Level 3 – Defined: Companies normally get to level 3 after learning with their mistakes and spending a lot of time and effort going through levels 1 and 2. In level 3 companies have learned that there is a better way to deploy their applications by using CAML in Features, Solution Packages and Site Definitions. Unfortunately there are quite a few configuration artefacts (like security, search, etc) that cannot be deployed via CAML. To overcome this problem companies develop a mixed automated\manual deployment process where some artefacts are deployed via CAML scripts and others by following a long list of manual steps (see Deployment Pete Syndrome below).
   
4. Level 4 – Automated: At this level companies have managed to fully automate their deployment process by using a mixture of CAML and .Net code. This normally comes at great cost because not only developers will need to learn CAML but they will also need to learn the SharePoint API so that they can automate the deployment of configuration artefacts not covered by CAML.
   
5. Level 5 - Optimised: This is the SharePoint deployment Nirvana. At this level companies not only have their deployment process fully optimised and automated but they have also started to manage those processes has shared Intellectual Property (IP) across projects. At the end of every project there is an IP harvesting process that brings any new deployment code into a central location so that it can be reused and shared by future projects. The time required by these companies to setup a new SharePoint project is minimal and quality of their deployment scripts is high because it's based on tested code used several times before.
   

The Deployment Pete Syndrome (DPS):

One of the constant complaints that I seem to find when meeting new customers is about the exceeding amount of time that it takes to move their SharePoint application between environments. Especially when they have been promised -normally by a sales person more interested in hitting their sales target than in providing a fit for purpose solution- that most of their requirements could be accomplished with out-of-the-box (OOB) functionality provided by SharePoint (I'll leave my OOB ramblings for another post). During my first meeting with these customers I'm normally introduced to their deployment expert which, for the purpose of this post, I will call Pete in honour of a great ex-colleague of mine that used to spend a lot of is time deploying SharePoint applications. In further talks with Pete I normally discover that he is given a huge list (sometimes several dozens of pages long) of manual steps that he needs to follow in order to move each SharePoint application between environments. I call this the "Deployment Pete Syndrome" (in lack of a better name) and unfortunately it seems to be a constant of most companies that I go to (especially those in levels 2 and 3 of the SMM).

In my opinion one of the ways to help companies adopting SharePoint and ease the pain of reaching the SMM level 5 (if they wish to do so) is to offer them a mixture of advice and knowledge transfer together with a series of packaged tools that have been designed and developed based on the experience past projects. This is one of the main objectives of a new UK based company called Collaboris (we are still finishing the web site so please bear with us) that I've started together with my colleague Mark Jones.

How to free your Data View Web Part (DVWP) from those nasty GUIDs

Ever want to use the same page layout containing a DVWP, and bind it to a local site list in each site of your site collection ?

Ever want to be able to deploy a page with a DVWP to a different server?

If you ever tried any of the above using the SharePoint Designer DataViewWebPart you would have found problems because, by default, SharePointDesigner binds the control to the list instance using the list instance GUID. To resolve this we need to replace the GUIDs by the list name. The steps to do this are:

1) On the attributes of the DataFormWebPart element replace the attribute ListName="{GUID}" by ListName="LIST_NAME" where LIST_NAME is the name of the list that you are binding to.

2) Go through all of the DataFormParameter elements and replace:    

    <WebPartPages:DataFormParameter Name="ListID" ParameterKey="ListID" PropertyName="ParameterValues" DefaultValue="{GUID}"/>

With:

     <WebPartPages:DataFormParameter Name="ListName" ParameterKey="ListName" PropertyName="ParameterValues" DefaultValue="LIST_NAME"/>

3) Go to the ParameterBindings element and replace

    <ParameterBinding Name="ListID" Location="None" DefaultValue="{GUID}"/>

With:

    <ParameterBinding Name="ListName" Location="None" DefaultValue="LIST_NAME"/>    

This should give you a GUID free DVWP that can be placed on a page layout used by multiple sites on your site collection (as long as the name of the list is the same on all sites).

Sharepoint list deployment

As we all know by now (unfortunately) deployment is one of the big "pain areas" in SharePoint 2007, specially the deployment of custom lists and document libraries. There are already several blog posts out there covering this topic but it seems that they all miss something (i.e. some cover out to deploy document libraries but not mention how to configure the library to use custom content types). The Microsoft documentation is, as usual, very thin on this area as well.

To cover this area I'm planing to post a blog called "One Stop Shop for creating Sharepoint libraries via features" that will cover:
1) How to deploy custom lists via features
2) How to deploy custom document libraries via features
3) How to use a custom content types in a list definition
4) How to deploy views.


In the mean time here are the pointers to some useful blog posts about custom lists deployment:

Creating a Custom Document Library Feature in WSS V3 / MOSS 2007
Add a custom list definition to SharePoint
Using a SharePoint 2007 content type in a List definition

SharePoint Applications – Design Lessons Learned (so far)

I can't believe that it has been 18 months since my last blog entry. The main reason for this absence is that I've been busy designing and implementing MOSS enterprise projects since June 2006 and, like they say, time flies when you are having fun (well sort of). I've decided to return mainly because I needed to create a list of "tricks" and lessons learned as a "reminder to self" so that I can apply them in future projects. If I can help other people on the way then even better.

In this blog post I'm going to highlight some of the design and architecture lessons that I've learned so far. In future posts I'll be drilling down in some of the areas and present concrete examples.

So here is my list so far:

1. Do not forget standard/proven design and architecture patterns just because you are using SharePoint. SharePoint should be seen as another layer in your technology stack that your code will interact with. But just because you are using this layer you should not forget to follow good and proven design practices. The following points describe some of the principles that sometimes seem to be forgotten when using SharePoint.
   
   1. Create a data access layer to access your lists. After all you would be doing this if you were programming against a database table so why not do the same for a SharePoint list. The last thing you want is to have several components of your application accessing the same list in several different ways. If the list schema, name or location changes then you would need to change the code in every component that uses that list (this could be acceptable for small proof of concept applications but not for enterprise applications). In a future post I'll be giving an example of designing and coding a simple SharePoint data access layer.
      
   2. Use SharePoint lists to store configuration data. Create a configuration site, on your site collection, containing all the configuration lists. Before SharePoint all configuration needed by you applications resided in a data store (normally a database). You then needed to build a UI to allow power users and/or administrators to change that information. You also had to implement security mechanisms to make sure that only the users with the correct permissions were allowed to change or access that data. With SharePoint you can accomplish all of that with out of the box functionality without writing a single line of code. You can also adopt the same principle for structured content. For example let's say that you want to create a very simple top navigation menu (like the one on the top right corner of this page) and allow your users to add new links or edit existing ones. You could create a TopNavs list on a "/config" site under your site collection with "Name" and "URL" columns. You could then create the menu using a simple ASP custom control and bind it to the TopNavs list. I'll be showing how to do exactly that in a future post.
      
   3. Do not forget about databases. Just because you are using SharePoint and have all of those nice lists that you can easily create and configure it doesn't mean that you should stop using databases (disregarding the fact that SharePoint is actually storing all of the list data on a database). You should always ask yourself the reasons why you should use a SharePoint list instead of a database table. Remember - SharePoint is a service layer - so if you are not using the services provided by that layer then perhaps you should be using a database table instead. SharePoint lists also have greater limitations than database tables (see this Microsoft document for more information on those limitations).
      
   4. Empower business users but try to keep them in their comfort zone. I know that this sounds like a sales cliché but it's actually one of the main SharePoint selling points. You should ear this sentence echoing in the back of your mind every time you are talking to a customer. This will help you making better decisions and keep your customers happy. Here is an example:
      
      1. You are responsible for the architecture and design of a brochureware site for a major financial institution. One of the main reasons for them to adopt SharePoint was to give more control to business users. You decided to base the site on the out of the box publishing site definition and you are just about to test your first content page (which is using the out of the box Rich HTML editing control).You look at the generated html code and suddenly freeze. Your heart starts pumping faster and you cannot believe on what your eyes are telling you; the HTML code generated by the control is appalling. Your professional pride kicks in and tells you that you cannot be responsible for code like this. What do you do?
         
         1. Talk to the customer about the problem and suggest replacing the control with a "professional quality" third party editing control (they would need to agree the extra costs).
            
         2. Tell the customer that you cannot be responsible about the quality of the code generated by the control and ask them to make a decision. You can point out that, although the quality of the generated code is far from ideal, it can be a viable option as long as they are not concerned with accessibility issues.
            
         3. Tell the customer to continue to use the control and teach them how to improve the code by showing them the control's HTML view.
            

      Option 3 is the least appealing one because it will require your business users to step out of their comfort zone and learn something that they shouldn't need to for their day to day jobs (unless they are technical minded and do not mind the learning curve).
      

      
       

 

SharePoint Services, SQL 2005 and PDF searches

WSS (2003) depends on SQL Full-text Search service (MSFTESQL) to perform document searches (SQL Express does not support Full-text search so, to be able to search document contents, you will need to use SQL Workgroup edition or above). Because this service cannot understand all different document formats (Word, Excel, PDF, etc) it depends on filter components provided by third parties (i.e. Adobe provides the pdf filter) to index their contents. These components are often called IFilter components because they all need to implement the well defined IFilter interface (for more on IFilters see this).

To provide PDF search functionality within WSS the PDF IFilter needs to be installed on the SQL server box (to download click here). Installing the PDF IFilter in a SQL 2000 box will automatically make it available to the SQL Full-text Search Engine (MSFTESQL) but, on a SQL 2005 box, MSFESQL will not, by default, load third party filters already installed in the operating system. To enable this the following extra steps are required (after installing the PDF IFilter):

• Run the following SQL statements against the WSS content database (be aware that this will reduce the security of SQL server so it should be restored after the IFilter is loaded):


• Exec sp_fulltext_service 'load_os_resources', 1 (This command tells the Microsoft Search service to load OS specific wordbreakers and stemmers)


• Exec sp_fulltext_service 'verify_signature', 0 (This allows the Microsoft Search service to load unsigned filters)


• Bounce SQL server and the Full-text Search engine (MSFTESQL).


• Check that PDFs have been added to the list of files available for Full-text indexing by running the following SQL statement against the WSS content database:


• SELECT * FROM sys.fulltext_document_types


• Restore the Full-text service settings to their default


• Exec sp_fulltext_service 'verify_signature', 1
• Exec sp_fulltext_service 'load_os_resources', 0


• To index existing PDFs in WSS go to the WSS administrative page disable and re-enable the full-text search and index component. This will rebuild the Full-text index and use the new IFilter to index existing PDFs.

Want to create ASP.NET's on a box that has sharepoint installed?

Neville tells you howto on his Banana Biscuit blog.

Single Sign On with Forms Authentication ?

Read this article to find out how (unfortunately will not work with SharePoint or ASP applications).

Keith's Security Book

Nice of Keith Brown to make his excellent security book available online for reference. But please remember to buy the hardcopy if you like the book. This would hopefully encourage other writers to do the same.

SharePoint Site Definitions

Site definitions are the way to go if you want to brand your SharePoint team site. Unfortunately there is no clever GUI tool to do this so you will need to go back to the old fashion way - Manual editing. Find out how in this article

Branding a Sharepoint Site

How to brand a SharePoint site. A bit messy but a good start...

Software Factories

Are Software Factories the solution to the reuse Holy Grail? What is the future for the Software Craftsman?
Have a look and decide for yourself.

Sharepoint Template

Keep forgetting how to create a Sharepoint web site template ? Here are the instructions.

XML to PDF Converter

If you want to generate sever side reports in PDF from XML data then Ibex PDF Creator is the tool for you. Some of my colleagues have used it and the results (and speed) are very impressive.

Requirements, Requirements, Requirements...

Those are the three most valuable artifacts of a project. Requirements should drive most (if not all) activities in a project. A project should be architectural centric and requirements driven... We have all heard these cliches before but how exactly does one manage requirements in an iterative process? How do we make sure that all team members are rowing the same way?

Supporting Iterative Development Through Requirements Management gives a good insight on how effective requirements management can be accomplished in an iterative process - in this case using the Rational unified Process (RUP). Even if you are using another iterative process (like SCRUM or XP) the article should give some good ideas.

If you are not using an iterative process then forget about the article and read "Why should I be using an iterative process" instead.

I Believe in BLOBs

I'm talking about SQL server Binary Large Objects. Storing files in a database BLOB it's a very clever way of improving your application maintainability and in avoiding possible permissions nightmares. The File Upload with ASP.Net article shows how to upload a file from a browser and store it in a BLOB.

Unit testing made easy

Unit testing has taken a new momentum with the rise of the agile development practices. NUnit is a fantastic and simple unit-testing framework for all .Net languages but if you used it before you have probably come across the difficulty on how to isolate the unit that you want to test. NMock was created to address this issue. Mock Objects to the Rescue is a very good article on how to use this useful tool.

If you have not used NUnit before you will find a good introduction on the Test Driven C# msdn article.

Hosting ASP.NET Web applications

For good practices on how to host multiple ASP applications have a look at chapter 20 of the Improving Web Application Security book from the Microsoft Patterns and Practices series. I quite like their recommendation on isolating applications by identity

Security Contexts in ASP.NET

If you, like me, were confused about the different security contexts in IIS and how can they affect the way that you connect to SQL server look no further, Keith Brown's article A tale of Three Security contexts in ASP.NET tells everything that you need to know. Just remember that in IIS6 things work slightly different (i.e. you now have application pools and the default ASPNET account has been replaced by the "Network Service" account).