To Dispose or Not To Dispose? Here is the question.

One of the best articles that I’ve read so far about the importance of properly disposing SPWeb and SPSite objects has been written by one of my favourite SharePoint bloggers Stefan Gossner and can be found here.

Unfortunately there are still some conflicting messages about what to do with SPWeb instances creates via the SPSite.RootWeb property. According to Stefan we need to call dispose but other people from Microsoft are saying the opposite as you can see here.

So the search continues….

SharePoint Lessons Learned – Clearly Define a Deployment Baseline (Part 2)

In part 1 of this post I’ve talked about the principles behind the creation of a Deployment Baseline during the development of SharePoint based applications. In this post I’m going to talk about how we, at Collaboris. normally group and categorise the different artefacts to create this baseline.

This post assumes that you are familiar with the concepts of SharePoint Features, Site Definitions and Solution Packages. For a primer on these concepts please refer to this page on MSDN.

There has always been a lot of debate around the best way to deploy SharePoint applications. Some people do not like Site Definitions or Features and prefer to use Site Templates (.stp), others prefer xcopy deployment and others like me stick with Features and Site Definitions deployed using Solution Packages. I’m not going into that debate here, I’m simply going to describe the approach that we have been taking for the last few years without any regrets.

Like I’ve mentioned on the first post I’m strong believer in using the SharePoint Features and Solution Framework for the delivery of SharePoint applications. We normally try to deploy most of the artefacts via Features which are activated via Site Definitions and deployed using Solution Packages (WSP). One of the biggest advantages of using WSPs as a delivery mechanism is that SharePoint will automatically deploy the WSP contents to all Web Front End servers on the Farm.

We normally try to identify the deployment artefacts very early in the development lifecycle. Normally just after the initial requirements analysis. We use Mind Maps during this stage because they are very easy to produce and can quickly give you an high level overview of what needs to be delivered. We call these documents “Deployment Artefacts Maps” and we normally create two; one showing all the sites, pages, doc libraries and lists (see example below) and another with all the UI controls. These are dynamic documents that will evolve during the project lifecycle.

 

Example of a Deployment Artefact Map

 

These “Artefact Maps” together with a high level domain model (that helps me to understand the complexity of the data) allows us to size and cost the project from very early stages.

The next step is to group the artefacts and define a deployment strategy for each group. We normally organise them into into three major groups:

1. Infrastructure Artefacts (CSS, Style Images, Page Layouts, Site Columns, Master Pages, Content Types, List Templates, etc) – normally deployed via Features and CAML scripts and activated via Site Definitions (so that they are automatically activated when a site is created).
2. Configuration Artefacts ( Permissions, Navigation, Search Scopes etc) – deployed via features but using a mixture of CAML and custom code called from feature receivers to overcome some of the CAML limitations. This is were tools like SAF are most useful and can dramatically decrease the amount of effort require to deploy your application (SAF is a fully extendable XML framework that incorporates a series of custom actions that extend the functionality provided by CAML. For more information on SAF please follow this link ).
3. Content Artefacts (documents, page layout content, content images, lists content etc) – we normally subdivide this into 4 subcategories:

• Static Content – This is content that it will not change and you want to be there from day one (when the site is first deployed). We normally use a feature with CAML to deploy this.
• Sample Content – It’s a lot easier for business users to be given a site with some sample content that exemplifies how the site can be used. This content is deployed via a feature that can be manually activated by a site administrator.
• Test Content – normally created by the test team to cover all the different test scenarios. This content is created once and then manually activated by the test team (simply by activating a feature) in every test iteration.
• Dynamic/Business content – This is content normally created by business users (sometimes even when the site is still under development). We deploy this content using a SAF deployment action that uses the CMP API to provision the content. One very important thing here is that our feature will only provision content (list items or documents) into an already existing lists that has been created by an infrastructure feature (see point ‘1’ above). This way we clearly keep a separation between infrastructure (the list template and instance) and content. This separation also helps delimiting the areas of responsibility between IT and the Business (IT department is responsible for infrastructure artefacts and business is responsible for content artefacts ).

This was a very high level overview of how in Collaboris we define a Deployment Baseline. The next step will be the creation of a Visual Studio solution to implement the deployment packages defined in this Baseline. But more on that on a future post.

SharePoint Lessons Learned – Clearly Define a Deployment Baseline (Part 1)

One of the major lessons that I’ve learned so far with SharePoint development is how important it is to clearly define your Deployment Baseline from very early stages in the development lifecycle. In part 1 of this blog post I will describe the concepts behind this Deployment Baseline and in part 2 I will describe how in Collaboris we apply them to the development of SharePoint applications.

A deployment baseline is a clear definition of the artefacts that are going to be built and deployed when creating a new application. These artefacts should be catalogued and grouped based on their purpose, functionality and on how we plan to deploy and maintain them once the application is live. This baseline will also help to clearly define the different areas of responsibility within the team.

Imagine that you are developing a new Web Content Managed application where the development team is responsible for the creation of all the technical artefacts (ASP, HTML, CSS etc) and the business team is responsible for the creation of content (pages and documents). In this scenario it will make sense to create two separate packages; one for the technical artefacts the other for the content artefacts. These packages can then be maintained, managed and version controlled separately. To do a release you would just need to release the package with the technical artefacts followed by the content package (you can also have more than one content package but more on that later).

In SharePoint development, the Features and Solution Framework gives you the right tools to define and create this Deployment Baseline and I think that it is a worthy investment to get familiar with it before starting to develop any SharePoint application (strongly recommend Andrew Connell’s book to learn this). Unfortunately this framework also has a lot of limitations that are not immediately obvious when you start to use it so be prepared to get your hands dirty and do some coding to overcome some of them (or have a look at SAF the free tool that Collaboris released on CodePlex).

In part 2 of this post I will go through the steps on how in Collaboris we organise our deployment artefacts to create a deployment baseline for our projects. Stay tuned.

SharePoint Lessons Learned – Don’t forget to use proven design patterns and practices

SharePoint should be seen as another layer in the technology stack that your code will interact with. But just because you are using this layer you should not forget to follow good and proven design practices. The following points describe some of the principles that sometimes seem to be forgotten when using SharePoint:

1. Create a data access layer to access your lists.

   After all you would be doing this if you were programming against a database table so why not do the same for a SharePoint list. The last thing you want is to have several components of your application accessing the same list in several different ways. If the list schema, name or location changes then you would need to change the code in every component that uses that list (this could be acceptable for small proof of concept applications but not for enterprise applications). In a future post I'll be giving an example of designing and coding a simple SharePoint data access layer.

2. Use SharePoint lists to store configuration data. Create a configuration site, on your site collection, containing all the configuration lists

   Before SharePoint all configuration needed by you applications resided in a data store (normally a database). You then needed to build a UI to allow power users and/or administrators to change that information. You also had to implement security mechanisms to make sure that only the users with the correct permissions were allowed to change or access that data. With SharePoint you can accomplish all of that with out of the box functionality without writing a single line of code.

   You can also adopt the same principle for structured content. For example let's say that you want to create a very simple top navigation menu (like the one on the top right corner of this page) and allow your users to add new links or edit existing ones. You could create a TopNavs list on a "/config" site under your site collection with "Name" and "URL" columns. You could then create the menu using a simple ASP custom control and bind it to the TopNavs list. I'll be showing how to do exactly that in a future post.

3. Do not forget about databases

   Just because you are using SharePoint and have all of those nice lists that you can easily create and configure it doesn't mean that you should stop using databases (disregarding the fact that SharePoint is actually storing all of the list data on a database). You should always ask yourself the reasons why you should use a SharePoint list instead of a database table. Remember - SharePoint is a service layer - so if you are not using the services provided by that layer then perhaps you should be using a database table instead. SharePoint lists also have greater limitations than database tables (see this Microsoft document for more information on those limitations).

4. Plan the deployment process early and be very careful with tools like SharePoint Designer

   Don’t assume, that just because you can use the server that sits under your desk to develop fantastic and feature rich SharePoint sites using SharePoint Designer, that you will be able to easily transform them into enterprise class applications that will run on your corporate servers. Sites developed using SharePoint designer are normally difficult to move between environments (development, test and production) and can bypass all the version control mechanisms your organisation already has in place to develop other types of applications (ex. TFS or VSS). So get familiar with the SharePoint Designer limitations before start using it (don’t make everything into a nail just because you have a new shinny hammer).

The SharePoint Maturity Model (SMM)

MOSS has been the most successful server product Microsoft ever released. Sales are growing much faster than Microsoft ever expected and apparently the UK is outstripping worldwide growth (for more see this). Unfortunately this quick growth is also highlighting one of the major problems that everyone seems to be struggling with - deployment. I'm a SharePoint consultant and I've been working with MOSS since Beta 2. I've worked across several companies in the UK and I have also debated this issue with other colleagues and we are all in agreement: deployment is the biggest pain on any SharePoint project. It's one of the areas that will give you more problems and cost you more money. What is curious is that all companies adopting SharePoint seem to go through the same evolution path. Finding a way to measure where my customers are on this path gives me a good idea on the challenges that I will be facing when moving their projects forward. The kind of measure that I'm talking about it's called a Maturity Model so I called it the SharePoint Maturity Model (SMM).

Like the Capability Maturity Model (CMM) I divided the SMM is divided into 5 levels (except for level 4, I have used the same names has the CMM because they are a good match):

1. Level 1 – Ad hoc (Chaos): Companies in level one normally do all development directly in the production system. There is no development or system test environment because no one knows how to move the site to a different server. These companies are normally just starting using MOSS and they do not have experienced MOSS resources.
   
2. Level 2 – Repeatable (Backup/Restore): Level 2 companies have found that SharePoint content and configuration can be moved across environments using backup and restore. As such they have a repeatable deployment process. Unfortunately these companies soon realise that using backup and restore will cause a series of problems that are not immediately noticeable (like the publishing pages still pointing to page layouts on the old environment or exceptions being raised when trying to edit publishing page settings). I was once called to a customer that was having a series of problems with their live environment; it was running too slow with frequent crashes, also they could not understand why the problem seemed to become worse when they turned off their development server (which was on the same network as live). It turned out to be the typical backup\restore syndrome that had to be fixed by copying the content of the publishing pages to new pages created in the live environment (we gave them a small C# script to do that).
   
3. Level 3 – Defined: Companies normally get to level 3 after learning with their mistakes and spending a lot of time and effort going through levels 1 and 2. In level 3 companies have learned that there is a better way to deploy their applications by using CAML in Features, Solution Packages and Site Definitions. Unfortunately there are quite a few configuration artefacts (like security, search, etc) that cannot be deployed via CAML. To overcome this problem companies develop a mixed automated\manual deployment process where some artefacts are deployed via CAML scripts and others by following a long list of manual steps (see Deployment Pete Syndrome below).
   
4. Level 4 – Automated: At this level companies have managed to fully automate their deployment process by using a mixture of CAML and .Net code. This normally comes at great cost because not only developers will need to learn CAML but they will also need to learn the SharePoint API so that they can automate the deployment of configuration artefacts not covered by CAML.
   
5. Level 5 - Optimised: This is the SharePoint deployment Nirvana. At this level companies not only have their deployment process fully optimised and automated but they have also started to manage those processes has shared Intellectual Property (IP) across projects. At the end of every project there is an IP harvesting process that brings any new deployment code into a central location so that it can be reused and shared by future projects. The time required by these companies to setup a new SharePoint project is minimal and quality of their deployment scripts is high because it's based on tested code used several times before.
   

The Deployment Pete Syndrome (DPS):

One of the constant complaints that I seem to find when meeting new customers is about the exceeding amount of time that it takes to move their SharePoint application between environments. Especially when they have been promised -normally by a sales person more interested in hitting their sales target than in providing a fit for purpose solution- that most of their requirements could be accomplished with out-of-the-box (OOB) functionality provided by SharePoint (I'll leave my OOB ramblings for another post). During my first meeting with these customers I'm normally introduced to their deployment expert which, for the purpose of this post, I will call Pete in honour of a great ex-colleague of mine that used to spend a lot of is time deploying SharePoint applications. In further talks with Pete I normally discover that he is given a huge list (sometimes several dozens of pages long) of manual steps that he needs to follow in order to move each SharePoint application between environments. I call this the "Deployment Pete Syndrome" (in lack of a better name) and unfortunately it seems to be a constant of most companies that I go to (especially those in levels 2 and 3 of the SMM).

In my opinion one of the ways to help companies adopting SharePoint and ease the pain of reaching the SMM level 5 (if they wish to do so) is to offer them a mixture of advice and knowledge transfer together with a series of packaged tools that have been designed and developed based on the experience past projects. This is one of the main objectives of a new UK based company called Collaboris (we are still finishing the web site so please bear with us) that I've started together with my colleague Mark Jones.

How to free your Data View Web Part (DVWP) from those nasty GUIDs

Ever want to use the same page layout containing a DVWP, and bind it to a local site list in each site of your site collection ?

Ever want to be able to deploy a page with a DVWP to a different server?

If you ever tried any of the above using the SharePoint Designer DataViewWebPart you would have found problems because, by default, SharePointDesigner binds the control to the list instance using the list instance GUID. To resolve this we need to replace the GUIDs by the list name. The steps to do this are:

1) On the attributes of the DataFormWebPart element replace the attribute ListName="{GUID}" by ListName="LIST_NAME" where LIST_NAME is the name of the list that you are binding to.

2) Go through all of the DataFormParameter elements and replace:    

    <WebPartPages:DataFormParameter Name="ListID" ParameterKey="ListID" PropertyName="ParameterValues" DefaultValue="{GUID}"/>

With:

     <WebPartPages:DataFormParameter Name="ListName" ParameterKey="ListName" PropertyName="ParameterValues" DefaultValue="LIST_NAME"/>

3) Go to the ParameterBindings element and replace

    <ParameterBinding Name="ListID" Location="None" DefaultValue="{GUID}"/>

With:

    <ParameterBinding Name="ListName" Location="None" DefaultValue="LIST_NAME"/>    

This should give you a GUID free DVWP that can be placed on a page layout used by multiple sites on your site collection (as long as the name of the list is the same on all sites).

Sharepoint list deployment

As we all know by now (unfortunately) deployment is one of the big "pain areas" in SharePoint 2007, specially the deployment of custom lists and document libraries. There are already several blog posts out there covering this topic but it seems that they all miss something (i.e. some cover out to deploy document libraries but not mention how to configure the library to use custom content types). The Microsoft documentation is, as usual, very thin on this area as well.

To cover this area I'm planing to post a blog called "One Stop Shop for creating Sharepoint libraries via features" that will cover:
1) How to deploy custom lists via features
2) How to deploy custom document libraries via features
3) How to use a custom content types in a list definition
4) How to deploy views.


In the mean time here are the pointers to some useful blog posts about custom lists deployment:

Creating a Custom Document Library Feature in WSS V3 / MOSS 2007
Add a custom list definition to SharePoint
Using a SharePoint 2007 content type in a List definition

SharePoint Applications – Design Lessons Learned (so far)

I can't believe that it has been 18 months since my last blog entry. The main reason for this absence is that I've been busy designing and implementing MOSS enterprise projects since June 2006 and, like they say, time flies when you are having fun (well sort of). I've decided to return mainly because I needed to create a list of "tricks" and lessons learned as a "reminder to self" so that I can apply them in future projects. If I can help other people on the way then even better.

In this blog post I'm going to highlight some of the design and architecture lessons that I've learned so far. In future posts I'll be drilling down in some of the areas and present concrete examples.

So here is my list so far:

1. Do not forget standard/proven design and architecture patterns just because you are using SharePoint. SharePoint should be seen as another layer in your technology stack that your code will interact with. But just because you are using this layer you should not forget to follow good and proven design practices. The following points describe some of the principles that sometimes seem to be forgotten when using SharePoint.
   
   1. Create a data access layer to access your lists. After all you would be doing this if you were programming against a database table so why not do the same for a SharePoint list. The last thing you want is to have several components of your application accessing the same list in several different ways. If the list schema, name or location changes then you would need to change the code in every component that uses that list (this could be acceptable for small proof of concept applications but not for enterprise applications). In a future post I'll be giving an example of designing and coding a simple SharePoint data access layer.
      
   2. Use SharePoint lists to store configuration data. Create a configuration site, on your site collection, containing all the configuration lists. Before SharePoint all configuration needed by you applications resided in a data store (normally a database). You then needed to build a UI to allow power users and/or administrators to change that information. You also had to implement security mechanisms to make sure that only the users with the correct permissions were allowed to change or access that data. With SharePoint you can accomplish all of that with out of the box functionality without writing a single line of code. You can also adopt the same principle for structured content. For example let's say that you want to create a very simple top navigation menu (like the one on the top right corner of this page) and allow your users to add new links or edit existing ones. You could create a TopNavs list on a "/config" site under your site collection with "Name" and "URL" columns. You could then create the menu using a simple ASP custom control and bind it to the TopNavs list. I'll be showing how to do exactly that in a future post.
      
   3. Do not forget about databases. Just because you are using SharePoint and have all of those nice lists that you can easily create and configure it doesn't mean that you should stop using databases (disregarding the fact that SharePoint is actually storing all of the list data on a database). You should always ask yourself the reasons why you should use a SharePoint list instead of a database table. Remember - SharePoint is a service layer - so if you are not using the services provided by that layer then perhaps you should be using a database table instead. SharePoint lists also have greater limitations than database tables (see this Microsoft document for more information on those limitations).
      
   4. Empower business users but try to keep them in their comfort zone. I know that this sounds like a sales cliché but it's actually one of the main SharePoint selling points. You should ear this sentence echoing in the back of your mind every time you are talking to a customer. This will help you making better decisions and keep your customers happy. Here is an example:
      
      1. You are responsible for the architecture and design of a brochureware site for a major financial institution. One of the main reasons for them to adopt SharePoint was to give more control to business users. You decided to base the site on the out of the box publishing site definition and you are just about to test your first content page (which is using the out of the box Rich HTML editing control).You look at the generated html code and suddenly freeze. Your heart starts pumping faster and you cannot believe on what your eyes are telling you; the HTML code generated by the control is appalling. Your professional pride kicks in and tells you that you cannot be responsible for code like this. What do you do?
         
         1. Talk to the customer about the problem and suggest replacing the control with a "professional quality" third party editing control (they would need to agree the extra costs).
            
         2. Tell the customer that you cannot be responsible about the quality of the code generated by the control and ask them to make a decision. You can point out that, although the quality of the generated code is far from ideal, it can be a viable option as long as they are not concerned with accessibility issues.
            
         3. Tell the customer to continue to use the control and teach them how to improve the code by showing them the control's HTML view.
            

      Option 3 is the least appealing one because it will require your business users to step out of their comfort zone and learn something that they shouldn't need to for their day to day jobs (unless they are technical minded and do not mind the learning curve).
      

      
       

 

SharePoint Services, SQL 2005 and PDF searches

WSS (2003) depends on SQL Full-text Search service (MSFTESQL) to perform document searches (SQL Express does not support Full-text search so, to be able to search document contents, you will need to use SQL Workgroup edition or above). Because this service cannot understand all different document formats (Word, Excel, PDF, etc) it depends on filter components provided by third parties (i.e. Adobe provides the pdf filter) to index their contents. These components are often called IFilter components because they all need to implement the well defined IFilter interface (for more on IFilters see this).

To provide PDF search functionality within WSS the PDF IFilter needs to be installed on the SQL server box (to download click here). Installing the PDF IFilter in a SQL 2000 box will automatically make it available to the SQL Full-text Search Engine (MSFTESQL) but, on a SQL 2005 box, MSFESQL will not, by default, load third party filters already installed in the operating system. To enable this the following extra steps are required (after installing the PDF IFilter):

• Run the following SQL statements against the WSS content database (be aware that this will reduce the security of SQL server so it should be restored after the IFilter is loaded):


• Exec sp_fulltext_service 'load_os_resources', 1 (This command tells the Microsoft Search service to load OS specific wordbreakers and stemmers)


• Exec sp_fulltext_service 'verify_signature', 0 (This allows the Microsoft Search service to load unsigned filters)


• Bounce SQL server and the Full-text Search engine (MSFTESQL).


• Check that PDFs have been added to the list of files available for Full-text indexing by running the following SQL statement against the WSS content database:


• SELECT * FROM sys.fulltext_document_types


• Restore the Full-text service settings to their default


• Exec sp_fulltext_service 'verify_signature', 1
• Exec sp_fulltext_service 'load_os_resources', 0


• To index existing PDFs in WSS go to the WSS administrative page disable and re-enable the full-text search and index component. This will rebuild the Full-text index and use the new IFilter to index existing PDFs.

Want to create ASP.NET's on a box that has sharepoint installed?

Neville tells you howto on his Banana Biscuit blog.