Microsoft

Blog Categories

Subscribe to RSS feed

Archives

Follow our Microsoft Technologies board on Pinterest

Posts Tagged ‘FAST for SharePoint’

Dealing with Single and Multi-Value Crawled Properties (FAST for SharePoint)

We were having some issues dealing with single and multi-value fields in SharePoint and how FAST interprets them for the index. The distinction between them is actually quite important and can have an impact when it comes down to refining your search results.

In our example, we were dealing with a multi-value site column lookup field in a list. FAST does some interesting things with multi-value and single value fields. For example, let’s use a Crawled Property called ows_region. If the field for the item is multi-valued, FAST will index all of those values in a Crawled Property called ows_region() with a variant type of 4127. If the field for the item is a single value, FAST will crawl this as a property called ows_region(text) with a variant type of 31.

cpname

These two crawled properties reference the same site column, however their content is separated. Since all of the multi-valued fields are in ows_region() and all of the single values are in ows_region(Text), we have to map both of them to a managed property if we want to cover all of the values.

To test this, we created three managed properties: owscountry (ows_countries(text)) – Single Value, multicountry (ows_countries()) – multi-value, and combinedcountry (ows_countries() and ows_countries(text)) – single and multivalue.

Using FS4SP Query Logger, we were able to see that the combinedcountry managed property would return all of the values, while owscountry and multicountry would only return single and multi-values respectively.

combinedcountry1

combinedcountry2

So on the FAST end, we have to ensure that we map all of the relevant Crawled Properties to Managed Properties to ensure that we have a full refiner that includes both Crawled and Managed Properties. Also we need to ensure that the “Include values from all crawled properties mapped” button is checked.

cpmappings

Pretty simple and straightforward. Perhaps slightly redundant, but at least the question is answered… yes you need both Crawled Properties in order to create a Managed Property that covers both multi-value and single values.

Tags: ,

Posted in Search

FAST for SharePoint 2010 Non-Admin node configuration

I spent considerable time last week trying to build 2-node FAST for SharePoint 2010 implementation. Creating the setup.xml file and building the admin FAST node was the easy part, but trying to connect the non-admin node is what took up most of my time.

(The default FAST installation creates 3 example files under C:FASTSearchetc directory, they are named deployment.sample.single.xml, deployment.sample.multi1(2).xml, most of the time you can tweak them to suit your needs instead of trying to create on from scratch.)

When running the configuration wizard, the error I kept getting was:

{drive}:FASTSearchbinMonitoringServiceConfig.exe” Output – Error: The file ‘{drive}:FASTSearchetcmiddleware.cfg’ was not found.

I searched through the blogs for quite some time, and I should here thank all the bloggers out there which provided useful information (including our very own David Sobiecki), links provided at the bottom here.

  1. Issue ncrtl status on the FAST admin node: we need to make sure all services show status running
  2. Try running setfastsearchipsec -create on the non-admin FAST node: this will prompt connection to the FAST admin node and parsing the setup.xml file. Depending on the error reported it’s usually either connectivity issue to the FAST admin node, or incorrect setup.xml formatting
  1. Recreate the setup.xml file using notepad: apparently fancier XML editors add invisible characters that FAST configuration wizard doesn’t like, so to be safe you should paste content on notepad and quickly proofread for any characters that are not supposed to be there
  1. Disable firewall on the FAST Admin node: now I know this is not best practice, but in my case it worked. FAST configuration wizard will complain if the firewall is disabled, so I tried many combinations. Having the firewall enabled on the non-admin node and disabled on the admin node is what worked for me
  2. Finally, unselect browser proxy settings (automatically detect settings, use proxy server etc.) on the non-admin node

Hope this summary will help other people who run into similar issue. If you feel inclined to read more, here is a list of blogs I used:

http://davemcmahon81.wordpress.com/2011/07/30/adding-a-non-admin-server-to-fast-search-for-sharepoint-2010-fails/

http://blogs.technet.com/b/speschka/archive/2010/05/25/solving-configuration-failures-with-non-admin-nodes-in-fast-search-for-sharepoint.aspx

http://blogs.msdn.com/b/scottos/archive/2010/07/16/we-need-to-talk-aka-it-s-not-me-it-s-you.aspx

http://social.technet.microsoft.com/Forums/en/sharepoint2010setup/thread/f80c1daf-50d0-4d33-84b5-c5533a6a129f

https://dotnetmafia.sys-con.com/node/1855150/mobile

http://blogs.pointbridge.com/Blogs/Sobiecki_David/Pages/Post.aspx?_ID=3

Map a Crawled Property to Managed Properties (FS4SP – PowerShell)

 

What we are doing:

1.) Create a Managed Property from a Crawled Property

Managed Properties are great! FAST pulls and indexes Managed Properties for multitudes of things, including boosting and search scopes. They are core to your FAST User Experience and an essential part to configuration

When a FAST crawl runs, it will pick up site column data and create a crawled property for that site column (given that there is data associated with this site column). In order to query this site column directly in search, first we will created a managed property from this site column.

Generally, most out of the box SharePoint site columns are already mapped to a managed property in FAST. However, custom site columns usually are not, and these are the ones that we need to target to map to a managed property.

First we will need to make sure that our site column was indeed picked up as a crawled property. In this example, we are using a custom site column called “country”. Make sure you have already executed a crawl and that there is data associated with this site column.

 

From PowerShell: Executing the command Get-FASTSearchMetadataCrawledProperty in the FAST PowerShell will return the entire list of crawled properties in FAST. This can get quite verbose in your PowerShell window, so if you would like to print that out to a text file for easier viewing, execute the following simple script in PowerShell:

1: Get-FASTSearchMetadataCrawledProperty | Out-File “C:crawledproperty.txt”

If you know the name of your column, you can shorten the search by adding –Name.

Get-FASTSearchMetadataCrawledProperty -Name 'country'

Basic Troubleshooting: If you get an error that the cmdlet Get-FASTSearchMetadataCrawledProperty doesn’t exist, make sure that you are FAST’s PowerShell. If the error continues, add the FAST PowerShell snap-in by typing in the command: Add-PSSnapin Microsoft.FASTSearch.Powershell

Troubleshooting

 

 

From the UI: In Central Admin, go to Service Applications –> <Your FAST Query SSA> –> FAST Search Administration –> Crawled Property Categories –> SharePoint. From here you can do a search to find your site column. In general, SharePoint site columns will be preceded with “ows_”, so our returned crawled property looks like this: ows_country().

countrycrawled

 

Creating Managed Property

Now that we know our site column exists as a crawled property, we need to create or find a managed property to map it to. In this case, we will simply create a new managed property.

From PowerShell:

Here, ‘-type 1’ is the type of information that we are storing in the property. Make sure you make it Queryable!

# Create New Managed Property
$nmp = New-FASTSearchMetadataManagedProperty –Name “country” –type 1 –description “Managed Property for country content type”
Set-FASTSearchMetadataManagedProperty –Name “country” –Queryable $true –StemmingEnabled
$false –RefinementEnabled  $false
$cp = Get-FASTSearchMetadataCrawledProperty –name 'ows_country'
New-FASTSearchMetadataCrawledPropertyMapping –Managedproperty $nmp –crawledproperty $cp

 

From the UI:Go to Service Applications –> <Your FAST Query SSA> –> FAST Search Administration –> Managed Properties. Click Add Managed Property. We will create a new managed property called “country”, type in a short description, and then click “Add Mapping”. This will bring up a menu where you will search for your crawled property. Once you find the managed property, add it and click ok. Finally, we will check “Query property”, which will allow you to use this managed property in search queries. There are also the options to make this managed property a sort property and deep refiner.addmapping

addingproperty

Either way we choose to create our managed property, we should be able to see it in our list of managed property names.

 

countryadded

 

Crawl it!

In order to use this new managed property in search and see filtered results, we must do a content crawl. Service Applications –> <Your FAST Connector> and execute an incremental or full crawl of your content source.

After your crawl, you can query directly on this column by using it as a scope in your search. For example “country:USA”.

scopedcolumn

Creating User Contexts, Customizing User Context Properties, and Site Promotions from PowerShell (FAST for SharePoint)

PowerShell is a powerful tool when combined with FAST. When dealing with a large site collection with multitudes of taxonomies and sites underneath it, promoting sites and keeping user contexts from environment to environment can be a long and taxing job, especially when you are working with the SharePoint/FAST UI. Luckily, this is where PowerShell comes in and saves your day.

User Context Properties

In SharePoint, you can create and edit user contexts through the UI. Under Site Settings –> FAST Search User Context, you will find all of your user contexts. The page to create a User Context looks like this out of the box:

OOTBUsercontext_001

Useful page, but not really right? If you want some more depth, you will have to do some digging into User Profiles. Go to Central Administration –> Service Applications –> <Your User Profile Service Application> –> Manage User Properties. Here you will find a list of user properties that are used in SharePoint, and these will be the key to making User Contexts and Site Promotions valuable and create a rich user experience for searches.

So first, let’s add a new property to the User Context. Let’s say that the User’s “Department” is an important factor in search relevancy. Therefore we want items relevant to the User’s Department to show up first in search results. To do this, we need to execute a couple PowerShell scripts.

View Current User Context Properties

First, let’s see what User Context properties are already available. If we haven’t touched them before, then we should see two of them, “SPS-Location,SPS-Responsibility”. We will be using the SharePoint PowerShell.

 

1: $properties = Get-SPEnterpriseSearchExtendedQueryProperty -SearchApplication “FASTQuery” -Identity “FASTSearchContextProperties” 2: $properties.Value

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

 

Where “FASTQuery” is the <Name of your FAST Query SSA>.

beforeuc

Add User Context Property

We need to make sure that we use the Name of the User Profile Property rather than the Display Name. This is especially important if we are using custom user profile properties. Also, we will need to include the old user context properties in this script, otherwise, they will be overwritten.

 

1: Set-SPEnterpriseSearchExtendedQueryProperty -SearchApplication “FASTQuery” -Identity “FASTSearchContextProperties” -Value “SPS-Location,SPS-Responsibility,Department”

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

 

 

 

 

 

 

 

 

Troubleshooting: Now if you go to your UI, often you won’t see any changes (the changes will be reflected in PowerShell however)! The fastest way that I’ve seen to get these changes to show is an IIS Reset. If there is a way around this, please let us know. Therefore, type in iisreset into your PowerShell, then check the UI after it has completed. It should now look like this:

afterDepartment

pcafterdepartment

 

Add User Contexts Using PowerShell

Now that we have some new properties to play around with, let’s add some User Contexts! In the FAST PowerShell, run the command Get-FASTSearchSearchSettingGroup. This will return the FAST Search Setting Group(s) that are on the machine. There should only be one straight out of the box. We will need the Name for the next step.

GSSGName

1: $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name “69d025ce-96a7-4131-adc0-7da1603e8d24″

2: $context = $searchSettingGroup.Contexts.AddContext(“USA Marketing”) 3: $andExpression = $context.AddAndExpression() 4: $orExpression = $andExpression.AddOrExpression() 5: $orExpression.AddMatchExpression(‘SPS-Location’,‘USA’) 6: $orExpression.AddMatchExpression(‘Department’,‘Marketing’) 7: 8: # Optional not Expression (remove #) 9: # $not = andExpression.AddNotExpression() 10: # $not.AddMatchExpression(‘SPS-Location’,‘Japan’) 11: # $not.AddMatchExpression(‘SPS-Location’,‘Canada’)

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

 

This next script will add a new User Context. The true usefulness of doing this in PowerShell is the added ability to use the “not” expression to truly filter your User Context. Besides that, why is this even useful over the UI? Mainly because it’s easily portable, and you can quickly execute your PowerShell script when it’s time to move from development to production. This is handy, especially if you have multiple User Contexts, but even more so when we begin adding Site Promotions.

addeducmarket

 

Add Site Promotions Using PowerShell

Handling site promotions and demotions can be a real hassle, especially when we begin dealing with an environment with many sites and sub-sites. Having PowerShell will make this process quick and portable from a development environment to a production environment.

Here, we begin talking about the use of “boosting” relevance and using Site Promotions. Here, we will be working with a promotion without using Keywords. This could be considered something closer to “scoping”, however the search will still extend beyond the scope of the sites included in the search. PowerShell allows you to choose your boost value automatically, assign multiple websites to your boost, and select the User Context(s) to include.

 

1: # This PowerShell Script Adds Site Promotions for USA Marketing 3: 4: # Register the FASTSearch Powershell Snapin (if needed) 5: Add-PSSnapin Microsoft.FASTSearch.Powershell 6: 7: # Current Search Setting Group 8: $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name “69d025ce-96a7-4131-adc0-7da1603e8d24″ 9: 10: # Type of Promotion 11: $globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword 12: 13: # Promotion Name 14: $globalPromotion = $globalPromotions.AddPromotion(“USA Marketing Sites”) 15: 16: # Boost Value 17: $globalPromotion.BoostValue = 12000 18: 19: 20: ######### Begin Site Promotions Section ######### 21: 22: $sitePromotionsToBeAdded = @( 23: 24: “http://intranet/USA/”, 25: “http://intranet/marketing/”, 26: “http://intranet/marketing/USA/”, 27: “http://intranet/HR/Marketing”, 28: “http://intranet/USA/Research/Marketing/”, 29: “http://intranet/USA/Support/Marketing/” 30: 31: ) 32: 33: # loop through site collection 34: foreach ($site in $sitePromotionsToBeAdded) 35: { 36: $uri = New-Object -Typename System.Uri -ArgumentList $site 37: $globalPromotion.PromotedItems.AddPromotedLocation($uri) 38: } 39: 40: # Add Site Promotion Individually (if needed) 41: # $uri = New-Object -TypeName System.Uri –ArgumentList http://intranet/USA 42: # $globalPromotion.PromotedItems.AddPromotedLocation($uri) 43: 44: 45: ########### End Site Promotions Section ########## 46: 47: # Added User Context 48: $userContexts = ({USA Marketing}) 49: $globalPromotion.Contexts.AddContext($userContexts)

 

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode, .ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{font-size:small;color:black;font-family:consolas, “Courier New”, courier, monospace;background-color:#ffffff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode pre
{margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .rem
{color:#008000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .kwrd
{color:#0000ff;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .str
{color:#006080;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .op
{color:#0000c0;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .preproc
{color:#cc6633;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .asp
{background-color:#ffff00;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .html
{color:#800000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .attr
{color:#ff0000;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .alt
{background-color:#f4f4f4;width:100%;margin:0em;}
.ExternalClass35C7684F8D3F46ACAC92B08CAAE6D498 .csharpcode .lnum
{color:#606060;}

PowerShell is a clean method of adding Site Promotions to specific User Contexts in FAST Search. If you check under Site Settings –> FAST Search site promotion and demotion, you should see your new site promotion with its User Context(s) attached.

sitepromotion

References: http://technet.microsoft.com/en-us/library/ff191225.aspx, http://blogs.technet.com/b/speschka/archive/2009/12/09/using-custom-properties-to-create-a-fast-search-for-sharepoint-2010-user-context.aspx

Start a SharePoint 2010 / FS4SP Crawl Remotely

With SharePoint 2010 and FAST Search For SharePoint 2010 (FS4SP), it’s easy to schedule crawls to run daily, hourly, or according to any other frequency. For most scenarios, scheduled crawls work perfectly.
 
Sometimes it makes more sense to kick off a crawl based on an event. For example, perhaps your organization runs an Extract/Transform/Load process to prepare data before being crawled. If that ETL job finishes at an inconsistent time, a scheduled crawl may either run too early and miss some updated data or run too late, making queries stale.
 
To fix that, we’d ideally kick off a new search crawl as soon as the ETL job is done running. With PowerShell, doing so is easy.
 
The PowerShell Script
 
$userName = “DOMAINserviceAccount”
$passWord = ConvertTo-SecureString “password” -Force -AsPlainText
$indexServerName = “serverName”
# Run the following commands on the remote computer
$credential = New-Object System.Management.Automation.PSCredential($userName, $passWord)
$session = New-PSSession $indexServerName -Authentication CredSSP -Credential $credential
Invoke-Command -Session $session -scriptBlock { `
Add-PSSnapin Microsoft.SharePoint.PowerShell; `
`
$indexServiceAppName = “Search Service Index Application”; `
`
$indexServiceApp = Get-SPServiceApplication -Name $indexServiceAppName; `
$contentSource = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $indexServiceApp `
$contentSource.StartFullCrawl() `
}

How It Works

The above script uses PowerShell remoting to issue requests on a SharePoint indexing server.
 
The following variables need to filled in:
  • $userName: The full username of an account with permissions to kick off a new search.
  • $passWord: The account’s password. Note that dollar signs need to be escaped with tick characters in PowerShell strings (e.g. “Pa`$`$word”).
  • $indexServerName: The name of a server running the index role.

An example usage is to run this script as part of a SQL job or SSIS step. The executable to call is “PowerShell.exe” with the above script saved in a “PS1″ file as the command’s argument.

Because SharePoint 2010 and FAST Search for SharePoint use the same service application architecture, this approach works for either system.

Tags:

Posted in Search

I Still Haven’t Found What I’m Looking For

SharePoint Search provides excellent search capabilities and features that many organizations can leverage with minimal customizations and configurations.Plus, with the addition of FAST for SharePoint and its big brother FAST Search for Internet Servers (FSIS), Microsoft has a deep search offerings to solve countless search opportunities.

However, even with features such as metadata-based results refinement and query suggestions, users of SharePoint search often need to climb the highest mountains and run through the fields, and sill not find what they are looking for (thanks, Bono).

Much of this is due to the flexible way SharePoint presents content, and how SharePoint crawls content.Let’s say you are searching for ‘Joshua Tree’ in your vast SharePoint-based library of discography built with an asset library of mp3 files with lookup columns into other custom lists to specify the Albums and Artists.By default, you’ll get back something like this:

At first glance, this looks great, but looking further you’ll see that SharePoint is returning duplicate results.Instead of just one ‘hit’ for the Joshua Tree album, we’re seeing quite a few – one for the album itself, one for each ‘View’ of the Albums list:All Items, By Artist, By Genre.In addition, because we have a multiple views in our Songs Library too, we’re getting multiple hits for each song as well.Once you’ve made this realization, the results appear ‘less great’ and frustration sets in.

Luckily, the solution is easy once you know what SharePoint is doing.When SharePoint crawls content, it is looking at each and every page of the site, indexing what it finds.The views we created to provide flexibility in navigating to our albums and songs by artist, genre, etc. are actually making our search results less effective.In addition, SharePoint is indexing the songs and albums themselves, so we end up with many search results, all pointing to the same thing.

Our solution is to configure search to ignore the types of results we don’t want displayed.In this case, we are only interested in the Albums and Songs themselves, not the lists that contain them.There are two ways in which we can configure SharePoint to accomplish this:Query Results or Crawl Rules

Query Results:

Every item displayed in search query has a Content Class.The Content Class indicates whether the result was found in a List, Library, Calendar, Page, etc., and is a search managed property so we can specify what Content Classes we want returned (or not returned) in our query.To test this, you can enter the following syntax into your query box:

Joshua Tree AND ContentClass<>STS_List_GenericList

Since the ‘Albums’ List was created as a Custom List, this query eliminates any result stemming from a custom list, thus eliminating the duplicate and unnecessary views from our search results, as shown below:

 

Further, we can also eliminate Picture Libraries be adding another clause:

Joshua Tree AND ContentClass<>STS_List_GenericList AND ContentClass<>STS_List_851

STS_List_851 represents a SharePoint 2010 Asset Library, and thus are query results are reduced to exactly we expect when performing our search:

 

So, now that we can form a query that will return the results we need, we can modify our search results page to automatically append our Content Class filter clauses to each and every query performed on the page.(A SharePoint Search Results page is simply a set of web parts that allow us to enter a search term and then display results, and filter on those results.)To do this, simply modify the ‘Results Query Options’ of the Search Results web part to include a query clause in the ‘Append Text to Query’ field as shown below:

 

Crawl Rules:

Modifying a Query Results web part is certainly easy and can provide some quick results, but when the number of results being returned is large, it is less than efficient to perform the filtering at query time; instead we should consider modifying our crawl rules to prevent these pages from ever reaching the index.

When specifying crawl rules, additional options are presented.Not only can we use the exact same query clauses to restrict results by content class, but we can also restrict results based on specific paths, such as:

*/Lists/Albums/By Artist.aspx, or

*/Lists/Albums/AllItems.aspx, or even

*/Lists/Albums/*.aspx

The exact techniques used to specify how crawl rules are created and managed are outside the scope of this blog, but details can be found here: http://technet.microsoft.com/en-us/library/ee792871.aspx

Office 365 Consideration:For those of us who are leveraging SharePoint Online in Microsoft’s Office 365 suite, specifying crawl rules is not an option as administering the Search Service Application is unavailable.

Summary

So, in summary, when the number of results is small as in this example, it is easy to understand the results and locate the correct link to follow, but when we think about the amount of content contained in most SharePoint implementations, it becomes critical to plan for how our search results will be returned and may the necessary adjustments to how content is crawled or how it is displayed to provide effective and relevant search results.

Tags: ,

Posted in Search

A couple of FAST for SharePoint Tidbits

This is a short post about two FAST for SharePoint topics that I have had trouble finding information about on the Web. I hope this post helps others who run into either of these issues.

The first topic is people search. It is well known that SharePoint continues to handle people search even when a web application is configured to use FAST for content search. The question I had a hard time finding an answer to is “how do you configure your service applications to crawl the profile store to support people search?” Configuring a SharePoint farm to use FAST requires the creation of two FAST search service applications. The first is the content or crawling application and the second is the query application. The content application is responsible for crawling content and passing it to the FAST server to be indexed. The query application passes queries along to FAST and then returns the results to the UI layer for display.

Which of these do you think you should configure to crawl the profile store for people search?

Although it is somewhat counter-intuitive (at least it is to me), it is the query service application that needs to be set up to crawl the profile store. You create your content source, crawl rules, and crawl schedule in the query application and, when a crawl is performed, it is this application that handles the indexing of the profile data. This is a completely separate index from the FAST index.

I have always thought it was strange that the FAST query application would have crawl related links in its left-hand navigation. Now I know why.

The second topic is security trimming of BCS content sources in FAST for SharePoint. BCS supports a method type called AccessChecker that can be used to check a user’s access to content at query time. This can be used as an alternative to including a WindowsSecurityDescriptorField on your BCS entity that provides security ACLs at crawl time. Both approaches are supported by SharePoint search. Unfortunately, only the crawl-time ACL-based approach is supported by FAST search.

I have received confirmation from Microsoft that the query-time access check is not supported by FAST, but I did not get any information on why this is the case. My guess is that it is because the query is processed by the FAST server which knows nothing about the BCS entity. The FAST server knows only about the data that it was provided by SharePoint and which it has stored in its index. Perhaps in the future, Microsoft could build a hook into the query service application that would allow it to do some additional trimming of the results returned by FAST before sending these results to the UI.

Tags: ,

Posted in Search

FAST Search Server 2010 for SharePoint: Creating a Custom Document Processing Step

FAST Search Server 2010 for SharePoint includes the capability to extend the document processing pipeline by adding additional steps. These steps are implemented as executable assemblies that accept, at minimum, two command line arguments: the path to an input file and the path to an output file. The input file is created by the document processing pipeline prior to calling the custom executable. It contains the crawled properties of the document currently being processed but is limited to those properties that are specified in the Input section of the Pipeline Extensibility configuration file. Using the input properties, the custom step executes whatever logic is necessary to generate the desired output and then writes the output to a file. The output file must contain the crawled properties that are specified in the Output section of the configuration file.

The executable runs in a sandboxed execution environment and, therefore, is limited in what resources it can access. For example, the executable cannot access any file system locations other than those passed in as the input and output paths. In my example, I will demonstrate that it is possible to connect to a SQL Server database using SQL Server authentication. However, I was not able to access the database using Windows authentication. In addition to accessing a database, the documentation from Microsoft indicates that it is possible to make web service calls from a custom step. I have not attempted to confirm this.

One challenge you will come across when trying to connect to a database or web service is how to pass in the connection information. One less than ideal approach would be to hard code this information into your assembly, but this is problematic for obvious reasons. A better approach is to use additional command line parameters and then specify the connection information in the configuration file. I will show this in my example.

Another challenge is knowing the identity under which the code will execute. I was able to confirm that the executable runs under the identity of the FAST service account. However, as I stated above, this knowledge did not do me much good since I was not able to use this identity to connect to a database or write to a file. Perhaps this identity will be more helpful in connecting to a web service.

The biggest challenge I encountered was debugging. My first thought here was to try to use the Visual Studio debugger. However, I was not able to find a way to attached the debugger to the executing process. My next thought was to write logging information to a file, but this does not appear to be possible either because of the limitations of the sandbox. I ultimately settled on writing logging information to a database table.

The only guidance I could find from Microsoft related to debugging mentioned two options. First, debug the executable outside of the document processing pipeline by creating an input file that matches the format specification. This is a great suggestion for ensuring that your logic does what you think it is supposed to do; however, it doesn’t help with observing what happens inside your code when it is running in the pipeline. The second suggestion is to set an exit code other than zero and write to the standard error stream. The message you write will be visible in the SharePoint Crawl Log.

In the example below I will walk through the steps to create, configure, and deploy the custom processing step as well as to utilize the managed property populated by the custom step as a search result refiner. My example showcases another feature of FAST for SharePoint called entity extraction. By default, the FAST document processing pipeline will attempt to extract entities that represent company names and place names. I am going to extend this functionality by mapping the extracted company names to the location of the company’s world headquarters. The company location will be stored in a new crawled property called CompanyLocation which is mapped to a new managed property also called CompanyLocation. I will use this managed property as a custom refiner on my search page.

Here are the steps to make this happen.

1) Create an executable that takes two arguments: an input file path and an output file path

I divided my code into three classes. The main Program class contains most of the logic; the DBLogger class helps with writing logging info to the database; and the CompanyLookup class gets the location value from the database. The code of all three classes is fairly straight forward so I will not explain it in detail. One thing I do want to point out, however, is the use of the u2029 character to delimit multi-valued crawled properties. Since the companies property can have multiple companies extracted to it, I need to split on this character when reading the companies and then delimit on this character when writing to the new managed property.

using System.Linq;

using System.Text;

using System.Xml.Linq;

namespace FASTCompanyLookupStep

{

class Program

{

static void Main(string[] args)

{

string input = args[0];

string output = args[1];

string connection = args[2];

DBLogger logger = new DBLogger(connection);

logger.Write(“This is a test to prove that I can write to a database table.”);

XDocument inDoc = XDocument.Load(input);

string[] companies = inDoc.Descendants(“CrawledProperty”).First(e => e.Attribute(“propertyName”).Value == “companies”).Value.Split(‘u2029′);

XDocument outDoc = new XDocument();

XElement docElement = new XElement(“Document”);

outDoc.Add(docElement);

XElement cpElement = new XElement(“CrawledProperty”);

docElement.Add(cpElement);

cpElement.Add(new XAttribute(“propertySet”, “1104E7BE-46CA-4b31-975F-4F37FB8303BE”));

cpElement.Add(new XAttribute(“varType”, “31”));

cpElement.Add(new XAttribute(“propertyName”, “CompanyLocation”));

StringBuilder sb = new StringBuilder();

CompanyLookup lookup = new CompanyLookup(connection);

foreach (string company in companies.Select(c => c.Trim().TrimEnd(‘u2029′)))

{

if (sb.Length > 0)

{

sb.Append(‘u2029′);

}

sb.Append(lookup.GetLocation(company));

}

cpElement.Value = sb.ToString();

outDoc.Save(output);

}

}

}

using System.Data.SqlClient;

namespace FASTCompanyLookupStep

{

public class CompanyLookup

{

private string ConnectionString { get; set; }

public CompanyLookup(string connectionString)

{

ConnectionString = connectionString;

}

public string GetLocation(string companyName)

{

using (SqlConnection conn = new SqlConnection(ConnectionString))

{

conn.Open();

SqlCommand cmd = new SqlCommand(string.Format(“SELECT Location FROM CompanyLocation WHERE CompanyName ='{0}'”, companyName));

cmd.Connection = conn;

return cmd.ExecuteScalar() as string;

}

}

}

}

using System;

using System.Data.SqlClient;

namespace FASTCompanyLookupStep

{

public class DBLogger

{

private string ConnectionString { get; set; }

public DBLogger(string connectionString)

{

ConnectionString = connectionString;

}

public void Write(string message)

{

using (SqlConnection conn = new SqlConnection(ConnectionString))

{

conn.Open();

SqlCommand cmd = new SqlCommand(

string.Format(“INSERT INTO Logging (ProcessName, LoggedTime, [Message], UserIdentity) VALUES (‘{0}’, ‘{1}’, ‘{2}’, ‘{3}’)”,

System.Diagnostics.Process.GetCurrentProcess().ProcessName, DateTime.Now, message, Environment.UserName));

cmd.Connection = conn;

cmd.ExecuteNonQuery();

}

}

}

}

After compiling the assembly, you deploy to the C:FASTSearchbin folder on all indexing servers.

The executable will work with an input file in the format below. The file will contain the crawled properties that are specified in the configuration file. If a crawled property contains multiple values, the values will be separated by the u2029 character. After performing the processing logic of the step, the executable will generate an output file in the same format as the input file which includes the crawled properties specified in the Output section of the configuration file.

<?xml version="1.0" encoding="utf-8"?> 
<Document> 
<CrawledProperty propertySet="48385c54-cdfc-4e84-8117-c95b3cf8911c" varType="31" propertyName="companies">test1test2test3</CrawledProperty> 
</Document> 

The executable will execute before crawled properties are mapped to managed properties.

2) Update C:FASTSearchetcpipelineextensibility.xml to add your custom step to the pipeline

The next step is to update the pipeline extensibility configuration file. This needs to be done on all indexing servers. The configuration file contains a <Run> element for all custom steps that are added to the pipeline. This element includes the command line to be used to call the executable as well as the input and output crawled properties. In our example, the output crawled property will be created in the next step.

The first two command line arguments are placeholders for the input and output file paths that will be provided by the pipeline. The third argument is the database connection string that will be used for logging and lookup.

<Run command="fastcompanylookupstep.exe %(input)s %(output)s Server=(local);Database=FASTCompanyLocation;uid=Test;pwd=Test;"> 
<Input> 
<CrawledProperty propertySet="48385c54-cdfc-4e84-8117-c95b3cf8911c" varType="31" propertyName="companies"/> 
</Input> 
<Output> 
<CrawledProperty propertySet="1104e7be-46ca-4b31-975f-4f37fb8303be" varType="31" propertyName="CompanyLocation"/> 
</Output> 
</Run> 

After updating the file, you will have to run

psctrl reset

from the command line to reset the document processors. Otherwise, the configuration changes will not take effect.

3) Run a PowerShell script from the FAST admin prompt to add new crawled properties and managed properties to the FAST configuration

The next step is to create the crawled and managed properties that will store the company locations for the document. The PowerShell script below will take care of this for you.

$guid = "1104E7BE-46CA-4b31-975F-4F37FB8303BE" 
$cat = New-FASTSearchMetadataCategory -name "Custom Pipeline Properties" -propset $guid 
$cp = New-FASTSearchMetadataCrawledProperty -name CompanyLocation -varianttype 31 -propset $guid 
$mp = New-FASTSearchMetadataManagedProperty -name CompanyLocation -type 1 
$mp.RefinementEnabled = $true 
$mp.MergeCrawledProperties = $true 
$mp.Update() 
New-FASTSearchMetadataCrawledPropertyMapping -CrawledProperty $cp -ManagedProperty $mp 

4) Create a new FAST Search Center site

Next, create a FAST Search Center site in a site collection that belongs to a web application that has been configured to use FAST as its search provider. Creating the search center will give you the ability to edit the web part properties on the search results page.

5) Update the Refinement Panel web part on the search results page to include your new managed property

The last step is to update the Refinement Panel web part to include your new managed property. Adding in the <Category> element shown below will include the CompanyLoocation property as a refiner. To do this, you need to update the Filter Category Definition property under Refinement. Be sure also to uncheck the Use Default Configuration property otherwise your changes will not take effect.

<FilterCategories> 
<Category Title="Company Location" Description="The location of the company" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" MetadataThreshold="1" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="20" ShowMoreLink="True" MappedProperty="companylocation" MoreLinkText="show more" LessLinkText="show fewer" ShowCounts="Count" />
...
</FilterCategories> 

To finish up my example, I created three Word documents in the Share Documents library to represent case studies about three different companies (Ford, Microsoft, and Starbucks). These companies also happen to be the same ones whose headquarters locations I have mapped in my database table. After running a crawl. I can execute a search on the phrase “Case Study” which is found in the title of the all three documents and my search results will include the Company Location refiner populated with the three headquarters locations.

 

One last thing to note. Your new managed property and refiner are available both via the search UI and the search web service. If you execute the following search against the web service using the QueryEx method, you will see the company location information come back as a refiner.

<QueryPacket> 
<Query> 
<Context> 
<QueryText>Case Study</QueryText> 
</Context> 
<IncludeRefinementResults> 
<Refiners> 
<Refiner>companylocation</Refiner> 
</Refiners> 
</IncludeRefinementResults> 
</Query> 
</QueryPacket> 
The results XML for the refiner will look something like this: 

Tags: ,

Posted in Search