Microsoft

Blog Categories

Subscribe to RSS feed

Archives

Follow Microsoft Technologies on Pinterest

Troubleshooting Search in SharePoint Online (O365)

SharePoint-logoSharePoint makes great use of its Search engine and it is pervasive throughout most SharePoint solutions today. Whether you are building Content Search WebParts leveraging Display Templates or creating a custom Search center you will be making use of SharePoint’s powerful and mature Search engine. However, what happens when things behave unexpectedly? How can you troubleshoot Search? This article will focus mainly on SharePoint Online (O365) but could also be applied to Search on premises.

Search is not magic!

When supporting others in their troubleshooting of Search issues I usually start by explaining that Search is not a black art. I would agree there are nuances but, for the most part, figuring out issues can follow simple scientific process. After all, SharePoint Search is just a collection of properties stored for query and retrieval by the Search engine.

The perception that Search is a black art often extends to users and their expectations. When search does not yield the results a user expects we should always investigate what they expect and why they expect it. Work directly with end users and try to help them understand why they are not getting what they expect.

Approach to Troubleshooting

Generally speaking it is advisable to try and make small but smart moves when changing configuration. It can sometimes take days or weeks to test the results of your configuration changes if long re-crawls are required. Above all, be scientific! Use the same test cases before and after changes and measure your results by tracking search results before and after.

The two sides of Search

There are two sides to Search, Crawl and Query. Potentially either side could be failing or not working as expected so it is important to investigate both.

SharePoint Online in O365 does not currently offer too much control nor insight into the Crawl side (this may change in future). SharePoint 2013 on premises offers lots of opportunity to troubleshoot the crawl. On premises we can:

  • Inspect the crawl log
  • Turn on verbose logging during crawls
  • Attach crawls to Fiddler

The Query side can be inspected almost equally online as it can be on premises. It’s important to understand that Search queries are transformed as they are passed to the search engine. The user can type a query into a search box but there are number of places the query could be transformed or results influenced by search configuration:

  • Search Results WebPart e.g. in a Search Center
  • Content by Search WebPart
  • Results Source
  • Query Rules
  • Synonyms

Inspect the Query using the Query Builder

There are at least two great ways to inspect the Query side of Search.

One, use the OOTB Query Builder to help you. From a Content by Search or Search Results WebPart you can open the Query Builder. Switch to Advanced Mode (you are most definitely Advanced now!) and then head over to the TEST tab. Once on the TEST tab click ‘Show more’ to see the full transformation of the query in the bottom text box (highlighted).

QueryBuilder

Seeing the full transformation of the query is important because there may well be transformations impacting results you had not thought of. In addition if any part of the query is using dynamic property values based upon the Page or User (e.g. {User.ProfileProperty}) these values will be expanded and displayed for inspection.

Inspect the Query using the 2013 Search Tool

The SharePoint 2013 Search Query Tool is now an essential part of my day-to-day work with SharePoint Search. It uses the REST API to retrieve results from SharePoint Search and can be used with SharePoint Online (O365) and On Premises SharePoint 2013.

http://sp2013searchtool.codeplex.com

The tool provides a fast, convenient and repeatable way to inspect Search results. A few tips for using this tool most effectively:

  • Be scientific! Use fixed queries, analyze results before and after configuration changes.
  • Turn off Trim Duplicates, it will be ON by default. Trimming duplicates can be really confusing when analyzing results, especially when you have repetitive test data in a system.
  • Ensure you are using the correct Results Source. By default this will be set to Local SharePoint Results. However, if custom Result Sources are in play be sure to set them in the tool. You will need the GUID identifying the Results Source, this can be obtained easily by browsing to the Results Source in SharePoint and grabbing the ID from the querystring.
  • Inspect the Rank values returned. The rank values will determine which results appear first. If any parts of the transformation use XRANK to boost results then this should be evident in the Rank value.

Check the basics

Are you writing valid Keyword Query Language (KQL)? SharePoint Search can only understand KQL and not syntax from other search engines such as Google, Solr, Westlaw, Lexis etc.

http://msdn.microsoft.com/en-us/library/office/ee558911.aspx

If the user is expecting a specific document to be returned has that document actually been indexed? This can be verified by performing a Path search e.g.:

Path:http://contoso.sharepoint.com/documents/document.docx

On Premises you could also check the crawl log to see if the item hit an error during indexing.

If the search query is expecting to hit document body content you should also check that the source text is readable by the search engine. In the case of PDF documents, they will need to be OCR’d.

Managed Properties

It’s likely that Search is not behaving the way users expect due to Managed Property configuration. Managed Properties are really the backbone of the search engine and our greatest opportunity for customization. They are also our greatest opportunity to make mistakes in configuration. We need to check the Managed Property settings. This article does not fully explore Managed Property configuration but here are the basic settings we should consider when troubleshooting:

  • Searchable – Enables us to query against the content of the property, the content is stored in the full-text index. On premises we can also set the ‘Context’ of the property which will allow different weighting to be applied.
  • Queryable – Enables us to query directly against the property e.g. Title:Hines . Note that this is different to simply being included in the full text index.
  • Retrieveable – Enables us to retrieve the value of the property in search results. If a property is not retrievable it can sometimes be confusing when trying to inspect it using the search tool. The value may be Queryable and used in queries but not retrieveable so that we can inspect it.
  • Refinable – Means that it can be used in a search by refinement. Note that this is different to querying properties.
  • Sortable – Means that results could be sorted by this property.
  • Mappings – Managed Properties will be mapped to one or more Crawled Properties. This relationship is worth reading more about and the necessity to re-crawl after changing these values in important to consider. When more than one Crawled Property is selected be sure that the configuration is resulting

It’s really important during any Search project to have consistent and well-understood Managed Property configuration. At Perficient we often build a Managed Properties Specification which allows us to collaborate on how properties will be configured. In addition it is highly recommended to script Managed Property configuration so that Development, QA and Production environments are the same. It’s very easy to let this slip and have misconfigurations confuse you when testing across multiple environments. More on script deployment to O365:

http://blogs.perficient.com/microsoft/2014/09/powershell-deployment-to-sharepoint-online

OOTB Title Managed Property

The ‘Title’ Managed Property is probably the single greatest point of confusion I see in SharePoint Search projects. Looking at the Mapping configuration it is already apparent that it’s complicated.

TitleManagedProperty

Unfortunately the MetadataExtractorTitle has, in my opinion, only added to this confusion. If your users are relaxed about seeing an interpreted Title in search results then you will most likely not get feedback. However, it appears most users get confused about this and require us to troubleshoot exactly why their Title is not displaying as expected in Search results. In response to this, you can of course look at the Crawled Property ordering and try to determine exactly why the Title Managed Property is populated as such. However, if it transpires this is a major problem for users I would look into creating your own separate Managed Property for handling Title.

Low level Debugging

Detailed debugging during a crawl or query is reserved for SharePoint on premises. During a crawl we can turn on VerboseEx logging and analyze the detail of what’s happening during acquisition. This could point to a number of problems including communication with the source during a crawl or an issue with security trimming when querying.

If you need to pursue a Search support incidents with Microsoft it is most likely Microsoft will ask for VerboseEx log. The only caution with this is that VerboseEx will grow the logs very quickly so it’s recommended you ensure it is only enabled for a short period of time.

Good luck!

SharePoint Search does have its challenges but it is also extremely powerful and offers great patterns for surfacing content. We see it making its way into new areas and services in Office 365 all the time. Embrace it! I think it’s here to stay.

Video Comes to Office 365 and SharePoint Online

This week, Microsoft announced the release of an Office 365 Video Portal (see Office 365 Video ).

This is an exciting first step into an area of great demand and large potential. In the past, many larger enterprises purchased dedicated 3rd party solutions for the management of video content. Smaller organizations typically leverage You Tube.  SharePoint deployments at small and mid-size companies sometimes use native Video content types and/or custom solutions.

The Office 365 Video portal goes above and beyond SharePoint video content type in many important ways:

  • The Video Portal is somewhat akin to the “old school” SharePoint Document Center.  The Video Portal is a dedicated “home” Site Collection with support for securable “channels”.  Each channel can have completely independent permissions and administrators and is mapped to a unique site collection.
  • Uploaded videos are converted for playback on a variety of formats/media/qualities.  At playback time, the video portal dynamically adjusts the playback (every 2 seconds) to use the most appropriate format for the device/bandwidth conditions.
  • The original upload is managed in a SharePoint site collection (and incurs storage cost); the converted files are managed via Azure Media Services and do NOT result in additional storage charges
  • Videos will appear in Enterprise Search results and views are tracked in analytic reports.

Its important to note that this is a v1 product with the following areas for future improvement:

  • Playback is via Flash, not HTML 5.  This represents a large roadblock for Mobile device use.  Microsoft acknowledges this and HTML 5 support is clearly on the roadmap
  • Uploaded videos are limited to 2GB with larger limits (aka OneDrive consumer version) undoubtedly in the works
  • Videos live in their “host” site collection and are not  integrated across all tenant site collections via apps; simpler link sharing/embedding is possible
  • Extranet and/or Intranet use cases are not currently supported

 

Even with the v1 caveats, this will be a great value added for many Office 365 customers.  The feature can be enabled/disabled at the tenant level, so organizations will have time to evaluate the ROI and launch when appropriate.

 

 

Office 365 – How to Handle “Large Messages” During Migration

Exchange Online provides for a fair amount of flexibility; that said, there are a few aspects of the service that cannot be changed. These service restrictions are documented in the Exchange Online Limits for each of the subscription types.

One limitation that you may encounter when migrating to Exchange Online is the “Message Size Limit” of 25 MB. Depending on your current on-premises limit, mailboxes may contain messages that exceed 25 MB; these “large messages” will cause issues during your migration. Your first encounter with large message may be a result of the error “This mailbox exceeded the maximum number of large items that were specified for this request” during a mailbox move.

There are a number of methods to address large messages with each method causing a varied level of impact to your users. After determining that there are large messages in your environment, one of the first questions to answer is what do you want to do with them?
Read the rest of this post »

Hybrid Analytics in Healthcare with O365 & Power BI Webinar Recap

Last week, we had our Microsoft last business intelligence focused webinar of the year, “Hybrid Analytics in Healthcare: Leveraging Power BI and Office 365 to Make Smarter Business Decisions.”  Heidi Rozmiarek, Assistant Director of IT Development for UnityPoint Health, spoke, along with our Microsoft BI team, on implementing an analytics platform in a hybrid environment. WebinarReplay

First, the Perficient team covered architectural components and functions, architecture options including on premises, hybrid, cloud,  and delivery considerations. Next, Steven Gregor, a technical consultant on our Microsoft BI team, reviewed Power BI and its features, including the security model and client side Data Management Gateway, and then walked through a live demo.

Last, Heidi shared how her organization is architecting a successful analytics infrastructure using Microsoft technologies. She explained how UnityPoint Health is leveraging Microsoft’s BI stack to provide simple solutions for complex questions. Heidi shared how they built the solution, collected and cleansed the data, modeled the data, and visualize and report the answer. She wrapped up by sharing her organization’s plans to move further to a hybrid on-premises/cloud solution in the next few months. Read the rest of this post »

All About ‘Clutter’ – Home Run for Microsoft Office Graph

We all receive email that we may have signed up for (such as a blog posts, newsletter) and that isn’t exactly junk, but is less probable to get our attention. Clutter uses the intelligence of Office Graph to see how important (or unimportant) this email is to you. It learns over time your levels of importance, then uses that analysis to separate the clutter from other inbox items.  You can quickly scan the clutter, mark individual items as “not clutter,” and take action on the rest of it, such as deleting it all. And if you don’t like the feature, you can turn it off.

Statement from office team – “Clutter learns from your actions to determine the messages you are likely to ignore,” . The underlying idea is quite simple: Microsoft wants to leverage the knowledge it gains about your daily work activities to automatically filter out emails that don’t immediately need your attention.

The flow works something like this : First emails considered as SPAM are rejected before its delivered to your mailbox. Then if a message lands in the inbox, it gets routed through the rules, and eventually Clutter does its magic. As a result, Clutter will get smarter over time, by learning your prior actions with similar messages, and determining the type of content and even how you are addressed in the message. This means the Clutter experience is personalized to each individual, based on their actions and their preferences.

Here is a two step process to turn ON Clutter for your Office 365 inbox. Remember it’s a individual user setting, and currently no way to automate the process for multiple users.

Clutter01

Clutter02

After turning ON Clutter, you should receive a welcome email similar to this

Clutter03

We’ve seen how you get this feature for your inbox, now let’s dig a bit deeper to understand the various facts which may help you.

SOME INTERESTING & FUN  FACTS

  • Available to all SKU’s

  • Clutter works in all clients (in terms of moving messages to the clutter folder and learning from your behavior), though it currently requires OWA to turn it on.

  • Clutter is only a feature on individuals mailboxes.

  • Not available for Office 365 groups mailbox.

  • In order get the Clutter feature earlier, enable your Office 365 tenant to receive First Release.

  • It is an Exchange feature, not a client feature. That means in mobile or desktop no special client is required. They will see Clutter emails in their Clutter folder. The items will not show up in their Inbox. Clutter is available in all mail clients that can display and sync folders from your mailbox. i.e. Smartphones, tablets, etc.

  • If the user is not reading the Yammer notification emails, then Clutter will see that they are not important to that individual and will move them out of the inbox into the Clutter folder. You will need to drag the Yammer notification email into their Inbox folder, and then Clutter will learn that messages from Yammer are important and will leave them in your Inbox in future.

  • Clutter is a user preference feature, similar to electing to use conversation view. No current plans to add clutter specific admin controls, however Microsoft is looking at how to add broader tenant rollout controls.

  • Clutter is trained approximately once per day. Moving items out of Clutter is learned during the next training cycle.

  • You can train Clutter from your phone.

  • Clutter looks at the folders at the Exchange level, independent of client.

  • Clutter is only available in online version. It is not planned for next version of Exchange Server. Clutter requires rapid feedback to fine-tune the machine learning which isn’t possible in an on-prem deployment and also requires additional server processing resources that would impact the sizing for the on-prem server.

  • Q: What is the processing order between Junk Mail, Clutter and custom rules?
    A. Junk Mail first, then custom rules, then Clutter. (AFAIK)

    Q: Can we have a rule which force something to go to clutter / not go to clutter?
    A. Yes, but if you force a message into Clutter it won’t go through the Inbox and therefore won’t be considered as a signal for learning your preferences

    Q: Can we have a rule which operates on e-mail which does arrive in Inbox/Clutter only?
    A. All rules operate on inbound mail destined for the Inbox unless redirected by a rule. Clutter currently applies after all server-side rules. Clutter is not aware of client-side rules. There is a trick that allows you to force an item to not go through Clutter processing. You can create a rule which moves the item to a folder or your Inbox and Clutter won’t be processed on messages which that rule applies to. You can’t control the order in which rules and Clutter run. Today Clutter will always run after your last rule.

    Q: Can we create a rule and specify when it should be processed (before/after Junk Mail, before/after Clutter)?
    A. No

  • Q: I added clutter yesterday afternoon and now most of my mail goes to clutter?
    A: Check for – If you get a lot of circulars and non-personal email that could be regard as clutter. Possibly most email in your inbox was left marked unread before enabling Clutter

  • Q: Does Clutter learn from the Content / subject of an email to classify?
    A: Yes. Move the Internal communication email into Clutter and leave the Sales email in your Inbox and all will be well.

  • Rules vs Clutter

    If you want to apply that level of precision, use rules. Or turn them off to let Clutter do its stuff… but accept that all of the messages will go into a single folder.
    Clutter only learns from your choices, as everyone prioritizes their inbox differently.

  • Diff between JUNK and CLUTTER

    As an example Clutter can learn which internal distribution lists you read and which ones you don’t and will move the ones that you don’t read to Clutter. Junk Email doesn’t do that. You can think of junk email as being something that is not personalized – junk for one person is junk for others. Clutter on the other hand, is highly personalized – it’s the set that you tend to ignore, but others may not. junk is something that you really do not want to see in the future. Clutter is email that you will probably want to look at, but not as a high priority activity.

  • If you are one of those users who select multiple messages and mark them as read then clutter will not learn from any signal. You need to either leave messages unread or move them to Clutter folder (via drag drop or right click move to clutter) or yet delete them as unread to generate a clutter signal. Microsoft is looking at improvements for the model that will learn from users who mark everything as read.

  • If you are using Clutter nothing has populated the folder then try moving some messages to the Clutter folder and it should get signals.

  • Clutter only works on emails which are going to your Inbox. If a rule moves the email first then Clutter won’t move it or train on it later.

 

Source: Office 365 Community Network; Microsoft.com; Office Blogs

Anglebrackets Conference – Day 4

Keynote – ASP.NET vNext and you
Speaker: Scott Hanselman

ASP.NET will run anywhere (Mac and Linux). Web server will be included.

New free SKU of Visual Studio. Community edition will replace Express.

image_dbbecd7b-9298-4dde-993a-acd9d9461515ASP.NET and modern web

  • Totally modular (dependency injection built in)
  • Faster development cycle
  • Seamless transition from on-premises to cloud
  • Choose your editors and tools
  • Open source with contribution
  • Cross-platform
  • FAST

Framework (CoreCLR) is packaged together with application, not shared, safe to deploy.

Split between .NET Framework (Full CLR) and .NET Core (Core CLR). Core CLR is open sourced and cross-platform.

Project file (project.json) is replacing both web.config and nuget package file).

It’s possible to reference ASP.NET is source form and debug it.

VS supports Bower (client-side package manager) and Grunt (client side build tool).

Read the rest of this post »

Office 365 and Salesforce: Integration Case Study Part II

o365

salesforce1

Given the central role that Office 365 occupies for more and more businesses, integration of the resources managed by Office 365 with other services is a challenge that Perficient often addresses for clients. The good news is that the Office 365 platform and the architecture of many other, key platforms provide countless integration possibilities, many of which can be leveraged without the need for custom coding.

Recently, I was asked to tackle an integration of Office 365 and Salesforce that serves as a good illustration of the possibilities.  I described the problem and solution in an earlier post . In that post, I described how to expose Office 365 information within Salesforce.  In this post, I would like to look at the problem from the opposite direction – how to expose Salesforce information within Office 365.

The Problem

In this case, the goal is to expose core Salesforce information within an Office 365 site.  Ideally, the information would behave as if it was “native” to the site.  In this case, “native” means web parts, lists, columns, etc. could all behave as expected.  This integration could, of course, be accomplished via creation of an appropriate SharePoint application.  As in the previous case, we are looking for a “no code” solution.

The Solution

Fortunately, Office 365 supports a technology designed to solve such external data integration scenarios – Business Connectivity Services (BCS) .  This technology was originally developed as a component/service for On-Premises SharePoint but is also supported within Office 365.

BCS supports the notion of an external content type, which can be used to describe Salesforce entities in a manner which will allow SharePoint to present the data as if it were internal.  Some of the information contained in a external content type for Salesforce includes the following:

  • Connection / Authentication information for the source Salesforce Organization.
  • A definition of the Entities, Fields, and data types (aka metadata).
  • Allowed data operations, such as Create, Read, Update, Delete, and Query (also called CRUDQ).
  • The identity field and display columns for an external content picker used to retrieve external data throughout the user interface.

As in SharePoint On-Premises, the BCS in Office 365 provides a means for importing the external content types definitions (see Import BDC Models):
bcssmallmarked

With the external content type in place, all the Salesforce-specifics can be ignored by users.

Creating a BDC Model

In addition to the external content type definition, the BDC Model for Salesforce must contain information about how to connect to the Salesforce data. BCS can consume data sources that are exposed as WCF services, SQL Azure data services, OData endpoints, and web services; Salesforce provides Web Service API for external data access. So, the question is “how do we match the requirements of the BCS client and the Salesforce service?”

In the case of Salesforce data, Visual Studio and SharePoint Designer tools do not provide a straightforward integration.  The good news is there are a number of 3rd party tool providers who solve this problem – e.g. RSSBus Salesforce Connector, BCS Meta Man .  These tools provide an GUI-based tool for the generation of the BDC Models and External Content Types. Under the covers, these tools provide an OData proxy for Salesforce web services.

 

Dealing with Security

Another important consideration of a Salesforce  integration is respecting security of the underlying sources system.  What is needed is some mechanism of associating the Office 365 authenticated user with an appropriate Salesforce user.  Fortunately, such mapping is supported by the Office 365 Secure Store service.  After determining the best authentication/identity mode for Salesforce, a target Secure Store application is created containing desired credentials mapping (see below).  Finally, the Secure Store application is associated with the Salesforce BDC Model.

securestoresmall

The Payoff

Once the Salesforce external content types are defined, a wide variety of SharePoint  elements can be used to create lists, columns, web parts, etc.  using Salesforce data.  See Salesforce demo for a quick example.

 

 

Posted in CRM, SharePoint Online

Anglebrackets Conference – Day 3

I have been fortunate to attend this year’s Anglebrackets conference in Las Vegas. (See my coverage of Day 1 here and Day 2 here.)

The following are my notes from the Day 3 keynote, “Conversations with Microsoft.”

image_dbbecd7b-9298-4dde-993a-acd9d9461515The speaker: Steve Guggenheimer, corporate VP and chief evangelist.

A lot of changes at Microsoft over last year. New CEO, Satya Nadella

Not changed: We always be partner-led

Our Core – productivity platform for mobile-first, cloud-first world

4 engineering units at MS

  1. Devices platform.

Converging Xbox, Windows , WP and RT into single platform. Universal apps and One Windows.

VS 2015 preview is out. Demo: creating universal app from template.

  1. First-party devices (Surface, Lumia phones, Band, etc).
  2. First-party services (Office 365, Bing, Yammer, Skype, et). Cross-platform APIs and SDKs.
  3. Azure 

Microsoft Developer approach:

  • Innovation
  • Agility
  • Openness

Demo: VS2015 – targeting multiple devices with Apache Cordova plugin for VS. Using Android emulator for VS.

Demo: using remote IE (Azure service for testing on IE) on Mac, iPad and Android tablet.

http://remote.modern.ie

Announcement: Open sourcing .NET Core Runtime and Framework. .NET running on Linux and Mac. Free VS 2013 Community edition.

Read the rest of this post »

Power BI Basics Inside Office 365 – A Video Series

Yesterday, we were fortunate to have a customer, Heidi Rozmiarek, Assistant Director of IT Development for UnityPoint Health, speak alongside our Microsoft BI team for the webinar, “Hybrid Analytics in Healthcare: Leveraging Power BI and Office 365 to Make Smarter Business Decisions.” power-bi

It was an informative session that began by covering architectural components and functions, architecture options including on premises, hybrid, cloud and delivery considerations. Following this, we had a live Power BI demo, and last but not least, Heidi shared how her organization is using the Microsoft BI stack to provide simple solutions for complex questions. Keep an eye out for a post describing the webinar in more detail, but in the meantime, you can view the replay here. 

Whether or not you attended the webinar, if you are interested in learning more about building a hybrid analytics platform with Power BI and Office 365,  I highly recommend you take a look at the following short video series.

  1. Introduction to Power BI:  The first video includes an introduction to Power BI, particularly around Power BI Sites, “My Power BI” and the Power BI Admin page.
  2. Administration and Permissions in Power BI: This video focuses on Site Admin and security basics.
  3. Data Exploration and Visualization in Power BI: The third video in the series discusses data exploration and visualization using Excel and related power tools, including Power Pivot and Power View.
  4. Data Management Gateway for Power BI: Here, we cover the steps to enable data feeds in Power BI using the Data Management Gateway.

.NET 2015 is coming! And then some

Yesterday in New York at Microsoft’s Connect(), a virtual event for developers focused on current and future technologies, Microsoft did what a lot of us in the developer community have been waiting for and  [finally] announced that the next .NET release is coming!

But wait, as the late night TV pitchman says, there’s more. There was an announcement from Microsoft Corporate VP of the Developer Division, Soma Somasegar and Miguel de Icaza of the Mono Project that the “open sourcing the full server-side .NET Core stack, from ASP.NET 5 down to the Core Runtime and Framework”. Visual Studio will be opened to all developers including those developing on Linux and OS X platforms with Visual Studio Community 2013,  a new, free and fully featured edition of Visual Studio. Both the open source .NET project and Visual Studio Community are available for download today.

Just think, it’s now possible to write apps that have a Windows UI, but run on a Mac natively .  >)  (Yes, that is an evil smiley).

Here  is the announcement on  Soma Somasegar’s blog.