Search is something we all use nearly every day, and it modifies our lives for the better. In less than 20 years since the public launch of Google, how we find information has changed dramatically. Thanks to modern search, we don’t have to remember facts as much because we know we can “Google it,” and we spend significantly less time researching questions because someone somewhere has answered it online. In as little as a few keystrokes and a click, modern search has transformed how we find what we are looking for and the speed in which we can make decisions and take action.
However, many organizations are nowhere near that same level of ease, speed, and accuracy when delivering results. Whether it’s searching for a product on an ecommerce site or trying to find information within a website, most people can think of a negative experience in searching on sites outside Google. With how advanced Google and other modern search tools are, these experiences stand out even more and lead to both consumer frustration and loss of business revenue. So what is different? Simply put, the answer is Modern Search Solutions.
What is “Search”?
First, it’s important to consider that “search” is more than search- that is to say, “search” is more than simply typing keywords into a search bar on search engines or a website. While that is one aspect most easily identified, “search” also encompasses much more. Consider a personalized home page for your favorite brand – dynamic content is displayed based on each unique user and their internet footprint. While a user might not be searching in a traditional sense, “search” is used to display the best fit products or content based on your history with that brand. Another component of “search” includes recommendations. This becomes relevant in display ads, for example, utilizing the same set of information to deliver best fit ads for every internet user, but is also powerful for driving next best action within a company’s experience to keep users engaged and traversing the funnel.
While the early days of web searching would only consider keywords (known as keyword search) when determining what content to show users, modern search uses far more information to deliver highly curated content based on who the user is. While keyword search is important for users, modern search has evolved and incorporates many other factors to determine a best fit. The magic, simply put, is in a layered approach, incorporating a variety of techniques leveraging data and machine learning to refine and optimize for the current individual.
Levels of Search
Keyword Search
The first level of search, most regarded and previously mentioned, is Keyword Search. Keyword search compares the words in content or products with the words entered by a user in a search experience to derive relevance. Optimizations for increase matching (recall) are performed using a variety of techniques, including: stemming, stop words, synonyms, spell-check, Boost and Bury, and Taxonomy + Ontology. Results are typically delivered based on the BM25 Algorithm, utilizing factors such as:
· How many times is a word found within a given document?
· What total percentage of a given document does a word make up?
· If two words are searched, on average, how close are they together?
While keyword search is important in determining results, the most advanced sites layer other search factors in.
Semantic Search
The next level up from keyword search is semantic search. This layer captures and compares the concepts of each result to derive relevance. Consider the following example: it’s winter and you are searching for a “warm jacket”- semantic search understands what this concept is and will deliver the result of a “winter coat” even though those were not the keywords input.
This is accomplished by using Machine Learning to capture the details of entire documents of products into “vector embeddings” that live within a vector datastore index. Searches are then accomplished by creating a “vector embedding” of a search query, and then comparing that with potential results in the vector datastore to find the closest matches in the index.
Behavioral Search
The next level up in modern search is behavioral search. Behavioral search leverages external signals and analytics to derive relevance. By continually capturing user behavior, contextual insights, and macrographic trends, AI models empower feedback loops to continually optimize results based on outcomes.
A simple example is to open Google and type in “shoes” and compare your answer with someone “different” than you. A 24-year-old female’s search results will look very different than a 50-year-old male’s results because of behavioral search. Modern search engines can leverage external signals to predict the best fit, regardless of how simple the search input is.
Incorporating the concept of digital twins, signal-driven interfaces are frequently personalized, providing each user with a tailor-made experience.
Generative Answering
The most recent evolving level of modern search is Generative Answering. Generative Answering leverages external large language models (LLMs) to generate answers using both knowledge spanning multiple documents along with the inherent knowledge of the LLM. This technique enables direct answering of complex questions based on your own knowledgebase, rather than the traditional approach of providing 10 blue links, and enables entirely new integrations with virtual agents that can dynamically learn from company content, and “concierge” shopping experiences where a conversational advisor can identify optimal products based on factors such as compatibility or matching styles –think: “find me shoes that match this dress” or “find me wines that would be good with this dinner”.
Retrieval Augmented Generation is a novel and growing technique where traditional search is seamlessly used to gather a set of information that can be used at query time to inform and constrain an answer generated by an LLM. This is the same technique used by Bing as they integrate ChatGPT, or ChatGPT’s browsing plugin in OpenAI’s commercial product. In an enterprise context, Retrival Augmented Generation provides numerous benefits, as it significantly reduces hallucinations, enables cited answers, and respects data security to generate information off privileged content.
What’s Next?
Modern search solutions can be implemented in a variety of ways to improve ecommerce ROI, improve back-end service providers, and drive real results. We will be exploring different use cases in this series to see how real businesses have seen measurable results after implementing modern search solutions.
At Perficient, we focus on search as a holistic discipline around how user experiences and journeys can be improved and optimized by providing immediate access to relevant data. With 100+ search implementations and counting, we help your customers and users find what they’re looking for faster. Learn more about how our team can help you provide a relevant search experience.
