Can I still use Examine with Umbraco?
I’ve been asked the same question a few times lately: When Umbraco Search replaces all things search in Umbraco, is Examine still a viable option?
Of course, it is!
If anything, Examine is better than ever, because Umbraco Search ships with the new and improved Examine 4 👏
Examine under the hood
Perhaps more to the point, Examine continues to power search with Umbraco Search.
Umbraco Search is search technology-agnostic by design and uses a provider based approach to make the underlying search technology swappable, but Examine remains the default (and only) search provider from Umbraco HQ.
Using Examine with Umbraco 19+
Umbraco Search is a part of Umbraco 19, which means it replaces search as we know it in the core CMS.
It is still entirely feasible to keep using Examine to power your site search beyond Umbraco 19. The core mechanics of Examine have evolved, but the feature set from Examine 3 remains intact.
What really changes is the index format, because Umbraco Search requires a vastly different index format than the trusty old ExternalIndex. As it happens, the ExternalIndex won’t even exist in Umbraco 19 and beyond 😱
The Umbraco Search index format
The indexes built by Umbraco Search are comprised of:
- System fields like name, content type, last update date and so on.
- Property values mapped by the property editors.
This is rather different from how indexes have traditionally been structured in Umbraco… but at the end of the day, it’s all just a bunch of fields in a search index, and you can still use Examine to query them.
The trick is really to figure out where your index values went.
Index values and the field naming scheme
With Umbraco Search, a single index field is subdivided into specific supported value types - text, keyword, integer, decimal and date. Text is further subdivided into four different buckets for relevance ranking.
In the Examine search provider for Umbraco Search, this all comes together in the field naming scheme Field_[field name]_[value type postfix], where the value type postfix is either texts, keywords, integers, decimals or datetimeoffsets. The text subdivision for ranking furthermore adds textsr1, textsr2and textsr3 in order of relevance.
Property values are added using their property type aliases as field name, whereas system values have specific field name annotations.
Using the links above, you can calculate the index field names for both system fields and fields generated by the built-in Umbraco property editors. Or, you can continue reading 👇
Listing field names directly from the index
The field naming scheme definitively takes a little getting used to.
If you happen to have Luke (or one of its .NET ports) available, I recommend peeking directly into the Examine index to find the available fields, at least until you get the hang of the naming scheme. You can also code your way around it 🤓
Here’s an API controller you can dump into your site, to list out the available fields in the default “published content” index for all content matching a specific textual query:
using Examine;
using Microsoft.AspNetCore.Mvc;
namespace Site.Controllers;
[ApiController]
[Route("api/[controller]")]
public class DebugController : ControllerBase
{
private readonly IExamineManager _examineManager;
public DebugController(IExamineManager examineManager) => _examineManager = examineManager;
[HttpGet("index-fields")]
public IActionResult IndexFields(string query)
{
if (query.IsNullOrWhiteSpace())
{
return BadRequest("No query");
}
if (_examineManager.TryGetIndex(Umbraco.Cms.Search.Core.Constants.IndexAliases.PublishedContent, out var index) is false)
{
return Problem("Index not found");
}
var result = index.Searcher.CreateQuery().ManagedQuery(query).Execute();
return Ok(result.SelectMany(r => r.Values.Keys).Distinct().OrderBy(s => s));
}
}
A few practical examples
Here are examples of common queries to showcase the field naming scheme. I have also added few examples which show some conceptual differences between the old and the new indexes.
Search for content by name
The old indexes used the nodeName field for content names, and a handy extension method NodeName() let you query content names without even using the field name.
The new indexes uses Field_Umb_Name_textsr1, because it’s a system field (Umb_Name), indexed as text with the highest textual relevance (textsr1):
var result = index
.Searcher
.CreateQuery()
// a filter for "name contains query"
.Field("Field_Umb_Name_textsr1", query)
Filter by property value
The old indexes used the property type alias for the property fields.
The new indexes apply the naming scheme to property fields; a textbox property with the alias title is indexed with the lowest textual relevance (texts):
var result = index
.Searcher
.CreateQuery()
// a filter for "title field contains query"
.Field("Field_title_texts", query)
Non-textual properties
Examine treats indexed field values as text unless explicitly instructed otherwise. To perform non-textual filtering (for example, numeric filtering), Examine must be configured to handle the index fields before any content hits the index.
This used to be done by configuring the LuceneDirectoryIndexOptions, like this:
using Examine;
using Examine.Lucene;
using Microsoft.Extensions.Options;
using Umbraco.Cms.Core;
using Umbraco.Cms.Core.Composing;
namespace Site.DependencyInjection;
public class ConfigureExamineComposer : IComposer
{
public void Compose(IUmbracoBuilder builder)
=> builder.Services.ConfigureOptions<ConfigureExternalIndexOptions>();
public class ConfigureExternalIndexOptions : IConfigureNamedOptions<LuceneDirectoryIndexOptions>
{
public void Configure(string? name, LuceneDirectoryIndexOptions options)
{
if (name?.Equals(Constants.UmbracoIndexes.ExternalIndexName) is true)
{
// define the "minutes" property as an integer field in the Examine index
options.FieldDefinitions.AddOrUpdate(new FieldDefinition("minutes", FieldDefinitionTypes.Integer));
}
}
// part of the interface, but does not need to be implemented for this
public void Configure(LuceneDirectoryIndexOptions options)
{
throw new NotImplementedException();
}
}
}
In principle, this is still doable for the new indexes, as long as you swap the ExternalIndex for the Umbraco Search index alias and apply the new index naming scheme to the field. However, the Examine search provider has its own field configuration, which you should probably use instead:
using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Search.Provider.Examine.Configuration;
namespace Site.DependencyInjection;
public class ConfigureExamineComposer : IComposer
{
public void Compose(IUmbracoBuilder builder)
=> builder.Services.Configure<FieldOptions>(options =>
{
options.Fields =
[
new()
{
// define the "minutes" property as an integer field in the Examine index
PropertyName = "minutes",
FieldValues = FieldValues.Integers
}
];
}
);
}
And now you can perform numeric filtering by adding the value type postfix integers to the property alias minutes:
var result = index
.Searcher
.CreateQuery()
// a range filter for "minutes >= 2 and <= 7"
.RangeQuery<int>(["Field_minutes_integers"], 2, 7)
Filter by content type
In the old indexes, the content type alias was added as an index field. This provided a handy means to filtering by content type, using the NodeTypeAlias() extension method:
index
.Searcher
.CreateQuery()
// a filter for "content type is page"
.NodeTypeAlias("page")
While this is very practical, it also comes with a negative performance impact: If the content type alias chances, all content of that type will have to be re-indexed 🥵
To avoid this performance penalty, Umbraco Search only indexes the content type ID (key), which is immutable.
The cost? You’ll need to know the ID of the content type you’re looking for. Here’s a simplified example of how you could do that:
private string? _pageContentTypeKey;
...
_pageContentTypeKey ??= _publishedContentTypeCache
.Get(PublishedItemType.Content, "page")
.Key
.AsKeyword();
var result = index
.Searcher
.CreateQuery()
// a filter for "content type is page"
.Field("Field_Umb_ContentTypeId_keywords", _pageContentTypeKey)
The content type ID is a system field named Umb_ContentTypeId. It’s indexed as a keyword for filtering and faceting, so it gets the keywords value type postfix.
Note also the AsKeyword() extension method on the content type ID. Since a Guid can be formatted in a number of ways, this extension method ensures you query the index correctly.
Culture variance
In the old indexes, culture variance was handled at index field level, by appending the culture to the relevant field names. To search specific culture(s) you’d probably use a GroupedOr() across the relevant culture fields, omitting the culture postfix to include invariant matches:
var result = index
.Searcher
.CreateQuery()
// a filter for "title field contains query", including
// invariant (no culture postfix) and en-US in the results
.GroupedOr(["title", "title_en-us"], query)
In the new indexes you add a culture discriminator to the entire query instead, by applying GroupedOr() to a special field for culture variance handling. The field names stay the same, regardless of variance:
var result = index
.Searcher
.CreateQuery()
// include invariant ("none") and en-US in the results
.GroupedOr(["Sys_Culture"], "none", "en-US")
.And()
// a filter for "title field contains query"
.Field("Field_title_texts", query)
Extracting search result IDs
The ISearchResult returned by an Examine query has an Id property. In the old indexes, this would contain a string representation of the numeric ID for the result content. You could parse it up and use it to fetch (published) content from the Umbraco services.
In the new indexes, the Id field contains the content key instead. However, for variant content, the content variance will be appended to create a unique Examine ID per content per variant. As such, the Id field cannot be used consistently to find the resulting content.
Instead, you’ll need to grab the content ID (key) from its designated system field, the Umb_Id field:
var result = index
.Searcher
.CreateQuery()
...
.Execute();
var ids = result
.Select(x => x.GetValues("Field_Umb_Id_keywords").FirstOrDefault())
.WhereNotNull()
.Select(Guid.Parse)
.ToArray();
Putting it all together
Here’s a fully-fledged example which puts all (or most) of the above together in a single search service:
using Examine;
using Examine.Search;
using Umbraco.Cms.Core.Models;
using Umbraco.Cms.Core.Models.PublishedContent;
using Umbraco.Cms.Core.PublishedCache;
using Umbraco.Cms.Search.Core;
using Umbraco.Cms.Search.Core.Extensions;
namespace Site.Services;
public interface IMyExamineSearcher
{
IEnumerable<Guid> FindPages(string query, Range<int> minutes, string? culture = null, int skip = 0, int take = 10);
}
public class MyExamineSearcher : IMyExamineSearcher
{
private readonly IExamineManager _examineManager;
private readonly IPublishedContentTypeCache _publishedContentTypeCache;
private string? _pageContentTypeKey;
private string? _languageVariantPageContentTypeKey;
public MyExamineSearcher(IExamineManager examineManager, IPublishedContentTypeCache publishedContentTypeCache)
{
_examineManager = examineManager;
_publishedContentTypeCache = publishedContentTypeCache;
}
public IEnumerable<Guid> FindPages(string query, Range<int> minutes, string? culture = null, int skip = 0, int take = 10)
{
if (query.IsNullOrWhiteSpace())
{
throw new ArgumentNullException(nameof(query));
}
if (minutes.Equals(0, 0))
{
throw new ArgumentNullException(nameof(minutes));
}
if (_examineManager.TryGetIndex(Constants.IndexAliases.PublishedContent, out var index) is false)
{
throw new InvalidOperationException("Could not find the index");
}
// cache the keys of the content types to include in the search scope
_pageContentTypeKey ??= _publishedContentTypeCache
.Get(PublishedItemType.Content, "page")
.Key
.AsKeyword();
_languageVariantPageContentTypeKey ??= _publishedContentTypeCache
.Get(PublishedItemType.Content, "languageVariantPage")
.Key
.AsKeyword();
// always include invariant content in the results
string[] cultures = culture is null ? ["none"] : [culture, "none"];
var result = index
.Searcher
.CreateQuery()
// must be content of type "page" or "languageVariantPage"
.GroupedOr(["Field_Umb_ContentTypeId_keywords"], _pageContentTypeKey, _languageVariantPageContentTypeKey)
.And()
// scope the result set to the relevant culture variance
.GroupedOr(["Sys_Culture"], cultures)
.And()
// query must be part of either the content name or the title field
.GroupedOr(["Field_Umb_Name_textsr1", "Field_title_texts"], query)
.And()
// the minutes field must be within the specified range
.RangeQuery<int>(["Field_minutes_integers"], minutes.Minimum, minutes.Maximum)
// only bring back the field needed for parsing the resulting content keys (don't overfetch)
.SelectField("Field_Umb_Id_keywords")
// execute with pagination
.Execute(QueryOptions.SkipTake(skip, take));
// return the content IDs (keys) from the "Umb_Id" system field
return result
.Select(x => x.GetValues("Field_Umb_Id_keywords").FirstOrDefault())
.WhereNotNull()
.Select(Guid.Parse)
.ToArray();
}
}
Examine is definitively still viable
So yes; it is indeed still an option to continue using Examine 🫶
I’d still urge you to port your site search to Umbraco Search, because I believe it will help you in the long run. Umbraco Search will continue to evolve in the CMS core after Umbraco 19, and as you can probably imagine, the Umbraco documentation is already being rewritten to match.
One thing is for certain, though: No matter how you choose to proceed, there will be changes you’ll need to tackle, as I’ve outlined in this blog post.
Happy examining 💜