Search engineering for Elasticsearch clusters

Search that finds it,
however it’s typed

We build search on Elasticsearch clusters and look after them: rules matched to how your users type, ranking checked against real searches, and upgrades taken in order, so each change to search is measured before it ships.

Thanks, we’ll be in touch soon.

{{ heroStatus }}

You’re in good company

A few of the companies we’ve worked with.

When people search in their own words

Elasticsearch is a distributed search and analytics engine. A cluster is made of nodes, each a running copy of Elasticsearch, and holds indexes, each a set of documents: records made of named fields. Each index is stored in one or more parts called shards, spread across the nodes, and copies of them, called replicas, can sit on other nodes. A mapping says how each field is stored, and for a text field an analyzer turns its words into the terms a search looks for, both when a document is stored and when someone searches.

People type plurals and typos, and use their own words for things; some type a phrase rather than a product’s name. A search that only matches words as they’re written finds nothing for many of those, and a ranking left at its defaults can put the closest match below others.

Elasticsearch offers rules for text, words that mean the same, room for typos and search by meaning, each set field by field, and ways to shape the ranking. Each one changes what other searches find, so a change that fixes one search can move another.

So the real work is tuning search to what your users actually type, and checking each change against a set of real searches before it ships.

What we build

Search matched to how people type, indexes planned for change, and clusters kept running and upgraded in order.

  • Results that match how people search

    Rules for each field that set plurals aside, allow for typos and add your users’ own words for things, with the ranking shaped by what matters to you, all checked against a set of real searches.

  • Indexes planned for change

    Each field mapped on purpose rather than guessed from the first document that holds it, templates for new indexes, and a name in front of each index, so a new one can be built and switched in while search keeps answering.

  • Moved from the search you have now

    Results compared search by search, old against new, before the switch, and each app moved over once its results match.

  • Set up to keep running

    Nodes, shards and replicas sized by testing on your own data, on your own servers or in a deployment Elastic hosts, with snapshots, copies of the data kept off the cluster, restored to a separate cluster on a schedule and how long each took written down.

  • Searches that stay quick

    The slowest searches found from the slow log, their profiles read on a copy, shards kept to the sizes the docs advise, and filters used where they fit.

  • Upgrades through each last minor version

    Each cluster brought to the last minor version of its series, the newest release that shares its first number, what the Upgrade Assistant finds fixed, older indexes reindexed or set to read-only, then the next major version, one node at a time, each step tried on a copy first.

  • Access kept to what each app needs

    Security on, a key for each app with only the rights it needs, roles for people, and connections encrypted.

  • Watched once it’s running

    The cluster’s health, slow searches, disk use against its limits, searches that find nothing, and alerts sent to the people who act on them.

  • AI where it helps

    Search by meaning with vectors, combined with keyword search where both help, and the same index answering an AI assistant’s lookups under the same rights.

Found the way people type it

When Elasticsearch stores text, an analyzer, a chain of rules for text, breaks it into words, and each search goes through the same kind of rules, so a search finds a document where their words meet.

An analyzer can lowercase words, set plurals aside with a stemmer, which cuts each word back to a common form, and add words you’ve said mean the same. A search can allow for typos, within a number of letters set by each word’s length.

Every rule finds more, and can also find what shouldn’t come up, so each is set field by field and checked.

So we start from what your users type, and check every change against a set of real searches with the results that should come first.

Type a search for the shop’s bedding, and see what each rule finds, product by product.

Search at Thistledown Bedding

The shop’s rules: capitals don’t matter, plurals are set aside, “comforter” and “duvet” mean the same, typos get a letter or two of room by each word’s length, and every word typed has to be found.

  • {{ p.n }}{{ p.d }}{{ p.sr }}{{ p.r }}

{{ mtSaid }}

Through the last minor version, one node at a time

Elastic publishes an end of life date for each series, a major version with its minor versions, and maintains the newest minor versions of the current major version and the last minor version of the one before.

A major upgrade starts from the last minor version of the series before. Run there, the Upgrade Assistant finds settings that are going away and indexes made too long ago. Indexes work across neighboring major versions only, so older ones are reindexed or set to read-only first; otherwise a node won’t start on the new version.

A rolling upgrade then takes one node at a time, so the cluster can keep answering, and each app’s client library is checked against the new version first, then upgraded after it. Where Elastic runs the cluster as a service, its upgrades follow that service’s own steps.

So we keep a list of each cluster’s version, its oldest index and each series’ dates. We deal with old indexes well ahead, rehearse each upgrade on a cluster restored from a snapshot, and go one node at a time.

Start with the searches that found nothing

The review covers what your users search for, from whatever record of searches you keep: the searches that find nothing, and those whose first results aren’t opened; each field’s mapping and rules, and the synonyms in use, the words set to mean the same; how the ranking is shaped, its boosts and anything added by hand; how data reaches the index, and how soon a change can be found; the indexes, their templates, and whether each sits behind a name of its own; shards and replicas against the sizes the docs advise; each cluster’s version, its oldest index and each series’ end of life date; the client libraries each app uses, and whether each works with the next major version; snapshots, where they go, when one was last restored and how long it took; access, whether security is on, the keys and roles each app has, and encrypted connections; the slow log, disk use against its limits, and alerts; search by meaning, where it’s in use or planned; and how it runs, in the cloud service Elastic runs or on your own servers.

We then propose one first step, tried on a copy before anything changes: a set of real searches with the results that should come first, checked against search as it is now; the rules and synonyms for one busy field tuned against that set; old indexes reindexed or set to read-only, ahead of the next major upgrade; the next upgrade rehearsed on a cluster restored from a snapshot; or a restore to a separate cluster, timed.

What you receive is a written review and a proposal for that first step.

  1. Should search live in our main database, or beside it?

    Many databases can search their own text, and for a simple search that’s often enough. A search engine of its own fits when search is central to the product: ranking tuned to your users, room for typos, your own words for things, search by meaning, and results drawn from several systems. We’ll say which fits.

  2. How do we know whether search got better?

    We keep a set of real searches, each with the results that should come first, and check every change against it before it ships. Elasticsearch can score search against a set like that itself, so each change is compared on the same searches.

  3. Can people search in more than one language?

    Yes. Each language gets rules of its own, field by field, from the language analyzers Elasticsearch offers.

  4. What happens to indexes made in older versions?

    Indexes work across neighboring major versions only. So before the major upgrade that would leave one behind, we reindex it, or set it to read-only where its data no longer changes.

  5. What does it cost to work with you?

    That depends on the clusters and fields involved, and on the first step you choose. Before any work starts, you get a written proposal with the scope, the price or rate, and how you pay.

  6. Can we hire developers for our own team?

    Yes, through a dedicated team: a Webair team that stays close to your product over time and works alongside your own people.

Elasticsearch is a trademark of elasticsearch B.V., registered in the U.S. and in other countries. Elastic is a trademark of elasticsearch B.V. Webair is independent of Elastic and elasticsearch B.V., and isn’t sponsored by, endorsed by or affiliated with either.

Let’s build it better

Tell us what your users search for, and what they don’t find, and we’ll suggest where to begin, in a short call.

Thanks, we’ll be in touch soon.

{{ ctaStatus }}