Query Recipes | Lycan documentation

Various queries and examples for pulling and filtering property data

Improve Docs

Required Reading

https://www.elastic.co/guide/en/elasticsearch/reference/master/search.html for general searching
https://www.elastic.co/guide/en/elasticsearch/painless/master/index.html getting to grips wih Painless
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-query.html for using the availability scripts
Any other elastic documentation that pertains to your implementation

Basic Queries

Most of the time you will be using query filters aka exact match filters instead of fuzzy based searching that Elasticsearch is so good at.

The only time you’ll use text searching is most likely when searching the property name. See https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html

Outlined below are some example filter queries. See

Range Query eg. Sleeps

GET /index_example/_search
{
  "query": {
      "range" : {
          "listing.sleeps" : {
              "gte" : 4,
              "lte" : 12
          }
      }
  }
}

Specific Property Type eg. Villa

GET /index_example/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": {"listing.type": "LISTING_TYPE_VILLA" }}
      ]
    }
  }
}

If you were to combine these two queries, you must put the range filter inside of the bool.must otherwise Elasticsearch will through an error. See the provided links above for more information.

Eg.

GET /index_example/_search
{
  "query": {
    "bool": {
      "must": [
        { 
          "match": {
            "listing.type": "LISTING_TYPE_VILLA" 
          }
        },
        {
          "range" : {
            "listing.sleeps" : {
              "gte" : 4,
              "lte" : 12
            }
          }
        }
      ]
    }
  }
}

Amenity/Feature Search

Since features is an array of object seg. features[].type we need to use the nested term.

Info on nested types: https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

All together:

GET /index_example/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "features",
            "query": {
              "bool": {
                "must": [
                  { "match": { "features.type": "ENTERTAINMENT_TV" }}
                ]
              }
            }
          }
        },
        { 
          "match": {
            "listing.type": "LISTING_TYPE_VILLA" 
          }
        },
        {
          "range" : {
            "listing.sleeps" : {
              "gte" : 4,
              "lte" : 12
            }
          }
        }
      ]
    }
  }
}

Geo Queries

Geo guide: https://www.elastic.co/blog/geo-location-and-search

Simple Distance

This example will search for all properties that exist within a certain distance of a point along with all the others.

GET /index_example/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "geo_distance" : {
              "distance" : "10km",
              "location.geoPoint" : {
                  "lat" : 40.91,
                  "lon" : 8.21
              }
          }
        },
        {
          "nested": {
            "path": "features",
            "query": {
              "bool": {
                "must": [
                  { "match": { "features.type": "ENTERTAINMENT_TV" }}
                ]
              }
            }
          }
        },
        { 
          "match": {
            "listing.type": "LISTING_TYPE_VILLA" 
          }
        },
        {
          "range" : {
            "listing.sleeps" : {
              "gte" : 4,
              "lte" : 12
            }
          }
        }
      ]
    }
  }
}

Bounding Box

GET /index_example/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "geo_bounding_box" : {
              "location.geoPoint" : {
                  "top_left" : {
                      "lat" : 43.90,
                      "lon" : 8.15
                  },
                  "bottom_right" : {
                      "lat" : 38,
                      "lon" : 8.25
                  }
              }
          }
        },
        {
          "nested": {
            "path": "features",
            "query": {
              "bool": {
                "must": [
                  { "match": { "features.type": "ENTERTAINMENT_TV" }}
                ]
              }
            }
          }
        },
        { 
          "match": {
            "listing.type": "LISTING_TYPE_VILLA" 
          }
        },
        {
          "range" : {
            "listing.sleeps" : {
              "gte" : 4,
              "lte" : 12
            }
          }
        }
      ]
    }
  }
}

Availability Scripts

You’ll need to use the Painless scripts we’ve provided (or roll you own) to filter availability and provide useful information listed below. These scripts provide parameters that you’ll need to set which get injected at runtime.

Please note this these examples may be out of date - check the links provided the latest source.

Availability Filter

The first script acts as a filter and will only return properties that are available for the date parameters provided.

The parameters are:

arrivalDate - ISO601 Date
departureDate - ISO601 Date

It can be found here: https://github.com/aptenex/lycan-elasticsearch-config/blob/master/src/availability-filter.painless

GET /index_example/_search
{
    "query": {
        "bool" : {
            "must" : {
                "script" : {
                    "script" : {
                        "source": """
                            if (doc['unitAvailability.dateRange.startDate'].size() == 0 || doc['unitAvailability.configuration.availability.keyword'].size() == 0) {
                                return false;
                            }
                            
                            // Get the difference in days between the arrival and departure
                            LocalDate cArrival = LocalDate.parse(params.arrivalDate);
                            LocalDate cDeparture = LocalDate.parse(params.departureDate);
                            
                            int nights = (int) ChronoUnit.DAYS.between(cArrival, cDeparture);
                            
                            // Get the difference in days between the arrival
                            // and unitAvailability start date sequence
                            
                            LocalDate uaStart = doc['unitAvailability.dateRange.startDate'].value.toLocalDate();
                            
                            int uaDiffDays = (int) ChronoUnit.DAYS.between(uaStart, cArrival);
                            if (uaDiffDays < 0) {
                                return false;
                            }
                            
                            String as = doc['unitAvailability.configuration.availability.keyword'].value;
                            
                            if (uaDiffDays > as.length()) {
                                throw new Exception(as.length() + "");
                            }
                            
                            int nightsSequence = (uaDiffDays + nights);
                            
                            String asSpliced = as.substring(uaDiffDays, nightsSequence).toString();
                            
                            return (asSpliced.indexOf('N') < 0);
                        """,
                        "lang": "painless",
                        "params" : {
                            "arrivalDate" : "2021-01-01",
                            "departureDate": "2021-01-07"
                        }
                     }
                }
            }
        }
    }
}

Availability Fields

Instead of completely filtering out unavailable properties you can use this return data on why a property is not available.

The parameters are:

arrivalDate - ISO601 Date
departureDate - ISO601 Date

It can be found here: https://github.com/aptenex/lycan-elasticsearch-config/blob/master/src/availability-fields.painless

You’ll need to include "_source": true in your query otherwise only the script_fields will be returned.

If none of these availabilityFailures (you can call the field whatever you want) appear then the property is 100% available to book. Elasticsearch in this instance will return no field at all if there are no failures so an existence check is necessary.

If an array of states (strings) is returned, these states are described below:

State	Meaning
AVAILABILITY_NOT_SATISFIED	The provided arrival & departure dates are not available
CHANGEOVER_ARRIVAL_NOT_SATISFIED	Arrival day is invalid
CHANGEOVER_DEPARTURE_NOT_SATISFIED	Departure day is invalid
CHANGEOVER_NO_DATA_AVAILABLE	No changeover data provided to calculate
MINIMUM_STAY_NOT_SATISFIED	Minimum stay is not satisfied
MINIMUM_STAY_NOT_SATISFIED_WITH_AVAILABILITY	Minimum stay is not satisfied but there is availability to book the minimum amount of nights
MINIMUM_STAY_NO_DATA_AVAILABLE	No minimum stay data provided to calculate
MAXIMUM_STAY_NOT_SATISFIED	Maximum stay is not satisfied
MAXIMUM_STAY_NO_DATA_AVAILABLE	No maximum stay data provided to calculate

Only one MINIMUM_STAY_* condition will be provided at a time.

GET /index_example/_search
{
  "_source": true,
  "script_fields": {
    "availabilityFailures": {
      "script": {
        "source": """
            ArrayList stateList = new ArrayList();
            if (doc['unitAvailability.dateRange.startDate'].size() == 0 || doc['unitAvailability.configuration.availability.keyword'].size() == 0) {
                stateList.add("UNIT_AVAILABILITY_NOT_FOUND");
                return stateList;
            }
            
            // Get the difference in days between the arrival and departure
            LocalDate cArrival = LocalDate.parse(params.arrivalDate);
            LocalDate cDeparture = LocalDate.parse(params.departureDate);
            
            int nights = (int) ChronoUnit.DAYS.between(cArrival, cDeparture);
            
            // Get the difference in days between the arrival
            // and unitAvailability start date sequence
            
            LocalDate uaStart = doc['unitAvailability.dateRange.startDate'].value.toLocalDate();           
            
            int uaDiffDays = (int) ChronoUnit.DAYS.between(uaStart, cArrival);
            if (uaDiffDays < 0) {
                throw new Exception("Cannot run query against past dates");
            }
            
            String[] minStayList = /,/.split(doc['unitAvailability.configuration.minStay.keyword'].value);
            String[] maxStayList = /,/.split(doc['unitAvailability.configuration.maxStay.keyword'].value);
            char[] changeoverList = doc['unitAvailability.configuration.changeover.keyword'].value.toCharArray();
            
            // 1 = arrival, 2 = departure, 3 = both 0 = neither
            
            int arrivalIndex = uaDiffDays;
            int departureIndex = uaDiffDays + nights;
            
            // Lets get arrival and departure
            char arrival = changeoverList[arrivalIndex];
            char departure = changeoverList[departureIndex];
            
            if (!arrival.toString().equals("1") && !arrival.toString().equals("3")) {
                stateList.add("CHANGEOVER_ARRIVAL_NOT_SATISFIED");
            }
            
            if (!departure.toString().equals("2") &&! departure.toString().equals("3")) {
                stateList.add("CHANGEOVER_DEPARTURE_NOT_SATISFIED");
            }
            
            int minStay = Integer.parseInt(minStayList[arrivalIndex]);
            int maxStay = Integer.parseInt(maxStayList[departureIndex]);
            
            if (nights < minStay) {
                // We want to see if this minimum stay can be made valid by checking ahead on the availability string
                // Do this by adding the difference onto the nights and check if that is valid
                // We subtract one as the departure day can be a N, with removing that we can just check
                // for the existence of N
                int minStayDiff = minStay - nights;
                int futureNightsSequence = uaDiffDays + nights + minStayDiff;
            
                String as = doc['unitAvailability.configuration.availability.keyword'].value;
                if (uaDiffDays > as.length()) {
                    throw new Exception(as.length() + "");
                }
            
                String asSpliced = as.substring(uaDiffDays, futureNightsSequence).toString();
            
                if (asSpliced.indexOf('N') < 0) {
                    stateList.add("MINIMUM_STAY_NOT_SATISFIED_WITH_AVAILABILITY")
                } else {
                    stateList.add("MINIMUM_STAY_NOT_SATISFIED");
                }
            }
            
            if (nights > maxStay) {
                stateList.add("MAXIMUM_STAY_NOT_SATISFIED");
            }
            
            return stateList;
          """,
        "lang": "painless",
        "params": {
          "arrivalDate": "2018-09-03",
          "departureDate": "2018-09-04"
        }
      }
    }
  }
}

An example with failures:

{
  "took": 38,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "index_localhostdocker_sy56o5",
        "_type": "_doc",
        "_id": "2dccfec6-b103-4125-9987-6d494592328a",
        "_score": 1,
        "_source": ** REMOVED FOR BREVITY **
        "fields": {
          "availabilityFailures": [
            "CHANGEOVER_ARRIVAL_NOT_SATISFIED"
          ]
        }
      }
    ]
  }
}

Aggregations

Aggregations are what allows you to filter down results on property search pages. Eg property type = villa, and then it shows various counts against features such as swimming pool (5 properties) etc.

You can combine aggregations and queries if you wish. As an example of simple aggregations:

GET /index_localhostdocker_4mwj8y/_search
{
  "aggs": {
    "unique_listing_types": {
      "terms": {
        "field": "listing.type.keyword"
      }
    },
    "sleep_counts": {
      "terms": {
        "field": "listing.sleeps"
      }
    }
  }
}

This will give you a list of all document counts for each listing type and sleep count.

Tags: