Greetings everyone!
Almost a month has passed since my last publication, so sorry for not writing anything for a so long time. But now I wanna share with you my first (positive) impressions about working with Elasticsearch.
Well, let's start from what Elasticsearch is. Frankly speaking, Elasticsearch is flexible and powerful open source, distributed real-time search and analytics engine for the cloud based on Apache Lucene.
So, if you want to have a full-text advanced search in your application, Elasticsearch could be a good option, because most likely it would fit to all of yours requirements. I would like to note that the entry barrier is not very high and I hope everybody will understand how it works in reasonable (short) time.
Let's try to touch and play a little bit with Elasticsearch. The first step would be installing and running:
- Download latest release here.
- Elasticsearch requires at least Java 6 in order to run.
- Follow installation guide for installing and running Elasticsearch.
- The first question after running Elasticsearch could be: where is the front-end for that? You have to install additional plugin to have Web-UI for Elasticsearch. For more information visit: http://www.elasticsearch.org/guide/reference/modules/plugins/
After everything is installed, up and running you should be able to access Web-UI by the following URL: http://host_name:9200/_plugin/head/
There are a lot of tutorials and blogs across internet where you can easily find information about how to load data to Elasticsearch index, how to querying documents afterwards, but I want to concentrate more on a practical part of a question and only partially on a theoretical part.
So, together we will see how to build a small application for filling out the index and will see how Elasticsearch behaves when doing search requests.
I'm gonna build 2 different indexes to show you the diversity of its capabilities:
- Wiki articles
- Products in shop
I've downloaded Wiki database archive which contains only id and article name and prepared one file for filling second index with random products.
First we need to implement basic functionality for indexing the data:
public abstract class AbstractFileResourceIndexBuilder<T> {
/**
* Allows to build the index
*
* @throws IOException can be thrown
*/
public void buildIndex() throws IOException {
long startTimeInMillis = System.currentTimeMillis();
//build the index
long count = doBulkLoad(getBufferedReader());
//close all opened resources
closeResources();
long endTimeInMillis = System.currentTimeMillis();
System.out.println("Indexing finished in " + (endTimeInMillis - startTimeInMillis)
+ ". Total number of items: " + count);
}
/**
* Gets BufferedReader object from the implementation
*
* @return BufferedReader for current implementation
* @throws IOException
*/
protected abstract BufferedReader getBufferedReader() throws IOException;
/**
* Closes all resources which were opened previously
*
* @throws IOException
*/
protected abstract void closeResources() throws IOException;
/**
* Does bulk load. In the end all data should be in the Lucene index
*
* @param reader buffered reader for current implementation
* @return number of loaded items
* @throws IOException
*/
protected abstract long doBulkLoad(BufferedReader reader) throws IOException;
/**
* Prepares request builder for one item
*
* @param item T
* @return current request builder
* @throws IOException
*/
protected abstract IndexRequestBuilder getIndexRequestBuilder(T item) throws IOException;
/**
* Allows to flush items from collection to ElasticSearch/Lucene index
*
* @param items items to be loaded to the index
* @throws IOException
*/
protected final void flush(Queue<T> items) throws IOException {
BulkRequestBuilder bulkRequestBuilder = ElasticSearchClientFactory
.getClient().prepareBulk();
while (!items.isEmpty()) {
T item = items.poll();
IndexRequestBuilder req = getIndexRequestBuilder(item);
bulkRequestBuilder.add(req);
}
BulkResponse bulkResponse = bulkRequestBuilder.execute()
.actionGet();
if (bulkResponse.hasFailures()) {
System.out.println("Error occurred");
}
}
}
If you curious about Java client for Elasticsearch and want to know more you can take a look here.Let's implement Index Builder for Wiki articles:
/**
* Allows to build WIKI index.
* It will load each 10000 articles from the file and then flush it to ElasticSearch
*/
public class WikiIndexBuilder extends AbstractFileResourceIndexBuilder<WikiArticle> {
private static final int STEP = 10000;
private InputStream in;
private BZip2CompressorInputStream bzIn;
private BufferedReader reader;
@Override
protected BufferedReader getBufferedReader() throws IOException {
in = WikiIndexBuilder.class.getResourceAsStream("nlwiki.txt.bz2");
bzIn = new BZip2CompressorInputStream(in);
reader = new BufferedReader(new InputStreamReader(bzIn));
return reader;
}
@Override
protected void closeResources() throws IOException {
reader.close();
bzIn.close();
in.close();
}
protected long doBulkLoad(BufferedReader reader) throws IOException {
Queue<WikiArticle> articles = new LinkedList<>();
int count = 0;
String strLine;
// Read File Line By Line
while ((strLine = reader.readLine()) != null) {
// Print the content
System.out.println(strLine);
count++;
String[] article = strLine.split(":");
//add an article to the queue
articles.add(new WikiArticle(article[0] + article[1], article[2]));
//flush the data to ElasticSearch index if need
if (articles.size() == STEP) {
flush(articles);
}
}
//flush the data if need
flush(articles);
return count;
}
@Override
protected IndexRequestBuilder getIndexRequestBuilder(WikiArticle article) throws IOException {
return ElasticSearchClientFactory
.getClient()
.prepareIndex("wiki", "article", article.getId()).setSource(jsonBuilder().
startObject().field("name", article.getName()).endObject());
}
}
Of course, we also have to add POJO class:public class WikiArticle {
private String id;
private String name;
public WikiArticle(String id, String name) {
this.id = id;
this.name = name;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
We are now ready for building the Wiki index:
@Test
public void testBuildWikiIndex() {
try {
new WikiIndexBuilder().buildIndex();
} catch (IOException e) {
e.printStackTrace();
Assert.fail("Indexing process must not fail");
}
}
The process of indexing on my machine usually takes about 4-5 minutes, because the resource file contains more than 2 million articles.For the "Products" index we need to implement second builder:
/**
* Allows to build Products index.
* Loads data from products.txt.
* User: vlotar
*/
public class ProductIndexBuilder extends AbstractFileResourceIndexBuilder<Product> {
private InputStream in;
private BufferedReader reader;
@Override
protected void closeResources() throws IOException {
reader.close();
in.close();
}
@Override
protected BufferedReader getBufferedReader() throws IOException {
in = ProductIndexBuilder.class.getResourceAsStream("products.txt");
reader = new BufferedReader(new InputStreamReader(in));
return reader;
}
@Override
protected long doBulkLoad(BufferedReader reader) throws IOException {
Queue<Product> products = new LinkedList<>();
int count = 0;
String strLine;
// Read File Line By Line
while ((strLine = reader.readLine()) != null) {
// Print the content
System.out.println(strLine);
count++;
String[] product = strLine.split(";");
products.add(new Product(product[0],product[1],product[2],new BigDecimal(product[3])));
}
flush(products);
return count;
}
@Override
protected IndexRequestBuilder getIndexRequestBuilder(Product product) throws IOException {
return ElasticSearchClientFactory
.getClient()
.prepareIndex("products", "product", product.getId()).setSource(jsonBuilder().startObject()
.field("brand", product.getBrand())
.field("name", product.getName())
.field("price", product.getPrice())
.endObject());
}
}
POJO class for Products index:public class Product {
private String id;
private String brand;
private String name;
private BigDecimal price;
public Product(String id, String brand, String name, BigDecimal price) {
this.id = id;
this.brand = brand;
this.name = name;
this.price = price;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getBrand() {
return brand;
}
public void setBrand(String brand) {
this.brand = brand;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public BigDecimal getPrice() {
return price;
}
public void setPrice(BigDecimal price) {
this.price = price;
}
}
In order to run building the "Products" index: @Test
public void testBuildProductsIndex() {
try {
new ProductIndexBuilder().buildIndex();
} catch (IOException e) {
e.printStackTrace();
Assert.fail("Indexing process must not fail");
}
}
Once both indexes are built we can go to the Web-UI and see them there:
There is a tab which you can use for querying:
One of the most important things is that Elasticsearch is schemaless. So you don't necessary need to define schema, mapping and all other staff. Thus Elasticsearch will use standard analyzer which seems ok for most of the cases related to English language. For example, it will use "stopwords" filter by default, so if somebody will search for term "the", the result would be empty. You need to play a little bit with the search and you feel how it works, what it does and ... what's missed.
Elasticsearch has a good documentation so you might want to have a look how you can configure advanced search. Here I'm gonna give you few examples.
In order to tell Elasticsearch how to perform search, how to analyze the data, what filters should be applied, you need to specify index configuration and mapping. But first you have to delete old index. You can do that either via Web-UI or by sending curl command, e.g.:
curl -XDELETE localhost:9200/wiki
Once you deleted both indexes we can continue with configuration and mapping for the Wiki index.
curl -XPUT 'http://localhost:9200/wiki' -d '{
"index": {
"analysis": {
"filter": {
"snowball": {
"type": "snowball",
"language": "English"
}
},
"analyzer": {
"a2": {
"filter": [
"standard",
"stop",
"lowercase",
"snowball"
],
"type": "my_analyzer",
"tokenizer": "standard"
}
}
}
}
}'
By this command we tell Elasticsearch that we might want to use our own analyzer "a2" for some fields, for example for the name of the Wiki article:
curl -XPUT localhost:9200/wiki/article/_mapping -d '{
"article": {
"properties": {
"name": {
"type": "string",
"analyzer": "a2"
}
}
}
}'
Now Elasticsearch will use our own analyzer for the name field according to our configuration and mapping. Notice that both configuration and mapping can be set up programatically.
Let's consider search phrases "The Cats" and "the Cats" as an example. Is there a difference? According to our configuration there is. Going step by step applying the filters we see that after applying standard filter it will apply stop token filter, but because the term "The" starts from the uppercase letter it won't be removed. Then we are using lowercase filter and after this applying stemming. So, in the end it would be something like: "the cats/cat". For the second search phrase the result would be "cats/cat", because the term "the" is going to be removed during applying stop filter. You can check how it works by sending search requests using Web-UI. If you want to have the same behavior you should move lowercase filter before stop filter.
Let's change a little bit configuration for "Products" index. So, index and mapping would be like this:
curl -XPUT 'http://localhost:9200/products' -d '{
"index": {
"analysis": {
"filter": {
"synonym": {
"type": "synonym",
"synonyms_path" : "analysis/synonym.txt"
}
},
"analyzer": {
"a2": {
"filter": [
"stop",
"lowercase",
"synonym"
],
"type": "my_analyzer1",
"tokenizer": "standard"
}
}
}
}
}'
curl -XPUT localhost:9200/products/product/_mapping -d '{
"article": {
"properties": {
"name": {
"type": "string",
"analyzer": "a2"
}
}
}
}'
Here I wanna show you how to use synonyms when doing search. In the configuration of index we specified new filter called "synonym" and the path to the file which contains rules.
#Explicit mappings match any token sequence on the LHS of "=>"
#and replace with all alternatives on the RHS. These types of mappings
#ignore the expand parameter in the schema.
#Examples:
i-pod, i pod => ipod
sea biscuit, sea biscit => seabiscuit
foozball, foosball, fussball, football => soccer, football
ps3 => playstation
i phone, i-phone => iphone
Build the index and try to search:
Examples of different search requests:
Now you've got basic overview what you can achieve with Elasticsearch in relatively small period of time. My impressions are very positive, however we are still using SOLR, but who knows what will happen in the future.
For those who like Spring framework there is an integration:
- https://github.com/SpringSource/spring-data-elasticsearch
- http://techo-ecco.com/blog/elasticsearch-with-spring/
Thanks for reading!
