Kibana Connector to Kinetica

Kibana depends on Elasticsearch to perform the data queries that it visualizes, but Elasticsearch doesn't natively work with most databases. While some have taken the approach of creating connectors from their product to Elasticsearch, we have taken a more direct approach. We have built a small service which bypasses Elasticsearch and connects Kibana directly to Kinetica. This service takes the place of Elasticsearch, receiving queries from Kibana, converting them to Kinetica queries, and returning the results in the format Kibana expects. Currently we only support Kibana version 4.x

Configuration

Configuring Kinetica

1. Edit /opt/gpudb/core/etc/gpudb.conf, set enable_kibana_connector:

   enable_kibana_connector = true
   

2. Restart Kinetica:

   sudo service gpudb restart
   

If you are using the Kibana that we have packaged with Kinetica, you can stop here. The URL for Kibana is:

http://<kinetica_host>:5601

If you will be using an external instance of Kibana, continue to the next section.

Configuring Kibana

Since our connector takes the place of Elasticsearch, we must point Kibana to ours instead.

Edit /opt/kibana/config/kibana.yml

elasticsearch.url: "http://<kinetica_host>:8090"

Configuring the Connector

You probably will never need to configure the connector, but you can change the port, loglevel, and other parameters by editing the file /opt/gpudb/connectors/kibana/config.json. This file contains the parameters:

"port": 8090,
"logLevel": "INFO",
"gpudbUrl": "http://localhost:9191",
"gpudbLogLevel": "FATAL",
"pingInterval": 60000,
"fetchSize": 10000,
"maxFetch": 99999,
"maxAggregationCalls": 100

Differences and Gaps Between Elasticsearch and our Connector

Cautions and Alternatives

1. Geohash grid aggregation: when zooming in on a region, Kibana doesn’t filter its query to just the area in view. Therefore, when zoomed in it can take a really long time because it’s pulling down thousands and thousands (if not potentially millions) of points that it’s not actually displaying. There is a record count cutoff in the config file after which it will error out to prevent crashing the system.
2. Scripted fields use Kinetica's expression syntax, not Elasticsearch's. Therefore, the Kibana help screen for this syntax should be ignored.
3. Full text search bar:
   • Wildcards work, but cannot be at the start of a search term
   • By default it will only search all text_search fields, not all fields, and not particular specified fields.
   • To compensate for the text search restrictions, we have created an extension to the search. Kinetica filter expressions, rather than a text search, can be used by prefacing the expression with the ^ character; for example, ^hasRain = 1 would find only records where it’s raining.
4. Kinetica supports nested aggregations in Kibana, but many of them require per-bucket Kinetica calls. If, for example, you nest a histogram in a histogram in a histogram, the system will be making an exponential number of Kinetica calls and probably won’t perform very well. There is a query count cutoff in the config file after which it will error out to prevent crashing the system.

Not Implemented

1. NaN values, since our connector uses JSON which does not natively support NaN.
2. Significant Terms aggregation
3. IPv4 Range aggregation
4. The Link to ... link in Discover view

Partially Implemented

1. Terms aggregation - not all sorting options are supported:
   • sum, avg, min, max, and count are supported
   • cardinality is not supported
2. Scripted fields:
   • Scripted fields work in visualizations but not the Discover view.
3. Advanced settings:
   • Only support
     • dateFormat:tz (which should be left at UTC, since all Kinetica timestamps are in UTC)
     • defaultIndex.
   • Changing any of the other settings in the UI does nothing.