Reveal SDK

Introduction

Reveal is a web-based visual data exploration and insight discovery tool. It allows users who may or may not have extensive experience with big data or data analytics to quickly build charts, graphs, and maps in a meaningful way to explore their dataset. Reveal was designed to be interactive and easy to use.

While Reveal comes included with many common charts and map slices (collectively, what we call widgets in Reveal), the Reveal SDK allows developers to create their own custom slice types. From these, slices can be created, which can interact with existing dashboards and slices. The Reveal architecture allows for complete control and customization of what is rendered in the slice, as well as what interactions are available to users.

Software Stack

Reveal is a web application with a Python (currently requires version 2.7) backend server and mostly JavaScript/React frontend client. It uses Flask AppBuilder as its server-side web framework. The client-side code is primarily a single page app design which leverages bundled JavaScript built using webpack from a Node (requires version 6.9.0+) project structure.

For database connections, Reveal uses PyODBC to query Kinetica via a custom SQLAlchemy connector. Some slices also interact with Kinetica via API endpoints. It is recommended that, in addition to basic knowledge of the endpoint API, a custom slice type developer have some familiarity with the following technologies:

Architecture

The following diagram represents the general data architecture and flow of a typical Reveal dashboard. We find that many use cases integrate both Map and Filter slices. While these particular slices are not required to build a dashboard, the Filter slice is required to enable global/cross filtering in the dashboard. This means any filtering interaction such as drawing an area selection on a map or clicking on a bar in a Histogram slice will trigger a filtering of all data represented in the dashboard by those parameters and a re-rendering of other slices in the dashboard.

SDK Concepts

The Reveal SDK is a set of command line tools which allows and assists developers in creating new custom slice types for Reveal. These tools will help with:

• Creating a new boilerplate template for new custom slice types for a quick start
• Watching a custom slice type development codebase so any new changes will be reflected in the local development Reveal app immediately
• Extracting a custom slice type into a portable package for migration to another Kinetica installation; useful for deploying to production
• Installing a custom slice type package from another Kinetica environment into a local development Reveal app

Reveal is designed in such a way that all code and static resources for a given slice type reside in their own folder. Each slice type folder is located under the slices folder in the Reveal SDK home directory. A slice type folder must contain at least four files:

• Node Project File - package.json: defines some attributes of the slice type and any required dependencies
• Python Slice Class - __init__.py: defines the server-side data request and response handlers that will read data from the database
• JavaScript Slice Renderer - <slice_name>.jsx: contains a plain function that must return an object with a render function
• Thumbnail Icon - <slice_name>.png: image that represents the icon for the slice type

Additionally, the folder can also contain any external *.css or additional React *.jsx component files required by the slice.

In the following sections, we’ll be using our Kinetica Big Number slice type as an example. It is structured as follows, under the Reveal SDK home directory:

• /slices
  • /kinetica_big_number
    • __init__.py
    • kinetica_big_number.css
    • kinetica_big_number.jsx
    • kinetica_big_number.png
    • package.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "name": "kinetica_big_number",
  "version": "1.0.0",
  "description": "Big Number slice.",
  "main": "kinetica_big_number.jsx",
  "author": "Kinetica",
  "license": "ISC",
  "dependencies": {
    "argparse": "1.0.10",
    "glob": "7.1.2"
  }
}

Python Slice Class

The __init__.py Python class must extend the BaseViz class, which implements all the required methods to handle database queries and return the data back to the client. There are several other methods involved, but we’ll cover just two primary ones that are most often extended:

• query_obj() - responsible for defining the structure of the data query request such as specifying group-bys, columns, and metrics
• get_data() - responsible for transforming the results from the database query request into a consumable JSON format for the client slice component to consume

Fieldsets

Another important Python class configuration is the fieldsets parameter. It defines which field(s) will be exposed for slice customization in the slice editor screen in Reveal. All currently provided fields are defined in the FormFactory class in a file named forms.py in the Reveal SDK home directory.

Some commonly-used fields include:

• metrics - a multi-select field with a list of all available metrics for a given table
• metric - a select field with a list of all available metrics for a given table
• groupby - a multi-select field with a list of all groupable columns for a given table
• all_columns - a multi-select field with a list of all available columns for a given table
• limit - a free-form select field with a list of suggestions of 0, 5, 10, 25, 50, 100, and 500

Note

The Time fieldset is always included for every slice by default.

Custom Fields

Several form field types are available to use from the wtforms library. However, custom slice type development will likely require developers to define their own form field controls in order to configure their slice type. New controls can be defined in the __init__.py file. Simply provide a configuration map parameter called form_custom in the Python class, which will contain the definition of any custom fields.

The following example defines two custom fields called Polling Interval & Text Content. Using these, two fieldsets will be created. The first fieldset will contain the default metric field and the Polling Interval custom field. The second fieldset will contain the Text Content field.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

from wtforms import ( SelectMultipleField, SelectField, TextField, TextAreaField, BooleanField, IntegerField, HiddenField, DecimalField) class CustomFieldExampleViz(BaseViz): # Below is a simple example of how to implement custom form fields for a # custom slice. # For this custom slice we will create two custom fields: interval & # content. Two fieldsets will be created: Group A will contain the default # metric field and the custom interval field, and Group B will contain the # custom content field only. These fields will be defined further below in # form_custom. fieldsets = ({ 'label': _('Group A'), 'fields': ( 'metric', 'interval', ) },{ 'label': _('Group B'), 'fields': ( 'content', ) },) # Define your custom form controls in the form_custom map. The name # of the key must match the value used above in fieldsets. # The first param is the form field type. They are imported from # wtforms lib and you can see what is available in the above imports. # The second param is the configuration. The following fields can be # specified, but not all may be required: # * label - displayed above the form control on screen # * default - the default value to select # * choices - a tuple of label/value pairs for list controls # * description - displayed in mouseover of info icon next to label # Interval field will be a dropdown list which will display various # time interval options such as Minute, Hour, Day, etc. It will be # labeled 'Polling Interval' and its default selection will be Day. # Content field will be a textfield which will allow users to enter # any value. It will be labeled 'Text Content' and the default value # will be 'Enter content text here'. form_custom = { 'interval': (SelectField, { "label": _("Polling Interval"), "default": "day", "choices": ( ('none', _('None')), ('minute', _('Minute')), ('hour', _('Hour')), ('day', _('Day')), ('month', _('Month')), ('year', _('Year')), ), "description": _("Interval for time data") }), 'content': (TextField, { "label": _("Text Content"), "default": "Enter content text here", "description": _("Content for page") }) }

The two custom fields will appear on-screen like this:

The Kinetica Big Number slice type fieldset looks like this:

The following is the __init__.py for the example Kinetica Big Number slice type fieldset above:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

from caravel.baseviz import BaseViz from flask_babel import lazy_gettext as _ class KineticaBigNumberViz(BaseViz): """Put emphasis on a single metric with this big number viz""" viz_type = "kinetica_big_number" verbose_name = _("Big Number") is_timeseries = False # Form field configuration for the slice. fieldsets = ({ 'label': _('Number'), 'fields': ( 'metric', 'subheader', 'y_axis_format', ) },) # You can override form field default labels but specifying # custom values in the form_overrides map. The following will # display the y_axis_format text field with the label # 'Number format'. form_overrides = { 'y_axis_format': { 'label': _('Number format'), } }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

# The query_obj method allows you to customize the data # request SQL query, often passing form field values entered # from the slice configuration. In the example below, the slice # is going to use the metric selected from the UI and using it # as the metrics parameter for the query. An example metric # would be COUNT(*). This slice also requires the user to # select a metric to continue. def query_obj(self): d = super(KineticaBigNumberViz, self).query_obj() metric = self.form_data.get('metric') if not metric: raise Exception("Pick a metric!") d['metrics'] = [self.form_data.get('metric')] self.form_data['metric'] = metric return d # The get_data method allows user to take results from the # data request SQL query and shape it into any desired JSON # format for the client to use in rendering the slice. In this # case, an object with property 'data' which contains the # metric result and 'subheader', the label the user entered # in the subheader text field in the slice configuration screen. def get_data(self): form_data = self.form_data df = self.get_df() df.sort_values(by=df.columns[0], inplace=True) return { 'data': df.values.tolist(), 'subheader': form_data.get('subheader', ''), } def getclass(): return KineticaBigNumberViz

1

import KineticaTheme from 'kReveal/utils/kineticaTheme';

1
2
3
4
5

// This will NOT work!
import bluebird from 'bluebird';

// This will work.
import bluebird from './node_modules/bluebird';

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107

// Import any needed modules
import "./kinetica_big_number.css";

// Main slice function
function kineticaBigNumber(slice) {

  // Handle to the slice DOM node for manipulation
  const div = d3.select(slice.selector);

  // Render method called by dashboard when rendering slice
  let render = kineticaData => {
    let endpoint = slice.jsonEndpoint({ extraFilters: true });

    // Check if data sent from global filter slice
    if (kineticaData) {
      // Get filtered table name
      const tableName = kineticaData.k_table_name
        ? kineticaData.k_table_name
        : null;

      // Only use view name from same source table
      const primaryTable = getParamFromQryStr(endpoint, "datasource_name");
      if (tableName && tableName.startsWith(primaryTable)) {
        // Use the new filtered table for data queries
        endpoint = updateQryStrParam(
          endpoint,
          "datasource_name",
          encodeURIComponent(tableName)
        );
      }
    }

    // This is the call to fetch data
    KineticaData.jsonCache(
      endpoint,
      loading,
      (error, payload, cached = false) => {
        // Clear the slice
        div.selectAll("*").remove();

        const fd = payload.form_data;
        const json = payload.data;


        ///////////////////////////////////////////////////////
        // START - CUSTOM SLICE CODE TO RENDER VISUALIZATION //
        ///////////////////////////////////////////////////////

        // Get data from response
        const data = json.data;
        const v = data[0][0];

        const g = svg.append("g");

        // Printing big number
        g.append("g").append("text").text(f(v));

        // Printing big number subheader text
        if (json.subheader !== null && json.subheader !== "") {
          g.append("text").text(json.subheader);
        }


        /////////////////////////////////////////////////////
        // END - CUSTOM SLICE CODE TO RENDER VISUALIZATION //
        /////////////////////////////////////////////////////

        if (!cached) {
          slice.done(payload);
        }
      }
    );
  };

  // Throttle rendering of slices to prevent too many
  // renderings triggered by resizing of window
  render = debounce(render, config.kinetica.render.delay);

  // Subscribe to pubsub to listen for new filtered table
  // data for which we will re-render the slice
  KineticaPubSub.subscribe(
    "kinetica_big_number",
    TBL + kPBConsts.FILTER,
    (msg, data) => {
      switch (msg) {
        case TBL + kPBConsts.FILTER_TERMS: {
          render(data);
          break;
        }
        case TBL + kPBConsts.FILTER_TERMS_NOMAP: {
          render(data);
          break;
        }
        default: {
          break;
        }
      }
    }
  );

  return {
    render: render,
    resize: render
  };
}

module.exports = kineticaBigNumber;

Thumbnail Icon

The thumbnail icon is required to display the slice type as a dropdown option in the slice editor where users can select the slice type. It is recommended that the PNG image be 600 x 600 pixels, square, and have the same name as the slice type: <slice_name>.png.

Image	Image Within Dropdown

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

.kinetica_big_number .slice_container {
  overflow: hidden !important;
}

.kinetica_big_number g.axis text,
.kinetica_big_number_total g.axis text {
  font-size: 10px;
  font-weight: normal;
  color: gray;
  fill: gray;
  text-anchor: middle;
  alignment-baseline: middle;
  font-weight: none;
}

Setup/Installation

The setup sequence must be run on both development and deployment environments, so that the tools for managing custom Reveal components are available for use.

git config --global url."https://".insteadOf git://

1. Install Kinetica database and make sure the service is up and running. Refer to Kinetica Installation.

2. Install git. See Git Downloads for details.

3. Reveal is packaged with Node & npm, located in /opt/gpudb/connectors/reveal/bin. This directory should be added to the system path for development:

   1 2	# This only sets the path for the current session export PATH=/opt/gpudb/connectors/reveal/bin:$PATH

4. Install the Reveal SDK:

   1	npm install -g /opt/gpudb/connectors/reveal/sdk

Development Workflow

1. For development, it is recommended to have a dedicated directory for all working custom slice types. This directory should be version-controlled as it will contain the slice type configuration & script files. The structure should mirror this:

   • /<target_development_directory>
     • /<slice_type_1_name>
     • /<slice_type_2_name>
     • /<slice_type_3_name>

2. Use the reveal-sdk script from the SDK to run the slice-create command:

   reveal-sdk slice-create <target_development_directory>
   

   This will perform the following actions:

   • Prompt the user for basic slice type-specific information to create the package.json file

     • What is the slice name?

       Note

       Name must begin with alphabetic/underscore character and be followed by zero or more alphanumeric/underscore characters

     • Who is the author?

     • What is the display name?

       Note

       This is the name/label that will be displayed in the slice type dropdown in the GUI

     • What is the slice description?

   • Use the included SDK slice type template files to generate a new custom slice type directory based on the information provided in the previous step

   • Put the new slice type directory/files into the target development directory

3. Use the reveal-sdk script from the SDK to run the slice-watch command in a separate shell session:

   reveal-sdk slice-watch <target_development_directory>/<slice_name>
   

   This will perform the following actions:

   • Run a full rebuild of all the slice types including the newly created slice type
   • Restart Reveal to activate the new slice type
   • Start watching for changes to the specified slice type in the target development directory and rebuild/restart when necessary

   Note

   This is a continuous process so it should be run in a separate shell. The command takes some time to run, as it installs node modules and bundles all the JavaScript code.

4. Develop the custom slice type by modifying its files in the target development directory. The slice type watch process will continually update the Reveal application instance to reflect your changes. Test and repeat process.

   Note

   During iterative development, it may be necessary to do a hard reload (clear asset cache) in your browser to make sure it downloads the latest updated assets from the server.

   Important

   When installing and using custom npm packages in custom slice types, the import path must reference the slice type project’s node_modules directory directly:

   1	import $ from './node_modules/jquery';

Deployment Workflow

1. In the development environment, where the slice type was created, use the reveal-slice-extract script in the SDK Home directory to extract the desired slice type code for migration to a production environment. This command will compress the contents of the specified custom slice type into a tarball.

   1	reveal-slice-extract <slice_name> [<slice_tarball.tgz>]

   Note

   By default, this will create a <slice_name>.tar.gz file with the packaged slice type in the current directory. Optionally, a name for the tarball can be specified in the command.

2. Transfer the newly-created slice type package to the production server or hosted repository.

3. Ensure that the Reveal SDK is installed on the target production server. See Reveal Setup/Installation for details.

4. In the production environment, where the slice type is to be deployed, use the reveal-slice-install script in the SDK Home directory to install the slice type package. This will extract the package into the Reveal installation source slices folder and re-run the production build. This will also restart Reveal to activate the new slice type.

   If Kinetica is installed in a non-standard location, set the KINETICA_INSTALLED_PATH to the top-level Kinetica directory before running either reveal-slice-install or reveal-slice-extract.

   Warning

   Currently Kinetica upgrades will overwrite your slices. Make sure you save and archive all your custom slice type export tar files as you MUST reapply them after each upgrade.

   To deploy the slice type package, run:

   1	reveal-slice-install <slice_package_tarball>

   Important

   This process may take several minutes, due to node modules being installed and webpack needing to bundle all the JavaScript files.

White Labeling

It is possible to white label Reveal by customizing the logo and stylesheets. For instance, the Reveal splash page can be modified to look like this:

1
2
3

# Configure login page title and subtitle
LOGIN_TITLE = 'Reveal'
LOGIN_SUBTITLE = 'Visual Data Exploration'

1

reveal-theme backup <backup_tarball_file>

1

reveal-theme restore <backup_tarball_file>

Upgrades

Currently, when performing an upgrade of Kinetica, the existing instance of Reveal will be replaced by a newer version. While the Reveal settings and database where dashboards and slices are stored will be preserved, custom slice types are not. To remedy this, the SDK provides a tool that will ensure that custom slice types are backed up and can be reapplied after the upgrade.

Before upgrading, use the reveal-backup script from the SDK to backup custom slice types. If Reveal has been white-labeled, it will need to be backed using the reveal-theme script. The following commands will backup both the slice types as well as any white-labeling artifacts.

1
2
3

reveal-backup <backup_tarball_file>

reveal-theme backup <theme_backup_tarball_file>

Next, upgrade Kinetica. After the upgrade, use the reveal-restore script from the SDK to restore custom slice types to the new Reveal instance. If there are white-labeling backups, those can be restored as well with the reveal-theme script.

1
2
3

reveal-restore <backup_tarball_file>

reveal-theme restore <theme_backup_tarball_file>

Login to Reveal application and verify your custom slice types and any custom white-labeling.

Extras