7.1 Release Notes

Publish Date: 2020.08.19

Features

  • Cluster Resiliency allows failed processes within a cluster to be shifted to other active or passive hosts within the cluster
  • Schemas (formerly collections) are now namespaced, allowing two or more schemas to contain a table with the same name; the native API will now accept <schema_name>.<table_name> references to tables & views
  • Externally-managed data sources can be the source of both Tables and data load commands (LOAD INTO, /insert/records/fromfiles)
  • Improved system & cluster monitoring capability

Core

  • Ring Resiliency (High Availability) is now integrated into the core of the product; a separate package & application suite is no longer needed
  • As part of the namespaced schemas change, all top-level tables & views will be relocated to the home schema.

APIs/Tools/Connectors

  • All APIs, KIO, and the Spark connector have been updated to support schemas.
  • APIs:
    • The Java API has been updated to support cluster resiliency and the new ring resiliency changes
    • The Java API now supports logging via Apache Log4j
    • The Java API setHostManagerPort option will no longer work after the database object has been initialized
  • KIO:
    • Due to the advent of schemas, the --d-collection-name Kinetica format option has been deprecated

KAgent

  • Added cluster resiliency support
  • Added support for switching responsibility for one host's processes to another host, on demand
  • Added support for configuring metric-based and event-based alerts
  • Added support for configuring delivery channels (e.g, email) for alerts
  • Added/improved monitoring of:
    • Rings
    • Clusters
    • Nodes
    • Resource groups and storage tiers
    • Ring resiliency
    • SQL query usage
    • System events and errors

UI

KAgent

  • KAgent has undergone a visual refresh as well as received some usability improvements.
  • The KAgent user interface has added support for all new features listed in the KAgent section above.

GAdmin

  • GAdmin has undergone a visual refresh as well as received some usability improvements.
  • Viewing a list of tables/views will no longer prevent them from expiring by TTL
  • Transitioned support for collections to schemas.

Reveal

  • Kinetica Reveal has undergone a visual refresh as well as received some usability improvements.
  • Transitioned support for collections to schemas

Version 7.1.7

Build Date: 2022.06.24

Highlights

  • Kinetica Workbench web interface for developing and collaborating with interactive SQL workbooks
  • JDBC ingress and egress
  • Import and export data to/from other data sources and enterprise applications
  • Reveal filter slice allows additional map layers to be used for filtering base layer
  • New track SQL functions

SQL/OLAP

  • New functions:
    • ST_TRACK_DWITHIN
    • ST_TRACKINTERSECTS
    • ST_TRACKDURATION
    • ST_TRACKLENGTH
  • Other new functions:
    • ISNUMERIC
    • WIDTH_BUCKET
  • Added support for ILIKE operator
  • New SQL command to upload a file to KiFS from a URL
  • Support for renaming schemas
  • Support for renaming and moving materialized views
  • Added SQL Support for generating isochrones
  • SQL Procedures can now invoke UDFs
  • Added support for pg_roles and pg_type catalog tables
  • Statistics and errors now saved for data imports and exports
  • Track rendering now supported on joins and materialized views
  • Track rendering now supports alternate ID and timestamp column names
  • Logical views, logical external tables, and catalog tables are now accessible from the native API
  • Materialized views are now updated, when needed, after multi-head inserts
  • PostgreSQL Wire Protocol support now includes Extended Query Protocol
  • KiSQL now supports line editing and line history
  • Improved performance of ingestion and copying of data with init_with_uuid column property
  • Improved UUID generation algorithm to produce more unique values for large batch inserts
  • Added support for chunk-skipping in series partitions, when query includes the partition key
  • Input UUIDs do not require hyphens
  • Support for optional AM/PM in default date format
  • Added LLL (whole milliseconds) and $ (start of optional section) code to TO_CHAR family of functions
  • Updated CAST function to work more consistently with other databases
  • K-means clustering algorithm now supported on CPUs, join views, and materialized views
  • Support for multiple subscriptions to same Kafka topic, if done by different users

Infrastructure

  • Google Cloud Storage support
    • Direct ingestion from files in GCS
    • External tables from files in GCS
    • GCS-based cold storage tier
  • JDBC ingress support
    • Direct ingestion from queries on JDBC-accessible databases
    • External tables from queries on JDBC-accessible databases
  • JDBC Egress support
    • Export Kinetica query results into JDBC-accessible databases
  • Broader support for SQL & NoSQL databases and enterprise applications via Kinetica data sources
  • Support for KiFS files in configuration settings and credential, data source, & data sink properties that reference files
  • Support customer-managed keys for AWS S3 access
  • Allow S3 data source or cold storage user to be able specify the server-side encryption method and key
  • Enable S3 IAM role parameter in configuration settings for cold storage

Geospatial/Network Graph/Visualization

  • New /match/graph solver, match_charging_stations, for finding an optimal route involving multiple electric-vehicle charging stations
  • Distributed graph support for /query/graph

APIs/Tools/Connectors

  • Java API
    • Added capability to pass in self-signed certificates & passwords as options
    • Updated dependencies to more secure versions
    • Fixed unhandled exception when an Avro encoding error occurs
    • Fixed error where a type's column order would not match a table created from it
    • Removed client-side primary key check, to improve performance and make returned errors more consistently delivered
  • Python API
    • Made the API more Python3 compatible
    • Prevented client hanging when connection IP/URL does not match any known to the server; client will operate in degraded mode (no multi-head, etc.)
    • Removed client-side primary key check, to improve performance and make returned errors more consistently delivered
    • Rectified a formatting issue while building expressions for keyed lookups that was resulting in a failure on Python 2.7.x.
    • Corrected some string/null comparisons
  • C++/C#
    • Removed client-side primary key check, to improve performance and make returned errors more consistently delivered

UI

  • Kinetica Workbench web interface for developing and collaborating with interactive SQL workbooks
  • GAdmin viewer for SVG animation results in Graph Match UI
  • Allow Reveal additional map layers to be used for filtering base layer
  • Allow Reveal map tracks click and highlight
  • Allow Reveal to be embedded in an iframe

OS Support

  • Added support for Kinetica database on Alma Linux 8.6
  • Added support for Kinetica database on Rocky Linux 8.5

Version 7.1.6

Build Date: 2022.01.28

Highlights

  • PostgreSQL wire protocol support
  • PREDICT & OUTLIER functions
  • Continuous ML model evaluation
  • Avro file ingestion
  • Metadata catalogs
  • Partitioned graph rebalancing
  • Time-based flush for ingestion via Java

SQL/OLAP

  • Added Metadata Catalogs:
    • ki_catalog - Kinetica-specific
    • information_schema - SQL-92 ANSI standard
    • pg_catalog - PostgreSQL-specific
  • Added schema-based namespacing for credentials, data sources, data sinks, graphs, & streams; allowing any of these object types with a given name to exist in more than one schema
  • Added PREDICT & OUTLIERS SQL ML functions for predicting values and calculating outliers from a data set without a model
  • Added SKEW & KURTOSIS (and variants of each) SQL aggregation functions
  • Improved the consistency of the security model across object types:
    • Data sources, data sinks, & streams now have an administrative permission
    • Graphs now have read, write, & admin permissions manageable through SQL
  • Added PostgreSQL Wire Protocol support for simple query mode and metadata tables
  • Added support for ingesting Avro files using an external table or LOAD INTO
  • Added support for streams on materialized views
  • Improved support for date/time formatting codes across conversion functions
  • Added functions to convert from a string date/time into the corresponding data type:
    • TO_DATE
    • TO_TIME
    • TO_DATETIME
    • TO_TIMESTAMP

Geospatial/Network Graph/Visualization

  • Added a new endpoint (/repartition/graph) for rebalancing partitioned graphs into clusters for random sharding on IDs/names
  • Differentiated node configurations by the order of identifiers specified; for example:
    • NODE_WKTPOINT, NODE_ID - fuses nodes based on the NODE_WKTPOINT
    • NODE_ID, NODE_WKTPOINT - creates unique nodes for each NODE_ID and will not fuse points based on the same NODE_WKTPOINT
  • Improved error reporting for graph SQL
  • Made EDGE_LABEL & EDGE_WEIGHT_VALUESPECIFIED optional identifiers for all edge combinations
  • Added the ability to define multiple labels over nodes and edges using a delimiter for any graph type--previously this was only possible on undirected graphs
  • Shortened identifier names for graph components when creating graphs, query identifiers when querying graphs, and match identifiers when matching graphs; for example, WKTPOINT can be used in place of NODE_WKTPOINT as a node identifier when creating a graph, though either will still work
  • Graph resources are now governed by resource groups
  • Improved performance of dissolve processing

APIs/Tools/Connectors

  • Java API
    • Changed logging from using Log4j 1.x to SLF4J with a default Logback stdout logger
    • Added timed flush mechanism for the BulkInserter to improve ingest operation in data starvation scenarios
    • Added automatic clean up of BulkInserter resources upon shutdown
    • Made failed records during BulkInserter ingestion accessible via getErrors()
    • Improved HA failover logging
  • Python API
    • Fixed password-in-the-clear security issue when running in debug mode
  • Storm Connector
    • Updated log4j to version 2.17.1

ML

  • Added support for introspection of model and custom ingestor images from both public and private (credential based) registries including docker.io & ACR

Version 7.1.5

Build Date: 2021.08.13

SQL/OLAP

  • New SQL commands for managing files & directories in KiFS
  • New SQL command for loading Reveal dashboards from GitHub, Amazon S3, or a local file
  • Added support for data sources on public S3 buckets and Azure BLOB stores
  • Added data sinks, which are references to external systems that can receive data streamed from Kinetica, in SQL and natively
  • Added filter-based triggers for detecting changes in data over time
  • Added a default set of libraries for use with Python UDFs

Geospatial/Network Graph/Visualization

  • Added graph security
  • Ported non-path solvers to work with partitioned graphs
  • Added graph elasticity on cloud deployments

APIs/Tools/Connectors

  • Added support for uploading & downloading of files from a client using KiSQL
  • Added multipart uploading of files to KiFS in the Java API

UI

Workbench

  • Enabled multipart file uploading, which removes previous limits on the total file sizes allowed by the drag & drop uploader
  • Track & contour rendering options added to Map widget
  • Scatter plot chart added as visualization option for SQL query block

Reveal

  • UX streamlined with various workflow improvements

Version 7.1.4

Build Date: 2021.07.26

SQL/OLAP

  • Support for streaming from Azure, Kafka, and S3 data sources
  • Support for shapefile ingestion
  • Added managed credential framework for authenticating to external data sources and model repositories, in SQL and natively
  • New SQL commands for managing & executing UDFs
  • New SQL commands for managing & querying/matching/solving graphs
  • New SQL commands for managing & executing ML models
  • Support for looking up specific granted permissions and roles, in SQL and natively
  • Improved KiFS and its integration within the database

Geospatial/Network Graph/Visualization

  • Enabled distributed solves for name (string) based property graphs
  • Added feature attributes to the Vector Tile Service output
  • Added new Eulerian Paths solver, useful for fraud detection; see MATCH_LOOPS solver type in /match/graph
  • Added graph statistics, including diameter, average valence, clusters, average and maximum cluster sizes available as STATS_ALL solver type in /solve/graph
  • Added new Centrality Closeness solver, useful for fraud detection; see CLOSENESS solver type in /solve/graph
  • Added max_truck_drop_limit to the Multiple Supply Demand solver
  • Added ability to export SVG embedded html output for Multiple Supply Demand solver for real time dynamic simulations with routing animation (speed adjustable for tracking)
  • Added API endpoints for creating and retrieving WMS videos; can be overlaid on many basemap providers
  • Enabled distributed shortest path solves for non-geospatial graphs
  • Added maximum solution targets limit for Centrality and Page Rank solvers for performance tuning
  • Various performance improvements and bug fixes

AAW

  • Support for ML management & execution commands via SQL
  • Automatic completion and teardown of batch mode model jobs
  • Improved logging (custom log levels, redirection etc.)
  • Server level performance and stability improvements
  • More efficient reporting of model running state in user interface

APIs/Tools/Connectors

UI

Workbench

  • Added class break rendering mode to the map block visualization.
  • Added advanced WMS parameter input overrides to the map block visualization.

Reveal

  • Streamlined the dashboard and slice creation workflow, navigation, and user experience.

Version 7.1.3

Build Date: 2021.02.22

SQL/OLAP

  • Pattern matching for fixed-width string columns and full text search for unrestricted-width string columns via the new SQL function FILTER_BY_STRING
  • External tables & ingestion (via data sources) can now access Azure resources
  • Added JSON & GeoJSON format to supported file types for ingest

Geospatial/Network Graph/Visualization

  • Added new graph identifiers for large social graph partitioning
  • Enabled distributed solves on large social graphs
  • Enabled distributed solves for one source to many destinations
  • Various performance improvements for WMS track rendering on CPU
  • Added WMS parameter for track line simplification threshold

AAW

  • New UUID data type used for inference tracking
  • Configurable Node Selector for model deployments across alternative Scale Sets
  • Configurable and independent Image Pull Policies for system containers and model/custom ingestor containers
  • Optional custom ingress path names for on-demand models
  • Improved model startup and shutdown time
  • Capability to run in Cloud Native mode (K8s managed application)

APIs/Tools/Connectors

  • Multi-head key lookups now support replicated tables

UI

  • GAdmin
    • Added another digit of precision to the resource statistics memory usage page
    • Added support for new graph partitioning identifiers

Version 7.1.2

Build Date: 2020.11.12

SQL/OLAP

  • Added support for SQL procedures to be executed as another user using the EXECUTE AS syntax
  • Added support for short-circuiting errors using the conditional function IF and operators AND & OR
  • Added four new functions:
    • LOG1P(expr) -- Returns the natural logarithm of one plus expr (a double). Effectively evaluates as log(1+expr)
    • LOG2(expr) -- Returns the binary (base-2) logarithm of the given expression expr (a double)
    • LOG1PF(expr) -- Returns the natural logarithm of one plus expr (a float). Effectively evaluates as log(1+expr)
    • LOG2F(expr) -- Returns the binary (base-2) logarithm of the given expression expr (a float)
  • Removed support for using temporary tables within procedures
  • Added support for new SQL statement EXECUTE ENDPOINT that allows users to execute any REST API endpoint
  • Improved performance of handling GROUP BY operations resulting in a sharded table
  • Improved performance of JOIN operations involving inequalities between columns in different tables
  • Improved performance of /append/records operations, i.e. SQL INSERT with SELECT

Geospatial/Network Graph/Visualization

  • Added support for symbology for tracks
  • Corrected issues with invalid VTS render types
  • Added support for solving replicated graphs using the shortest path solver
  • Improved performance of track marker rendering
  • Added support for tracks in the solution output for the match_supply_demand and markov_chain /match/graph solvers:
    • For markov_chain solutions, the output details table's TRIPID column, a long, has been changed to TRACKID, a char64 sharded column. This new column can be consumed in appropriate /wms track-style calls
    • For match_supply_demand solutions, a new output_tracks option is available that will update the output table to comply with TRACKID and TIMESTAMP formatting, where the TIMESTAMP is correct with respect to real-time truck travel (if the weight field of the graph was time-based). This option is useful if real-time dynamic routing simulation is desired

APIs/Tools/Connectors

  • Python API
    • Added support for custom headers, which enables SQL-like user impersonation at the native API level
    • Improved and simplified the table monitor API
    • Added support for sharding on UUID type columns in multi-head egress/ingest
  • JavaScript API
    • Added support for custom headers, which enables SQL-like user impersonation at the native API level
  • Node.js API
    • Added support for custom headers, which enables SQL-like user impersonation at the native API level

UI

  • Reveal
    • Added support for sorting histogram columns by field name

Version 7.1.1

Build Date: 2020.10.22

SQL/OLAP

  • Added support for partitioning by series
  • Added support for row- and column-level security to materialized views
  • Added a new ON QUERY refresh mode for materialized views.
  • Added support for UPDATE statements that contain JOINs with other tables, including updating values from expressions involving other tables
  • Improved efficiency of JOIN operations that use inequality within the JOIN expression
  • Added support for a new uuid (GUID) type as well as related functions

Infrastructure

Geospatial/Network Graph/Visualization

  • Improvements to track and shape rendering:
    • Improved memory management
    • Corrected WMS rendering that straddled hemisphere lines
    • Improved the /filter/byradius endpoint to be spherical aware

APIs/Tools/Connectors

  • Java API
    • Added support for overriding system IP addresses with user-defined IP addresses provided to the API
  • JavaScript API
    • Added support for custom headers, which enables SQL-like user impersonation at the native API level
  • NodeJS API
    • Added support for custom headers, which enables SQL-like user impersonation at the native API level
  • Python API
    • Improved and simplified the table monitor API
    • Added support for using the new uuid type with multi-head input and output
    • Added support for the trace log level
    • Added support for custom headers, which enables SQL-like user impersonation at the native API level
  • Kafka Connector
    • Added a new option, kinetica.enable_multihead, to enable multi-head data ingest for the sink connector
  • RKinetica
    • Added support for schema namespacing

UI

  • Reveal
    • Improved general usability
  • GAdmin
    • Added support for a new uuid (GUID) type
    • Improved general usability
  • KAgent
    • Added support for mapping external authentication roles during security configuration
    • Added a new LDAPS option to the security configuration