7.1 Release Notes

Core

• Ring Resiliency (High Availability) is now integrated into the core of the product; a separate package & application suite is no longer needed
• As part of the namespaced schemas change, all top-level tables & views will be relocated to the home schema.

APIs/Tools/Connectors

• All APIs, KIO, and the Spark connector have been updated to support schemas.
• APIs:
  • The Java API has been updated to support cluster resiliency and the new ring resiliency changes
  • The Java API now supports logging via Apache Log4j
  • The Java API setHostManagerPort option will no longer work after the database object has been initialized
• KIO:
  • Due to the advent of schemas, the --d-collection-name Kinetica format option has been deprecated

KAgent

• Added cluster resiliency support
• Added support for switching responsibility for one host's processes to another host, on demand
• Added support for configuring metric-based and event-based alerts
• Added support for configuring delivery channels (e.g, email) for alerts
• Added/improved monitoring of:
  • Rings
  • Clusters
  • Nodes
  • Resource groups and storage tiers
  • Ring resiliency
  • SQL query usage
  • System events and errors

UI

Highlights

• PostgreSQL wire protocol support
• PREDICT & OUTLIER functions
• Continuous ML model evaluation
• Avro file ingestion
• Metadata catalogs
• Partitioned graph rebalancing
• Time-based flush for ingestion via Java

SQL/OLAP

• Added Metadata Catalogs:
  • ki_catalog - Kinetica-specific
  • information_schema - SQL-92 ANSI standard
  • pg_catalog - PostgreSQL-specific
• Added schema-based namespacing for credentials, data sources, data sinks, graphs, & streams; allowing any of these object types with a given name to exist in more than one schema
• Added PREDICT & OUTLIERS SQL ML functions for predicting values and calculating outliers from a data set without a model
• Added SKEW & KURTOSIS (and variants of each) SQL aggregation functions
• Improved the consistency of the security model across object types:
  • Data sources, data sinks, & streams now have an administrative permission
  • Graphs now have read, write, & admin permissions manageable through SQL
• Added PostgreSQL Wire Protocol support for simple query mode and metadata tables
• Added support for ingesting Avro files using an external table or LOAD INTO
• Added support for streams on materialized views
• Improved support for date/time formatting codes across conversion functions
• Added functions to convert from a string date/time into the corresponding data type:
  • TO_DATE
  • TO_TIME
  • TO_DATETIME
  • TO_TIMESTAMP

Geospatial/Network Graph/Visualization

• Added a new endpoint (/repartition/graph) for rebalancing partitioned graphs into clusters for random sharding on IDs/names
• Differentiated node configurations by the order of identifiers specified; for example:
  • NODE_WKTPOINT, NODE_ID - fuses nodes based on the NODE_WKTPOINT
  • NODE_ID, NODE_WKTPOINT - creates unique nodes for each NODE_ID and will not fuse points based on the same NODE_WKTPOINT
• Improved error reporting for graph SQL
• Made EDGE_LABEL & EDGE_WEIGHT_VALUESPECIFIED optional identifiers for all edge combinations
• Added the ability to define multiple labels over nodes and edges using a delimiter for any graph type--previously this was only possible on undirected graphs
• Shortened identifier names for graph components when creating graphs, query identifiers when querying graphs, and match identifiers when matching graphs; for example, WKTPOINT can be used in place of NODE_WKTPOINT as a node identifier when creating a graph, though either will still work
• Graph resources are now governed by resource groups
• Improved performance of dissolve processing

APIs/Tools/Connectors

• Java API
  • Changed logging from using Log4j 1.x to SLF4J with a default Logback stdout logger
  • Added timed flush mechanism for the BulkInserter to improve ingest operation in data starvation scenarios
  • Added automatic clean up of BulkInserter resources upon shutdown
  • Made failed records during BulkInserter ingestion accessible via getErrors()
  • Improved HA failover logging
• Python API
  • Fixed password-in-the-clear security issue when running in debug mode
• Storm Connector
  • Updated log4j to version 2.17.1

ML

• Added support for introspection of model and custom ingestor images from both public and private (credential based) registries including docker.io & ACR

SQL/OLAP

• New SQL commands for managing files & directories in KiFS
• New SQL command for loading Reveal dashboards from GitHub, Amazon S3, or a local file
• Added support for data sources on public S3 buckets and Azure BLOB stores
• Added data sinks, which are references to external systems that can receive data streamed from Kinetica, in SQL and natively
• Added filter-based triggers for detecting changes in data over time
• Added a default set of libraries for use with Python UDFs

Geospatial/Network Graph/Visualization

• Added graph security
• Ported non-path solvers to work with partitioned graphs
• Added graph elasticity on cloud deployments

APIs/Tools/Connectors

• Added support for uploading & downloading of files from a client using KiSQL
• Added multipart uploading of files to KiFS in the Java API

UI

SQL/OLAP

• Support for streaming from Azure, Kafka, and S3 data sources
• Support for shapefile ingestion
• Added managed credential framework for authenticating to external data sources and model repositories, in SQL and natively
• New SQL commands for managing & executing UDFs
• New SQL commands for managing & querying/matching/solving graphs
• New SQL commands for managing & executing ML models
• Support for looking up specific granted permissions and roles, in SQL and natively
• Improved KiFS and its integration within the database

Geospatial/Network Graph/Visualization

• Enabled distributed solves for name (string) based property graphs
• Added feature attributes to the Vector Tile Service output
• Added new Eulerian Paths solver, useful for fraud detection; see MATCH_LOOPS solver type in /match/graph
• Added graph statistics, including diameter, average valence, clusters, average and maximum cluster sizes available as STATS_ALL solver type in /solve/graph
• Added new Centrality Closeness solver, useful for fraud detection; see CLOSENESS solver type in /solve/graph
• Added max_truck_drop_limit to the Multiple Supply Demand solver
• Added ability to export SVG embedded html output for Multiple Supply Demand solver for real time dynamic simulations with routing animation (speed adjustable for tracking)
• Added API endpoints for creating and retrieving WMS videos; can be overlaid on many basemap providers
• Enabled distributed shortest path solves for non-geospatial graphs
• Added maximum solution targets limit for Centrality and Page Rank solvers for performance tuning
• Various performance improvements and bug fixes

AAW

• Support for ML management & execution commands via SQL
• Automatic completion and teardown of batch mode model jobs
• Improved logging (custom log levels, redirection etc.)
• Server level performance and stability improvements
• More efficient reporting of model running state in user interface

APIs/Tools/Connectors

• Added capability to upload files to Kinetica File System (KiFS) to the Java API

UI

SQL/OLAP

• Pattern matching for fixed-width string columns and full text search for unrestricted-width string columns via the new SQL function FILTER_BY_STRING
• External tables & ingestion (via data sources) can now access Azure resources
• Added JSON & GeoJSON format to supported file types for ingest

Geospatial/Network Graph/Visualization

• Added new graph identifiers for large social graph partitioning
• Enabled distributed solves on large social graphs
• Enabled distributed solves for one source to many destinations
• Various performance improvements for WMS track rendering on CPU
• Added WMS parameter for track line simplification threshold

AAW

• New UUID data type used for inference tracking
• Configurable Node Selector for model deployments across alternative Scale Sets
• Configurable and independent Image Pull Policies for system containers and model/custom ingestor containers
• Optional custom ingress path names for on-demand models
• Improved model startup and shutdown time
• Capability to run in Cloud Native mode (K8s managed application)

APIs/Tools/Connectors

• Multi-head key lookups now support replicated tables

UI

• GAdmin
  • Added another digit of precision to the resource statistics memory usage page
  • Added support for new graph partitioning identifiers

SQL/OLAP

• Added support for SQL procedures to be executed as another user using the EXECUTE AS syntax
• Added support for short-circuiting errors using the conditional function IF and operators AND & OR
• Added four new functions:
  • LOG1P(expr) -- Returns the natural logarithm of one plus expr (a double). Effectively evaluates as log(1+expr)
  • LOG2(expr) -- Returns the binary (base-2) logarithm of the given expression expr (a double)
  • LOG1PF(expr) -- Returns the natural logarithm of one plus expr (a float). Effectively evaluates as log(1+expr)
  • LOG2F(expr) -- Returns the binary (base-2) logarithm of the given expression expr (a float)
• Removed support for using temporary tables within procedures
• Added support for new SQL statement EXECUTE ENDPOINT that allows users to execute any REST API endpoint
• Improved performance of handling GROUP BY operations resulting in a sharded table
• Improved performance of JOIN operations involving inequalities between columns in different tables
• Improved performance of /append/records operations, i.e. SQL INSERT with SELECT

Geospatial/Network Graph/Visualization

• Added support for symbology for tracks
• Corrected issues with invalid VTS render types
• Added support for solving replicated graphs using the shortest path solver
• Improved performance of track marker rendering
• Added support for tracks in the solution output for the match_supply_demand and markov_chain /match/graph solvers:
  • For markov_chain solutions, the output details table's TRIPID column, a long, has been changed to TRACKID, a char64 sharded column. This new column can be consumed in appropriate /wms track-style calls
  • For match_supply_demand solutions, a new output_tracks option is available that will update the output table to comply with TRACKID and TIMESTAMP formatting, where the TIMESTAMP is correct with respect to real-time truck travel (if the weight field of the graph was time-based). This option is useful if real-time dynamic routing simulation is desired

APIs/Tools/Connectors

• Python API
  • Added support for custom headers, which enables SQL-like user impersonation at the native API level
  • Improved and simplified the table monitor API
  • Added support for sharding on UUID type columns in multi-head egress/ingest
• JavaScript API
  • Added support for custom headers, which enables SQL-like user impersonation at the native API level
• Node.js API
  • Added support for custom headers, which enables SQL-like user impersonation at the native API level

UI

• Reveal
  • Added support for sorting histogram columns by field name

SQL/OLAP

• Added support for partitioning by series
• Added support for row- and column-level security to materialized views
• Added a new ON QUERY refresh mode for materialized views.
• Added support for UPDATE statements that contain JOINs with other tables, including updating values from expressions involving other tables
• Improved efficiency of JOIN operations that use inequality within the JOIN expression
• Added support for a new uuid (GUID) type as well as related functions

Infrastructure

• The cold storage tier for resource management now supports Microsoft Azure blob and data lake storage

Geospatial/Network Graph/Visualization

• Improvements to track and shape rendering:
  • Improved memory management
  • Corrected WMS rendering that straddled hemisphere lines
  • Improved the /filter/byradius endpoint to be spherical aware

APIs/Tools/Connectors

• Java API
  • Added support for overriding system IP addresses with user-defined IP addresses provided to the API
• JavaScript API
  • Added support for custom headers, which enables SQL-like user impersonation at the native API level
• NodeJS API
  • Added support for custom headers, which enables SQL-like user impersonation at the native API level
• Python API
  • Improved and simplified the table monitor API
  • Added support for using the new uuid type with multi-head input and output
  • Added support for the trace log level
  • Added support for custom headers, which enables SQL-like user impersonation at the native API level
• Kafka Connector
  • Added a new option, kinetica.enable_multihead, to enable multi-head data ingest for the sink connector
• RKinetica
  • Added support for schema namespacing

UI

• Reveal
  • Improved general usability
• GAdmin
  • Added support for a new uuid (GUID) type
  • Improved general usability
• KAgent
  • Added support for mapping external authentication roles during security configuration
  • Added a new LDAPS option to the security configuration