Publish Date: 2020.08.19
Features
- Cluster Resiliency allows failed processes within a cluster to be shifted to other active or passive hosts within the cluster
- Schemas (formerly collections) are now namespaced, allowing two or more schemas to contain a table with the same name; the native API will now accept <schema_name>.<table_name> references to tables & views
- Externally-managed data sources can be the source of both Tables and data load commands (LOAD INTO, /insert/records/fromfiles)
- Improved system & cluster monitoring capability
Core
- Ring Resiliency (High Availability) is now integrated into the core of the product; a separate package & application suite is no longer needed
- As part of the namespaced schemas change, all top-level tables & views will be relocated to the home schema.
APIs/Tools/Connectors
- All APIs, KIO, and the Spark connector have been updated to support schemas.
- APIs:
- The Java API has been updated to support cluster resiliency and the new ring resiliency changes
- The Java API now supports logging via Apache Log4j
- The Java API setHostManagerPort option will no longer work after the database object has been initialized
- KIO:
- Due to the advent of schemas, the --d-collection-name Kinetica format option has been deprecated
KAgent
- Added cluster resiliency support
- Added support for switching responsibility for one host's processes to another host, on demand
- Added support for configuring metric-based and event-based alerts
- Added support for configuring delivery channels (e.g, email) for alerts
- Added/improved monitoring of:
- Rings
- Clusters
- Nodes
- Resource groups and storage tiers
- Ring resiliency
- SQL query usage
- System events and errors
UI
KAgent
GAdmin
- GAdmin has undergone a visual refresh as well as received some usability improvements.
- Viewing a list of tables/views will no longer prevent them from expiring by TTL
- Transitioned support for collections to schemas.
Reveal
- Kinetica Reveal has undergone a visual refresh as well as received some usability improvements.
- Transitioned support for collections to schemas
Version 7.1.6
Build Date: 2022.01.28
Highlights
- PostgreSQL wire protocol support
- PREDICT & OUTLIER functions
- Continuous ML model evaluation
- Avro file ingestion
- Metadata catalogs
- Partitioned graph rebalancing
- Time-based flush for ingestion via Java
SQL/OLAP
- Added Metadata Catalogs:
- ki_catalog - Kinetica-specific
- information_schema - SQL-92 ANSI standard
- pg_catalog - PostgreSQL-specific
- Added schema-based namespacing for credentials, data sources, data sinks, graphs, & streams; allowing any of these object types with a given name to exist in more than one schema
- Added PREDICT & OUTLIERS SQL ML functions for predicting values and calculating outliers from a data set without a model
- Added SKEW & KURTOSIS (and variants of each) SQL aggregation functions
- Improved the consistency of the security model across object types:
- Data sources, data sinks, & streams now have an administrative permission
- Graphs now have read, write, & admin permissions manageable through SQL
- Added PostgreSQL Wire Protocol support for simple query mode and metadata tables
- Added support for ingesting Avro files using an external table or LOAD INTO
- Added support for streams on materialized views
- Improved support for date/time formatting codes across conversion functions
- Added functions to convert
from a string date/time into the corresponding data type:
- TO_DATE
- TO_TIME
- TO_DATETIME
- TO_TIMESTAMP
Geospatial/Network Graph/Visualization
- Added a new endpoint (/repartition/graph) for rebalancing partitioned graphs into clusters for random sharding on IDs/names
- Differentiated node configurations by the order of identifiers specified; for
example:
- NODE_WKTPOINT, NODE_ID - fuses nodes based on the NODE_WKTPOINT
- NODE_ID, NODE_WKTPOINT - creates unique nodes for each NODE_ID and will not fuse points based on the same NODE_WKTPOINT
- Improved error reporting for graph SQL
- Made EDGE_LABEL & EDGE_WEIGHT_VALUESPECIFIED optional identifiers for all edge combinations
- Added the ability to define multiple labels over nodes and edges using a delimiter for any graph type--previously this was only possible on undirected graphs
- Shortened identifier names for graph components when creating graphs, query identifiers when querying graphs, and match identifiers when matching graphs; for example, WKTPOINT can be used in place of NODE_WKTPOINT as a node identifier when creating a graph, though either will still work
- Graph resources are now governed by resource groups
- Improved performance of dissolve processing
APIs/Tools/Connectors
- Java API
- Changed logging from using Log4j 1.x to SLF4J with a default Logback stdout logger
- Added timed flush mechanism for the BulkInserter to improve ingest operation in data starvation scenarios
- Added automatic clean up of BulkInserter resources upon shutdown
- Made failed records during BulkInserter ingestion accessible via getErrors()
- Improved HA failover logging
- Python API
- Fixed password-in-the-clear security issue when running in debug mode
- Storm Connector
- Updated log4j to version 2.17.1
ML
- Added support for introspection of model and custom ingestor images from both public and private (credential based) registries including docker.io & ACR
Version 7.1.5
Build Date: 2021.08.13
SQL/OLAP
- New SQL commands for managing files & directories in KiFS
- New SQL command for loading Reveal dashboards from GitHub, Amazon S3, or a local file
- Added support for data sources on public S3 buckets and Azure BLOB stores
- Added data sinks, which are references to external systems that can receive data streamed from Kinetica, in SQL and natively
- Added filter-based triggers for detecting changes in data over time
- Added a default set of libraries for use with Python UDFs
Geospatial/Network Graph/Visualization
- Added graph security
- Ported non-path solvers to work with partitioned graphs
- Added graph elasticity on cloud deployments
APIs/Tools/Connectors
- Added support for uploading & downloading of files from a client using KiSQL
- Added multipart uploading of files to KiFS in the Java API
UI
Workbench
- Enabled multipart file uploading, which removes previous limits on the total file sizes allowed by the drag & drop uploader
- Track & contour rendering options added to Map widget
- Scatter plot chart added as visualization option for SQL query block
Reveal
- UX streamlined with various workflow improvements
Version 7.1.4
Build Date: 2021.07.26
SQL/OLAP
- Support for streaming from Azure, Kafka, and S3 data sources
- Support for shapefile ingestion
- Added managed credential framework for authenticating to external data sources and model repositories, in SQL and natively
- New SQL commands for managing & executing UDFs
- New SQL commands for managing & querying/matching/solving graphs
- New SQL commands for managing & executing ML models
- Support for looking up specific granted permissions and roles, in SQL and natively
- Improved KiFS and its integration within the database
Geospatial/Network Graph/Visualization
- Enabled distributed solves for name (string) based property graphs
- Added feature attributes to the Vector Tile Service output
- Added new Eulerian Paths solver, useful for fraud detection; see MATCH_LOOPS solver type in /match/graph
- Added graph statistics, including diameter, average valence, clusters, average and maximum cluster sizes available as STATS_ALL solver type in /solve/graph
- Added new Centrality Closeness solver, useful for fraud detection; see CLOSENESS solver type in /solve/graph
- Added max_truck_drop_limit to the Multiple Supply Demand solver
- Added ability to export SVG embedded html output for Multiple Supply Demand solver for real time dynamic simulations with routing animation (speed adjustable for tracking)
- Added API endpoints for creating and retrieving WMS videos; can be overlaid on many basemap providers
- Enabled distributed shortest path solves for non-geospatial graphs
- Added maximum solution targets limit for Centrality and Page Rank solvers for performance tuning
- Various performance improvements and bug fixes
AAW
- Support for ML management & execution commands via SQL
- Automatic completion and teardown of batch mode model jobs
- Improved logging (custom log levels, redirection etc.)
- Server level performance and stability improvements
- More efficient reporting of model running state in user interface
APIs/Tools/Connectors
- Added capability to upload files to Kinetica File System (KiFS) to the Java API
UI
Workbench
- Added class break rendering mode to the map block visualization.
- Added advanced WMS parameter input overrides to the map block visualization.
Reveal
- Streamlined the dashboard and slice creation workflow, navigation, and user experience.
Version 7.1.3
Build Date: 2021.02.22
SQL/OLAP
- Pattern matching for fixed-width string columns and full text search for unrestricted-width string columns via the new SQL function FILTER_BY_STRING
- External tables & ingestion (via data sources) can now access Azure resources
- Added JSON & GeoJSON format to supported file types for ingest
Geospatial/Network Graph/Visualization
- Added new graph identifiers for large social graph partitioning
- Enabled distributed solves on large social graphs
- Enabled distributed solves for one source to many destinations
- Various performance improvements for WMS track rendering on CPU
- Added WMS parameter for track line simplification threshold
AAW
- New UUID data type used for inference tracking
- Configurable Node Selector for model deployments across alternative Scale Sets
- Configurable and independent Image Pull Policies for system containers and model/custom ingestor containers
- Optional custom ingress path names for on-demand models
- Improved model startup and shutdown time
- Capability to run in Cloud Native mode (K8s managed application)
APIs/Tools/Connectors
- Multi-head key lookups now support replicated tables
UI
- GAdmin
- Added another digit of precision to the resource statistics memory usage page
- Added support for new graph partitioning identifiers
Version 7.1.2
Build Date: 2020.11.12
SQL/OLAP
- Added support for SQL procedures to be executed as another user using the EXECUTE AS syntax
- Added support for short-circuiting errors using the conditional function IF and operators AND & OR
- Added four new functions:
- LOG1P(expr) -- Returns the natural logarithm of one plus expr (a double). Effectively evaluates as log(1+expr)
- LOG2(expr) -- Returns the binary (base-2) logarithm of the given expression expr (a double)
- LOG1PF(expr) -- Returns the natural logarithm of one plus expr (a float). Effectively evaluates as log(1+expr)
- LOG2F(expr) -- Returns the binary (base-2) logarithm of the given expression expr (a float)
- Removed support for using temporary tables within procedures
- Added support for new SQL statement EXECUTE ENDPOINT that allows users to execute any REST API endpoint
- Improved performance of handling GROUP BY operations resulting in a sharded table
- Improved performance of JOIN operations involving inequalities between columns in different tables
- Improved performance of /append/records operations, i.e. SQL INSERT with SELECT
Geospatial/Network Graph/Visualization
- Added support for symbology for tracks
- Corrected issues with invalid VTS render types
- Added support for solving replicated graphs using the shortest path solver
- Improved performance of track marker rendering
- Added support for tracks in the solution output for the
match_supply_demand and markov_chain /match/graph
solvers:
- For markov_chain solutions, the output details table's TRIPID column, a long, has been changed to TRACKID, a char64 sharded column. This new column can be consumed in appropriate /wms track-style calls
- For match_supply_demand solutions, a new output_tracks option is available that will update the output table to comply with TRACKID and TIMESTAMP formatting, where the TIMESTAMP is correct with respect to real-time truck travel (if the weight field of the graph was time-based). This option is useful if real-time dynamic routing simulation is desired
APIs/Tools/Connectors
- Python API
- Added support for custom headers, which enables SQL-like user impersonation at the native API level
- Improved and simplified the table monitor API
- Added support for sharding on UUID type columns in multi-head egress/ingest
- JavaScript API
- Added support for custom headers, which enables SQL-like user impersonation at the native API level
- Node.js API
- Added support for custom headers, which enables SQL-like user impersonation at the native API level
UI
- Reveal
- Added support for sorting histogram columns by field name
Version 7.1.1
Build Date: 2020.10.22
SQL/OLAP
- Added support for partitioning by series
- Added support for row- and column-level security to materialized views
- Added a new ON QUERY refresh mode for materialized views.
- Added support for UPDATE statements that contain JOINs with other tables, including updating values from expressions involving other tables
- Improved efficiency of JOIN operations that use inequality within the JOIN expression
- Added support for a new uuid (GUID) type as well as related functions
Infrastructure
- The cold storage tier for resource management now supports Microsoft Azure blob and data lake storage
Geospatial/Network Graph/Visualization
- Improvements to track and shape rendering:
- Improved memory management
- Corrected WMS rendering that straddled hemisphere lines
- Improved the /filter/byradius endpoint to be spherical aware
APIs/Tools/Connectors
- Java API
- Added support for overriding system IP addresses with user-defined IP addresses provided to the API
- JavaScript API
- Added support for custom headers, which enables SQL-like user impersonation at the native API level
- NodeJS API
- Added support for custom headers, which enables SQL-like user impersonation at the native API level
- Python API
- Improved and simplified the table monitor API
- Added support for using the new uuid type with multi-head input and output
- Added support for the trace log level
- Added support for custom headers, which enables SQL-like user impersonation at the native API level
- Kafka Connector
- Added a new option, kinetica.enable_multihead, to enable multi-head data ingest for the sink connector
- RKinetica
- Added support for schema namespacing
UI
- Reveal
- Improved general usability
- GAdmin
- Added support for a new uuid (GUID) type
- Improved general usability
- KAgent
- Added support for mapping external authentication roles during security configuration
- Added a new LDAPS option to the security configuration