Before You Start
Note that provisioning Kinetica through the AWS Marketplace will incur charges in your AWS account. These charges will depend on the type and size of the cluster you provision as well as the hours of usage. Once the provisioning is complete, the database will start running.
Pre-requisites
AWS Account
An AWS account is required to provision Marketplace applications, including Kinetica.
Note
The EC2 compute instances required by Kinetica are not covered by the AWS free usage tier.
AWS Service Quota
Make sure you have an adequate quota in your AWS subscription and AWS region for the following resources. The number of required instances will depend on the size and configuration of your Kinetica cluster. For details on the capabilities and specifications of the different cluster sizes, see Hardware Configuration.
Quota Type | Service Code | # Required | Default Quota |
---|---|---|---|
VPCs per Region | vpc | 2 | 5 |
Subnets per VPC | vpc | 7 | 100 |
Route Tables per VPC | vpc | 6 | 200 |
Internet Gateways per Region | vpc | 2 | 5 |
Network Load Balancers per Region | elasticloadbalancing | 1 | 50 |
EC2-VPC Elastic IPs | ec2 | 2 | 5 |
NAT Gateways per Availability Zone | vpc | 2 | 5 |
Network ACLs per VPC | vpc | 2 | 200 |
VPC Security Groups per Region | vpc | 5 | 2500 |
Clusters | eks | 1 | 100 |
EC2 Quotas for CPU-Based Clusters
Quota Type | Cluster Size | # Required | Default Quota |
---|---|---|---|
Running On-Demand Standard Instances (A, C, D, H, I, M, R, T, Z) | XS | 20 | 5 |
S | 36 | ||
M | 72 |
EC2 Quotas for GPU-Based Clusters
Quota Type | Cluster Size | # Required | Default Quota |
---|---|---|---|
Running On-Demand Standard Instances (A, C, D, H, I, M, R, T, Z) | XS | 4 | 5 |
S | 4 | ||
M | 8 | ||
Running On-Demand G & VT Instances | XS | 32 | 0 |
S | 64 | ||
M | 128 |
AWS Permissions
The AWS user who provisions Kinetica from the AWS Marketplace must have the necessary permissions to buy and use AWS Marketplace applications and provision the required AWS resources.
For more information on IAM permissions to use the AWS Marketplace, see the AWS documentation:
The Kinetica Cloud Formation Template will create a VPC, EKS cluster, IAM roles, Lambda functions, and Step Functions during provisioning and users will need to have the necessary permissions to create those resources.
Note
It is recommended that the AWS user has Administrator Access to be able to provision Kinetica.
Provisioning
Choose an Offer
There are two different offers in the AWS Marketplace for Kinetica, depending on your preferred consumption and payment model:
Larger cluster sizes and configurations are only available in the Bring Your Own License (BYOL) offer. All other features and functions are the same between the two offers.
Kinetica Pay As You Go (PAYG)
With this offer, Kinetica license fees are based on your hourly usage and the size of the cluster. See Specify Stack Details for specifics on cluster sizes, hardware configurations, and hourly license fees.
To find the Kinetica Pay As You Go (PAYG) Offer, you can search for Kinetica under Discover Products in the AWS Marketplace:
Kinetica Bring Your Own License (BYOL)
For customers with an existing Kinetica license, you can choose this option and input your existing Kinetica license key. You will not incur additional Kinetica license charges, although you will be charged for AWS infrastructure costs. See Hardware Configuration for more details on the AWS infrastructure provisioned with each cluster configuration.
To find the Kinetica Bring Your Own License (BYOL) Offer, you can search for Kinetica (Bring Your Own License) under Discover Products in the AWS Marketplace:
Subscribe to the Product
If you are already familiar with AWS Marketplace and CloudFormation Stacks, you can skip to Specify Stack Details for information on configuring your Kinetica cluster.
The first step in the AWS Marketplace is to subscribe to the product. Click on Continue to Subscribe to purchase an entitlement for Kinetica.
Note
You will not incur any charges for purchasing an entitlement for Kinetica.
Review Terms and Conditions
Review the Terms and Conditions and click Accept Terms.
Once the request has completed, you will be able to configure Kinetica. Click Continue to Configuration.
Configure the Software
Kinetica has a single fulfillment option (On EKS with New VPC) which will use Cloud Formation to deploy Kinetica into an Elastic Kubernetes Service (EKS) cluster in a new VPC. This deployment option will deploy into your selected region in a single Availability Zone.
Choose the version of the product you would like to deploy and click Continue to Launch.
Launch the Software
Under Configuration details, you can review the Usage instructions and find links to this documentation.
Click on the link under Deployment template; e.g., Kinetica on EKS with New VPC.
Create Stack
A new browser tab will open where you can create a CloudFormation stack based on the Kinetica template. The fields are pre-populated with the S3 URL for the Kinetica template. You can download and inspect the Cloud Formation template or view it in the Designer.
Make sure you have selected the correct AWS region (in the top right corner) for your deployment and click Next to Specify stack details. The following regions are currently supported:
Region | Code |
---|---|
Asia Pacific (Hong Kong) | ap-east-1 |
Asia Pacific (Mumbai) | ap-south-1 |
Asia Pacific (Seoul) | ap-northeast-2 |
Asia Pacific (Singapore) | ap-southeast-1 |
Asia Pacific (Sydney) | ap-southeast-2 |
Asia Pacific (Tokyo) | ap-northeast-1 |
Canada (Central) | ca-central-1 |
Europe (Frankfurt) | eu-central-1 |
Europe (Ireland) | eu-west-1 |
Europe (London) | eu-west-2 |
Europe (Paris) | eu-west-3 |
Europe (Stockholm) | eu-north-1 |
US East (N.Virginia) | us-east-1 |
US East (Ohio) | us-east-2 |
US West (N. California) | us-west-1 |
US West (Oregon) | us-west-2 |
Specify Stack Details
See the instructions below for details on configuring your CloudFormation stack and the Kinetica cluster.
- Stack name: A unique name for this CloudFormation stack
Kinetica Cluster Parameters
- Kinetica Cluster Name: A unique name for your Kinetica cluster
- Kinetica Cluster Size: This setting will determine the performance of your database. Each increase in size roughly corresponds to a doubling in performance. For Kinetica PAYG, license fees are based on the size of the cluster and your hourly usage. See Hardware Configuration below for sizing comparisons and pricing details.
- GPU Acceleration: GPU Acceleration will use GPUs to improve the performance of your database. This option is recommended if you have use cases that would benefit from GPU performance, such as interactive visualization of large scale geographic data or high-cardinality joins. See Hardware Configuration below for GPU vs. non-GPU comparisons.
- Kinetica License Key (Kinetica BYOL only): Your Kinetica license key. Contact Kinetica (sales@kinetica.com) if you need a key.
Hardware Configuration
The tables below show the EC2 instances and EBS Storage used for the different sizes and configurations, as well as the availability of the different sizes in the PAYG and BYOL offers.
Note
- In addition to the database infrastructure shown in the charts below, additional AWS resources will be provisioned for other required components, including the Kinetica Workbench UI
- The total data capacity is unlimited through the use of AWS S3 blob storage. The Recommended Hot Data Size is a rough estimation of the amount of data at a time that can be efficiently used in queries without decreasing performance. However, this will also greatly depend on the size & complexity of the queries issued.
Standard (GPU Acceleration Off)
Size | EC2 Instances | EBS Storage | PAYG | BYOL | vCPUs | GPUs | RAM (GB) | Recommended Hot Data Size |
---|---|---|---|---|---|---|---|---|
XS | 1 r5.4xlarge | 2 TB | $1.50/hour | Yes | 16 | 0 | 128 | 250 GB |
S | 2 r5.4xlarge | 4 TB | $3.00/hour | Yes | 32 | 0 | 256 | 500 GB |
M | 4 r5.4xlarge | 8 TB | $6.00/hour | Yes | 64 | 0 | 512 | 1 TB |
L | 8 r5.4xlarge | 16 TB | N/A | Yes | 128 | 0 | 1,024 | 2 TB |
XL | 16 r5.4xlarge | 32 TB | N/A | Yes | 256 | 0 | 2,048 | 4 TB |
2XL | 32 r5.4xlarge | 64 TB | N/A | Yes | 512 | 0 | 4,096 | 8 TB |
3XL | 64 r5.4xlarge | 128 TB | N/A | Yes | 1,024 | 0 | 8,192 | >16 TB |
GPU (GPU Acceleration On)
Size | EC2 Instances | EBS Storage | PAYG | BYOL | vCPUs | GPUs | RAM (GB) | Recommended Hot Data Size |
---|---|---|---|---|---|---|---|---|
XS | 1 g3.8xlarge | 4 TB | $1.80/hour | Yes | 32 | 2 | 244 | 250 GB |
S | 2 g3.8xlarge | 8 TB | $3.60/hour | Yes | 64 | 4 | 488 | 500 GB |
M | 4 g3.8xlarge | 16 TB | $7.20/hour | Yes | 128 | 8 | 976 | 1 TB |
L | 8 g3.8xlarge | 32 TB | N/A | Yes | 256 | 16 | 1,952 | 2 TB |
XL | 16 g3.8xlarge | 64 TB | N/A | Yes | 512 | 32 | 3,904 | 4 TB |
2XL | 32 g3.8xlarge | 128 TB | N/A | Yes | 1,024 | 64 | 7,808 | 8 TB |
3XL | 64 g3.8xlarge | 256 TB | N/A | Yes | 2,048 | 128 | 15,616 | >16 TB |
Kinetica User Information
- Administrator Username: The administrator username that will be used for this cluster
- Administrator Password: The administrator password that will be used to authenticate to this cluster
- Contact First Name: First name of the primary contact for the Kinetica cluster (used by Kinetica support staff only)
- Contact Last Name: Last name of the primary contact for the Kinetica cluster (used by Kinetica support staff only)
- Contact Email Address: Email address of the primary contact for the Kinetica cluster (used by Kinetica support staff only)
Kinetica Networking
- VPC CIDR: This CloudFormation stack will create a new VPC. If you intend to peer this VPC with an existing VPC, choose a CIDR range that does not overlap with the addresses in the existing VPC.
Once you have specified the stack details, click Next.
Configure Stack Options
On this page, you can configure options for this CloudFormation stack, such as Tags, Permissions, and behavior on provisioning failure. For more details on these options, see the AWS documentation:
After making any necessary changes, click Next to continue.
Review
On this page, you can review the settings for the stack before it is created.
You will need to acknowledge the additional required capabilities before you can proceed with stack creation:
Click Create stack to begin provisioning. This should take about 1 hour to complete. You can monitor the progress of the stack (and nested stacks) in the AWS Console, and the process is complete once all of the stacks are marked as CREATE_COMPLETE.
Note
The Kinetica cluster will start running as soon as the provisioning process completes and you will start to incur costs from the associated AWS infrastructure.
Get the Workbench URL
After your deployment is complete, click on the stack you created (not the nested stacks) and then click on Outputs:
See the Key workbenchUrl for the URL of the Workbench application in your Kinetica cluster; e.g., https://fmnqfqx.cloud.kinetica.com/workbench.
You can use that URL to login to Workbench with the credentials you entered under Specify Stack Details. For details on using Workbench, see Kinetica Workbench.
Appendix
Topics
Network & Architecture Diagrams
The following diagram shows the high-level architecture of a Kinetica cluster (small, not GPU-accelerated) into a new VPC, including the private and public subnets and the primary infrastructure components used by the cluster. | The following diagram shows additional detail on the EKS Kubernetes cluster used by Kinetica, including the Kinetica Operator and the node groups for infrastructure and compute. |
Security Information and Considerations
- Kinetica in AWS does not require root privileges for deployment or operation. You should not use the AWS root user to deploy or operate Kinetica.
- Kinetica in AWS follows the principle of least privilege and permissions are scoped to specific resources, services and accounts. You should also follow the principle of least privilege when granting access to Kinetica and associated resources in your AWS account.
- Kinetica in AWS does not deploy any public resources (such as S3 buckets with policies allowing public access).
- There are no security keys that you will need to manage or rotate. Access to the nodes and EC2 machines is enabled through AWS SSM, not through keys.
- The EBS volumes used by Kinetica for the persist tier as well as the S3 buckets used by Kinetica for cold storage and backup are encrypted using Amazon-managed keys.
Troubleshooting
Troubleshooting Deployment Issues
When the application has successfully provisioned, you should see the CREATE_COMPLETE status for the original CloudFormation stack and all of the nested stacks:
If you see error messages in the original stack or any of the nested stacks, provisioning Kinetica in AWS was not successful. Check the error messages for details on the specific issue. The most common issues for provisioning failures include:
Insufficient AWS privileges. See AWS Permissions for details on required permissions.
Insufficient AWS Service Quota in the target AWS region. See AWS Service Quota for details on required service quota.
Timeout provisioning SSL certificate. Occasionally, the request for an SSL certificate may timeout. If you see the following error message, delete the stack and provision again:
The following resource(s) failed to create: [certificateStatusCheckerCR].
If you experience any other issues, contact Kinetica Support.
Troubleshooting Issues Using Kinetica
If you experience issues using Kinetica to ingest, query, or egress data, check the logs in AWS CloudFormation for details. See Logging in AWS for detailed instructions.
If you have questions about your specific issue, contact Kinetica Support.
Support
For details on Kinetica support levels and contact information, see Kinetica Support.
Deployed Resources
Here is a list of resources deployed into a CloudFormation stack in your AWS account when you provision Kinetica in AWS:
Resource | Number Deployed |
---|---|
AWS::CDK::Metadata | 5 |
AWS::CloudFormation::CustomResource | 6 |
AWS::CloudFormation::Stack | 4 |
AWS::EC2::EIP | 2 |
AWS::EC2::InternetGateway | 1 |
AWS::EC2::NatGateway | 2 |
AWS::EC2::Route | 4 |
AWS::EC2::RouteTable | 4 |
AWS::EC2::SecurityGroup | 1 |
AWS::EC2::Subnet | 4 |
AWS::EC2::SubnetRouteTableAssociation | 4 |
AWS::EC2::VPC | 1 |
AWS::EC2::VPCGatewayAttachment | 1 |
AWS::EKS::Nodegroup | 4 |
AWS::IAM::Policy | 13 |
AWS::IAM::Role | 20 |
AWS::Lambda::Function | 13 |
AWS::Lambda::LayerVersion | 1 |
AWS::S3::Bucket | 3 |
AWS::S3::BucketPolicy | 1 |
AWS::SSM::Parameter | 1 |
AWS::StepFunctions::StateMachine | 1 |
Custom::AWSCDK-EKS-Cluster | 1 |
Custom::AWSCDK-EKS-HelmChart | 6 |
Custom::AWSCDK-EKS-KubernetesResource | 9 |
Custom::AWSCDKCfnJson | 1 |
Custom::AWSCDKOpenIdConnectProvider | 1 |
Custom::S3AutoDeleteObjects | 1 |
Operating Costs
The total cost to operate Kinetica is based on the Kinetica license fee and the cost of the AWS infrastructure.
Kinetica License Fee
If you are using the Kinetica Pay As You Go offer (PAYG), the Kinetica license fee is based on the size of the Kinetica cluster and the hours of usage. If you suspend Kinetica (manually or through auto suspend), the cluster is suspended and you will not incur Kinetica license fees until the cluster is resumed. For details on the hourly license fees for different cluster sizes, see Hardware Configuration.
If you are using the Kinetica Bring Your Own License offer (BYOL), you will not be charged in AWS for a Kinetica license fee. Your Kinetica license fee has already been paid for outside the AWS Marketplace.
AWS Infrastructure Costs
Your AWS infrastructure costs will depend on the size of the Kinetica cluster, any discounted pricing you may receive (e.g., credits for EC2 Reserved Instances), and your overall usage of the cluster.
The primary cost components will be the EC2 instance(s) and EBS volume(s) used in the EKS cluster to power the Kinetica database and the EC2 instances used for additional cluster infrastructure services (e.g., Kinetica Workbench). When the Kinetica cluster is suspended, the EC2 instance(s) and EBS volume(s) used by the Kinetica database will be spun down and will not incur costs. However, the infrastructure services will continue to run and incur costs.
Finally, your usage of the cluster will also affect the overall cost, although the incremental costs should be minor. For instance, ingesting large volumes of data may increase your network charges while increased usage may increase the log volume stored in CloudWatch.
Below are the estimated costs to operate Kinetica clusters in AWS, assuming on-demand EC2 instance pricing. With EC2 Reserved Instances (especially for the compute nodes), you will be able to realize significant cost savings.
Type | Size | EC2 Instances | EBS Storage | Estimated Infrastructure Cost [running] ($/hour) | Estimated Infrastructure Cost [suspended] ($/hour) |
---|---|---|---|---|---|
CPU | XS | 1 r5.4xlarge (16 vCPU, 128 GB) | 2 TB | $1.68 | $0.67 |
CPU | S | 2 r5.4xlarge (16 vCPU, 128 GB) | 4 TB | $3.04 | $0.99 |
CPU | M | 4 r5.4xlarge (16 vCPU, 128 GB) | 8 TB | $5.69 | $1.60 |
GPU | XS | 1 g3.8xlarge (2 M60, 32 vCPU, 244 GB) | 4 TB | $2.95 | $0.67 |
GPU | S | 2 g3.8xlarge (2 M60, 32 vCPU, 244 GB) | 8 TB | $5.58 | $0.99 |
GPU | M | 4 g3.8xlarge (2 M60, 32 vCPU, 244 GB) | 16 TB | $10.72 | $1.60 |