Provisioning Kinetica in AWS

Before You Start

Note that provisioning Kinetica through the AWS Marketplace will incur charges in your AWS account. These charges will depend on the type and size of the cluster you provision as well as the hours of usage. Once the provisioning is complete, the database will start running.

Pre-requisites

AWS Account

An AWS account is required to provision Marketplace applications, including Kinetica.

Note

The EC2 compute instances required by Kinetica are not covered by the AWS free usage tier.

AWS Service Quota

Make sure you have an adequate quota in your AWS subscription and AWS region for the following resources. The number of required instances will depend on the size and configuration of your Kinetica cluster. For details on the capabilities and specifications of the different cluster sizes, see Hardware Configuration.

Quota Type Service Code # Required Default Quota
VPCs per Region vpc 2 5
Subnets per VPC vpc 7 100
Route Tables per VPC vpc 6 200
Internet Gateways per Region vpc 2 5
Network Load Balancers per Region elasticloadbalancing 1 50
EC2-VPC Elastic IPs ec2 2 5
NAT Gateways per Availability Zone vpc 2 5
Network ACLs per VPC vpc 2 200
VPC Security Groups per Region vpc 5 2500
Clusters eks 1 100

EC2 Quotas for CPU-Based Clusters

Quota Type Cluster Size # Required Default Quota

Running On-Demand Standard Instances

(A, C, D, H, I, M, R, T, Z)

XS 20 5
S 36
M 72

EC2 Quotas for GPU-Based Clusters

Quota Type Cluster Size # Required Default Quota

Running On-Demand Standard Instances

(A, C, D, H, I, M, R, T, Z)

XS 4 5
S 4
M 8
Running On-Demand G & VT Instances XS 32 0
S 64
M 128

AWS Permissions

The AWS user who provisions Kinetica from the AWS Marketplace must have the necessary permissions to buy and use AWS Marketplace applications and provision the required AWS resources.

For more information on IAM permissions to use the AWS Marketplace, see the AWS documentation:

The Kinetica Cloud Formation Template will create a VPC, EKS cluster, IAM roles, Lambda functions, and Step Functions during provisioning and users will need to have the necessary permissions to create those resources.

Note

It is recommended that the AWS user has Administrator Access to be able to provision Kinetica.

Provisioning

Choose an Offer

There are two different offers in the AWS Marketplace for Kinetica, depending on your preferred consumption and payment model:

Larger cluster sizes and configurations are only available in the Bring Your Own License (BYOL) offer. All other features and functions are the same between the two offers.

Kinetica Pay As You Go (PAYG)

With this offer, Kinetica license fees are based on your hourly usage and the size of the cluster. See Specify Stack Details for specifics on cluster sizes, hardware configurations, and hourly license fees.

To find the Kinetica Pay As You Go (PAYG) Offer, you can search for Kinetica under Discover Products in the AWS Marketplace:

Kinetica Bring Your Own License (BYOL)

For customers with an existing Kinetica license, you can choose this option and input your existing Kinetica license key. You will not incur additional Kinetica license charges, although you will be charged for AWS infrastructure costs. See Hardware Configuration for more details on the AWS infrastructure provisioned with each cluster configuration.

To find the Kinetica Bring Your Own License (BYOL) Offer, you can search for Kinetica (Bring Your Own License) under Discover Products in the AWS Marketplace:

Subscribe to the Product

If you are already familiar with AWS Marketplace and CloudFormation Stacks, you can skip to Specify Stack Details for information on configuring your Kinetica cluster.

The first step in the AWS Marketplace is to subscribe to the product. Click on Continue to Subscribe to purchase an entitlement for Kinetica.

Note

You will not incur any charges for purchasing an entitlement for Kinetica.

../img/prov.step_overview.png

Review Terms and Conditions

Review the Terms and Conditions and click Accept Terms.

../img/prov.step_terms.png

Once the request has completed, you will be able to configure Kinetica. Click Continue to Configuration.

../img/prov.step_subscribed.png

Configure the Software

Kinetica has a single fulfillment option (On EKS with New VPC) which will use Cloud Formation to deploy Kinetica into an Elastic Kubernetes Service (EKS) cluster in a new VPC. This deployment option will deploy into your selected region in a single Availability Zone.

Choose the version of the product you would like to deploy and click Continue to Launch.

../img/prov.step_configure.png

Launch the Software

Under Configuration details, you can review the Usage instructions and find links to this documentation.

Click on the link under Deployment template; e.g., Kinetica on EKS with New VPC.

../img/prov.step_launch.png

Create Stack

A new browser tab will open where you can create a CloudFormation stack based on the Kinetica template. The fields are pre-populated with the S3 URL for the Kinetica template. You can download and inspect the Cloud Formation template or view it in the Designer.

Make sure you have selected the correct AWS region (in the top right corner) for your deployment and click Next to Specify stack details. The following regions are currently supported:

Region Code
Asia Pacific (Hong Kong) ap-east-1
Asia Pacific (Mumbai) ap-south-1
Asia Pacific (Seoul) ap-northeast-2
Asia Pacific (Singapore) ap-southeast-1
Asia Pacific (Sydney) ap-southeast-2
Asia Pacific (Tokyo) ap-northeast-1
Canada (Central) ca-central-1
Europe (Frankfurt) eu-central-1
Europe (Ireland) eu-west-1
Europe (London) eu-west-2
Europe (Paris) eu-west-3
Europe (Stockholm) eu-north-1
US East (N.Virginia) us-east-1
US East (Ohio) us-east-2
US West (N. California) us-west-1
US West (Oregon) us-west-2
../img/prov.step_stack_create.png

Specify Stack Details

See the instructions below for details on configuring your CloudFormation stack and the Kinetica cluster.

../img/prov.step_stack_details.png
  • Stack name: A unique name for this CloudFormation stack

Kinetica Cluster Parameters

  • Kinetica Cluster Name: A unique name for your Kinetica cluster
  • Kinetica Cluster Size: This setting will determine the performance of your database. Each increase in size roughly corresponds to a doubling in performance. For Kinetica PAYG, license fees are based on the size of the cluster and your hourly usage. See Hardware Configuration below for sizing comparisons and pricing details.
  • GPU Acceleration: GPU Acceleration will use GPUs to improve the performance of your database. This option is recommended if you have use cases that would benefit from GPU performance, such as interactive visualization of large scale geographic data or high-cardinality joins. See Hardware Configuration below for GPU vs. non-GPU comparisons.
  • Kinetica License Key (Kinetica BYOL only): Your Kinetica license key. Contact Kinetica (sales@kinetica.com) if you need a key.
Hardware Configuration

The tables below show the EC2 instances and EBS Storage used for the different sizes and configurations, as well as the availability of the different sizes in the PAYG and BYOL offers.

Note

  • In addition to the database infrastructure shown in the charts below, additional AWS resources will be provisioned for other required components, including the Kinetica Workbench UI
  • The total data capacity is unlimited through the use of AWS S3 blob storage. The Recommended Hot Data Size is a rough estimation of the amount of data at a time that can be efficiently used in queries without decreasing performance. However, this will also greatly depend on the size & complexity of the queries issued.
Standard (GPU Acceleration Off)
Size EC2 Instances EBS Storage PAYG BYOL vCPUs GPUs RAM (GB) Recommended Hot Data Size
XS 1 r5.4xlarge 2 TB $1.50/hour Yes 16 0 128 250 GB
S 2 r5.4xlarge 4 TB $3.00/hour Yes 32 0 256 500 GB
M 4 r5.4xlarge 8 TB $6.00/hour Yes 64 0 512 1 TB
L 8 r5.4xlarge 16 TB N/A Yes 128 0 1,024 2 TB
XL 16 r5.4xlarge 32 TB N/A Yes 256 0 2,048 4 TB
2XL 32 r5.4xlarge 64 TB N/A Yes 512 0 4,096 8 TB
3XL 64 r5.4xlarge 128 TB N/A Yes 1,024 0 8,192 >16 TB
GPU (GPU Acceleration On)
Size EC2 Instances EBS Storage PAYG BYOL vCPUs GPUs RAM (GB) Recommended Hot Data Size
XS 1 g3.8xlarge 4 TB $1.80/hour Yes 32 2 244 250 GB
S 2 g3.8xlarge 8 TB $3.60/hour Yes 64 4 488 500 GB
M 4 g3.8xlarge 16 TB $7.20/hour Yes 128 8 976 1 TB
L 8 g3.8xlarge 32 TB N/A Yes 256 16 1,952 2 TB
XL 16 g3.8xlarge 64 TB N/A Yes 512 32 3,904 4 TB
2XL 32 g3.8xlarge 128 TB N/A Yes 1,024 64 7,808 8 TB
3XL 64 g3.8xlarge 256 TB N/A Yes 2,048 128 15,616 >16 TB

Kinetica User Information

  • Administrator Username: The administrator username that will be used for this cluster
  • Administrator Password: The administrator password that will be used to authenticate to this cluster
  • Contact First Name: First name of the primary contact for the Kinetica cluster (used by Kinetica support staff only)
  • Contact Last Name: Last name of the primary contact for the Kinetica cluster (used by Kinetica support staff only)
  • Contact Email Address: Email address of the primary contact for the Kinetica cluster (used by Kinetica support staff only)

Kinetica Networking

  • VPC CIDR: This CloudFormation stack will create a new VPC. If you intend to peer this VPC with an existing VPC, choose a CIDR range that does not overlap with the addresses in the existing VPC.

Once you have specified the stack details, click Next.

Configure Stack Options

On this page, you can configure options for this CloudFormation stack, such as Tags, Permissions, and behavior on provisioning failure. For more details on these options, see the AWS documentation:

After making any necessary changes, click Next to continue.

../img/prov.step_stack_options.png

Review

On this page, you can review the settings for the stack before it is created.

../img/prov.step_review.png

You will need to acknowledge the additional required capabilities before you can proceed with stack creation:

../img/prov.step_acknowledge.png

Click Create stack to begin provisioning. This should take about 1 hour to complete. You can monitor the progress of the stack (and nested stacks) in the AWS Console, and the process is complete once all of the stacks are marked as CREATE_COMPLETE.

../img/prov.step_provision.png

Note

The Kinetica cluster will start running as soon as the provisioning process completes and you will start to incur costs from the associated AWS infrastructure.

Get the Workbench URL

After your deployment is complete, click on the stack you created (not the nested stacks) and then click on Outputs:

../img/prov.outputs.png

See the Key workbenchUrl for the URL of the Workbench application in your Kinetica cluster; e.g., https://fmnqfqx.cloud.kinetica.com/workbench.

You can use that URL to login to Workbench with the credentials you entered under Specify Stack Details. For details on using Workbench, see Kinetica Workbench.

Appendix



Network & Architecture Diagrams

The following diagram shows the high-level architecture of a Kinetica cluster (small, not GPU-accelerated) into a new VPC, including the private and public subnets and the primary infrastructure components used by the cluster. The following diagram shows additional detail on the EKS Kubernetes cluster used by Kinetica, including the Kinetica Operator and the node groups for infrastructure and compute.
../img/architecture_hl.png

Click to enlarge.

../img/architecture.png

Click to enlarge.


Security Information and Considerations

  • Kinetica in AWS does not require root privileges for deployment or operation. You should not use the AWS root user to deploy or operate Kinetica.
  • Kinetica in AWS follows the principle of least privilege and permissions are scoped to specific resources, services and accounts. You should also follow the principle of least privilege when granting access to Kinetica and associated resources in your AWS account.
  • Kinetica in AWS does not deploy any public resources (such as S3 buckets with policies allowing public access).
  • There are no security keys that you will need to manage or rotate. Access to the nodes and EC2 machines is enabled through AWS SSM, not through keys.
  • The EBS volumes used by Kinetica for the persist tier as well as the S3 buckets used by Kinetica for cold storage and backup are encrypted using Amazon-managed keys.

Troubleshooting

Troubleshooting Deployment Issues

When the application has successfully provisioned, you should see the CREATE_COMPLETE status for the original CloudFormation stack and all of the nested stacks:

../img/prov.step_provision.png

If you see error messages in the original stack or any of the nested stacks, provisioning Kinetica in AWS was not successful. Check the error messages for details on the specific issue. The most common issues for provisioning failures include:

  • Insufficient AWS privileges. See AWS Permissions for details on required permissions.

  • Insufficient AWS Service Quota in the target AWS region. See AWS Service Quota for details on required service quota.

  • Timeout provisioning SSL certificate. Occasionally, the request for an SSL certificate may timeout. If you see the following error message, delete the stack and provision again:

    The following resource(s) failed to create: [certificateStatusCheckerCR].
    

If you experience any other issues, contact Kinetica Support.

Troubleshooting Issues Using Kinetica

If you experience issues using Kinetica to ingest, query, or egress data, check the logs in AWS CloudFormation for details. See Logging in AWS for detailed instructions.

If you have questions about your specific issue, contact Kinetica Support.

Support

For details on Kinetica support levels and contact information, see Kinetica Support.

Deployed Resources

Here is a list of resources deployed into a CloudFormation stack in your AWS account when you provision Kinetica in AWS:

Resource Number Deployed
AWS::CDK::Metadata 5
AWS::CloudFormation::CustomResource 6
AWS::CloudFormation::Stack 4
AWS::EC2::EIP 2
AWS::EC2::InternetGateway 1
AWS::EC2::NatGateway 2
AWS::EC2::Route 4
AWS::EC2::RouteTable 4
AWS::EC2::SecurityGroup 1
AWS::EC2::Subnet 4
AWS::EC2::SubnetRouteTableAssociation 4
AWS::EC2::VPC 1
AWS::EC2::VPCGatewayAttachment 1
AWS::EKS::Nodegroup 4
AWS::IAM::Policy 13
AWS::IAM::Role 20
AWS::Lambda::Function 13
AWS::Lambda::LayerVersion 1
AWS::S3::Bucket 3
AWS::S3::BucketPolicy 1
AWS::SSM::Parameter 1
AWS::StepFunctions::StateMachine 1
Custom::AWSCDK-EKS-Cluster 1
Custom::AWSCDK-EKS-HelmChart 6
Custom::AWSCDK-EKS-KubernetesResource 9
Custom::AWSCDKCfnJson 1
Custom::AWSCDKOpenIdConnectProvider 1
Custom::S3AutoDeleteObjects 1

Operating Costs

The total cost to operate Kinetica is based on the Kinetica license fee and the cost of the AWS infrastructure.

Kinetica License Fee

If you are using the Kinetica Pay As You Go offer (PAYG), the Kinetica license fee is based on the size of the Kinetica cluster and the hours of usage. If you suspend Kinetica (manually or through auto suspend), the cluster is suspended and you will not incur Kinetica license fees until the cluster is resumed. For details on the hourly license fees for different cluster sizes, see Hardware Configuration.

If you are using the Kinetica Bring Your Own License offer (BYOL), you will not be charged in AWS for a Kinetica license fee. Your Kinetica license fee has already been paid for outside the AWS Marketplace.

AWS Infrastructure Costs

Your AWS infrastructure costs will depend on the size of the Kinetica cluster, any discounted pricing you may receive (e.g., credits for EC2 Reserved Instances), and your overall usage of the cluster.

The primary cost components will be the EC2 instance(s) and EBS volume(s) used in the EKS cluster to power the Kinetica database and the EC2 instances used for additional cluster infrastructure services (e.g., Kinetica Workbench). When the Kinetica cluster is suspended, the EC2 instance(s) and EBS volume(s) used by the Kinetica database will be spun down and will not incur costs. However, the infrastructure services will continue to run and incur costs.

Finally, your usage of the cluster will also affect the overall cost, although the incremental costs should be minor. For instance, ingesting large volumes of data may increase your network charges while increased usage may increase the log volume stored in CloudWatch.

Below are the estimated costs to operate Kinetica clusters in AWS, assuming on-demand EC2 instance pricing. With EC2 Reserved Instances (especially for the compute nodes), you will be able to realize significant cost savings.

Type Size EC2 Instances EBS Storage Estimated Infrastructure Cost [running] ($/hour) Estimated Infrastructure Cost [suspended] ($/hour)
CPU XS 1 r5.4xlarge (16 vCPU, 128 GB) 2 TB $1.68 $0.67
CPU S 2 r5.4xlarge (16 vCPU, 128 GB) 4 TB $3.04 $0.99
CPU M 4 r5.4xlarge (16 vCPU, 128 GB) 8 TB $5.69 $1.60
GPU XS 1 g3.8xlarge (2 M60, 32 vCPU, 244 GB) 4 TB $2.95 $0.67
GPU S 2 g3.8xlarge (2 M60, 32 vCPU, 244 GB) 8 TB $5.58 $0.99
GPU M 4 g3.8xlarge (2 M60, 32 vCPU, 244 GB) 16 TB $10.72 $1.60