Skip to content

tapdata/tapdata-deploy

Repository files navigation

TapData One-Deploy

English | 中文

TapData One-Deploy is the official all-scenario automated deployment tool for TapData, covering the full delivery pipeline from single-node development to enterprise-grade cloud-native production.

Build Once, Run Anywhere

Mode Scenario Tech Stack Status
Lite Dev / POC Docker Compose
Cloud Huawei Cloud CCE Terraform + Helm
Cloud AWS EKS Terraform + Helm ⌛️
On-Prem Customer DC / Air-gapped Offline Image Bundle + Helm ⌛️

Quick Start

# Download
curl -L https://resource.tapdata.net/deploy/tap-deploy -o tap-deploy && chmod +x tap-deploy

# Init → Plan → Apply
./tap-deploy init      # Interactive configuration wizard
./tap-deploy plan      # Preview deployment
./tap-deploy apply     # Execute deployment

After deploying in Lite mode, open http://localhost:13030 in your browser to access the console (default account admin@admin.com, password admin).

Command reference:

Command Description
./tap-deploy init Interactive wizard, generates all config files
./tap-deploy plan Preview deployment (no execution)
./tap-deploy apply Execute deployment
./tap-deploy status View deployment status
./tap-deploy destroy Uninstall and clean up
./tap-deploy bundle Create offline image bundle (On-Prem)
./tap-deploy -d /path init Specify working directory

Workflow

tap-deploy uses a template rendering mechanism for configuration management:

  1. init phase: Collect parameters interactively → Save to .tap-deploy.env → Render templates into final config files
  2. apply phase: Read configuration → Execute deployment (Docker Compose / Terraform + Helm / Helm)
init interactive wizard
  ├─ Parameters → .tap-deploy.env (config store)
  ├─ Templates + .tap-deploy.env → docker-compose.yaml / values.yaml / application.yml / terraform.tfvars
  └─ Terraform init (Cloud mode)

apply deployment execution
  ├─ Lite:    docker compose up -d
  ├─ Cloud:   terraform apply → push images → helm install
  └─ On-Prem: helm install (optional offline image loading)

Configuration Management

Config File Reference

File Generated By Description
.tap-deploy.env init Core config store, source of all parameters
application.yml init TapData application config (MongoDB connection, Java options, etc.)
agent.yml init Agent identity config
docker-compose.yaml init (Lite) Docker Compose orchestration file
values.yaml init (Cloud/On-Prem) Helm deployment parameters
terraform.tfvars init (Cloud) Terraform infrastructure variables

Customizing Configuration

Option 1: Re-run the init wizard

The simplest way to change any parameter. The wizard regenerates all config files.

Option 2: Edit .tap-deploy.env directly

After manually editing, re-render templates or edit the generated files directly. Suitable for batch changes or automation.

Option 3: Edit generated files directly

Files like docker-compose.yaml, values.yaml, and application.yml are standard-format files that can be edited for deep customization. Use this for advanced parameters not covered by the init wizard (e.g., resource quotas, Ingress annotations, MongoDB storage engine settings).

Note: Re-running init overwrites generated config files and manual edits will be lost. Consider recording customizations in .tap-deploy.env or managing them through separate Helm values override files.


Lite Mode

Single-node deployment with 5-minute quick start, ideal for development, testing, and POC.

Prerequisites: Docker 20.10+ with Docker Compose plugin, current user has Docker execution permissions, public internet access.

Init Wizard Parameters

Parameter Env Variable Default Description
TapData version TAP_DEPLOY_IMAGE_TAG latest Image tag to pull from Docker Hub
MongoDB port MONGO_HOST_PORT 27006 Host port mapping for MongoDB
TapData port TAPDATA_HOST_PORT 13030 Host port mapping for TapData
MongoDB username TAP_DEPLOY_MONGO_USER root -
MongoDB password TAP_DEPLOY_MONGO_PASSWORD AbcDef123 -

Hidden Config Items

The following are defined in .tap-deploy.env but not prompted by the init wizard. Edit manually to adjust:

Env Variable Default Description
TAP_DEPLOY_TZ Asia/Shanghai Container timezone
TAP_DEPLOY_JAVA_VERSION java17 JDK version (java8 / java11 / java17)
TAP_DEPLOY_LICENSE_HOST - License server address

Deep Customization

Edit docker-compose.yaml directly to adjust:

  • Container resource limits (default: 4 CPU / 12G memory)
  • Volume mount paths
  • Health check policies
  • Environment variables (FRONTEND_WORKER_COUNT, API_WORKER_COUNT, etc.)

Cloud Mode

Automated cloud infrastructure provisioning + Kubernetes deployment, designed for production.

Prerequisites: Terraform 1.7+, Helm 3.14+, kubectl, Docker, cloud provider AK/SK permissions.

Deployment Flow

terraform apply → Create VPC / CCE(EKS) / SWR(ECR) / NAT
       ↓
Push images to cloud registry (SWR / ECR)
       ↓
Install MongoDB Community Operator
       ↓
helm install → Deploy TapData (MongoDB + server + engine + apiserver)

Init Wizard Parameters

Common Parameters

Parameter Env Variable Default Description
TapData version TAP_DEPLOY_IMAGE_TAG latest Image tag
Cloud provider TAP_DEPLOY_CLOUD_PROVIDER - huaweicloud / aws
Access Key TAP_DEPLOY_CLOUD_ACCESS_KEY - Masked input
Secret Key TAP_DEPLOY_CLOUD_SECRET_KEY - Masked input
Region TAP_DEPLOY_CLOUD_REGION ap-southeast-1 -
Availability zones TAP_DEPLOY_CLOUD_AZS ap-southeast-1a,ap-southeast-1b Comma-separated
Cluster name TAP_DEPLOY_CLOUD_CLUSTER_NAME tapdata K8s cluster name
Node count TAP_DEPLOY_CLOUD_NODE_COUNT 3 Worker node count
MongoDB username TAP_DEPLOY_MONGO_USER root -
MongoDB password TAP_DEPLOY_MONGO_PASSWORD AbcDef123 -
Helm Release TAP_DEPLOY_HELM_RELEASE tapdata -
Namespace TAP_DEPLOY_NAMESPACE tapdata -

Huawei Cloud Specific

Parameter Env Variable Default Description
Node flavor TAP_DEPLOY_CLOUD_NODE_FLAVOR s2.4xlarge.2 Flavor reference

AWS Specific

Parameter Env Variable Default Description
Node instance type TAP_DEPLOY_CLOUD_NODE_INSTANCE_TYPE t3.medium -
K8s version TAP_DEPLOY_CLOUD_CLUSTER_VERSION 1.29 -

Automatic Registry Configuration

In Cloud mode, registry information is automatically obtained after terraform applyno manual configuration needed:

  • Huawei Cloud: SWR internal pull URL + external push URL + temporary AK/SK authentication
  • AWS: ECR internal pull URL + external push URL + ECR authentication

Automatically written to .tap-deploy.env and rendered into values.yaml during the apply phase.

Ingress Configuration

Each cloud provider uses a dedicated Ingress config file, automatically merged during apply:

Cloud Provider File Load Balancer
Huawei Cloud values-ingress-huaweicloud.yaml ELB (auto-created public ELB)
AWS values-ingress-aws.yaml ALB (Internet-facing)

To customize Ingress (domain, TLS certificate, bandwidth, etc.), edit the corresponding values-ingress-*.yaml file.

Deep Customization

Edit values.yaml directly to adjust:

  • Replica counts (tapdata-server.replicaCount, etc., default: 2 each)
  • Resource quotas (CPU/memory limits & requests)
  • MongoDB replica set members, storage class, storage size
  • Service type (ClusterIP / NodePort)
  • API Server enable/disable

Edit terraform.tfvars to adjust:

  • VPC CIDR
  • Key pair name
  • Registry organization/repository name

Get Access Address

# Huawei Cloud
kubectl describe ingress -n tapdata -l app.kubernetes.io/name=tapdata | grep 'kubernetes.io/elb.ip'

# AWS
kubectl get ingress -n tapdata

(⌛️) On-Prem Mode

For customer data centers and air-gapped environments, deployed on existing Kubernetes clusters.

Prerequisites: Helm 3.14+, kubectl, access to a K8s cluster. Offline mode requires Docker and a pre-built image bundle.

Online Deployment

When the K8s cluster can access the public internet (or a private registry):

./tap-deploy init    # Select On-Prem → Non-offline
./tap-deploy apply

Offline Deployment

Build image bundle on an online machine:

./tap-deploy bundle --output images-bundle.tar.gz

Deploy in the offline environment:

# Load images
docker load -i images-bundle.tar.gz

# Push to internal registry
docker tag tapdata8/tapdata:latest <harbor>/tapdata:latest
docker push <harbor>/tapdata:latest

# Deploy
./tap-deploy init    # Select On-Prem → Offline, enter internal registry address
./tap-deploy apply

Init Wizard Parameters

Parameter Env Variable Default Description
TapData version TAP_DEPLOY_IMAGE_TAG latest Image tag
Offline mode TAP_DEPLOY_OFFLINE false Set to true for air-gapped
MongoDB username TAP_DEPLOY_MONGO_USER root -
MongoDB password TAP_DEPLOY_MONGO_PASSWORD AbcDef123 -
Helm Release TAP_DEPLOY_HELM_RELEASE tapdata -
Namespace TAP_DEPLOY_NAMESPACE tapdata -
Internal registry TAP_DEPLOY_HARBOR_REGISTRY - Required for offline mode
Registry prefix TAP_DEPLOY_IMAGE_REGISTRY - For online mode; leave empty for Docker Hub

Deep Customization

Same as Cloud mode — edit values.yaml directly to adjust replica counts, resource quotas, MongoDB parameters, etc.


Production Guidelines

Resource Planning

Component Replicas CPU Memory
tapdata-server 2 500m ~ 2 2Gi ~ 4Gi
tapdata-engine 2 500m ~ 2 2Gi ~ 4Gi
tapdata-apiserver 2 250m ~ 1 1Gi ~ 2Gi
MongoDB 3-node replica set - -

Storage

  • Huawei Cloud: Ultra-High I/O EVS (SSD)
  • AWS: gp3 / io2 EBS
  • Self-hosted: SSD recommended to reduce CDC latency

High Availability

  • MongoDB 3-node replica set with automatic failover
  • Multi-replica application + load balancing
  • Zero-downtime rolling updates, 30s graceful shutdown

Troubleshooting

Symptom Resolution
Pod fails to start kubectl -n tapdata describe pod <pod-name>
Service unreachable Check Ingress / ELB / security groups
MongoDB connection failure kubectl get mdbc -n tapdata
License expired Check $TAPDATA_WORK_DIR/license.txt

Tech Stack

Docker 20.10+ · Kubernetes 1.24+ · Helm 3.14+ · Terraform 1.7+ · MongoDB 6.0+ · AMD64 / ARM64


Important Notes

  • init requires downloading config templates; apply requires pulling images. Prepare offline bundles in advance for air-gapped environments
  • Use ultra-high I/O storage in production to reduce CDC latency
  • All Connectors are ARM-compiled and verified
  • macOS: ensure terminal has disk access permissions
  • 30-day free trial License included by default; contact TapData team for extensions

GitHub · Issues

About

TapData One-Deploy is an official TapData full-scenario automated deployment script. Leveraging mature IaC and container tools, it enables one-click delivery from single-machine testing to enterprise-grade cloud-native environments.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors