Airbyte Deployment

Overview

This example shows how to deploy Airbyte on Qovery using Terraform. Airbyte is deployed from its official Helm chart, with a PostgreSQL database and a proxy that protects the Airbyte web app with basic authentication.

This page is based on qovery-airbyte, an external example repository. The proxy application of this page builds from that repository.

What You’ll Deploy

PostgreSQL Database: Amazon RDS for PostgreSQL for Airbyte metadata, created from a blueprint of the service catalog
Airbyte (Helm): Airbyte deployed from the official Helm chart
Proxy Application: Reverse proxy with basic authentication in front of the Airbyte web app
Deployment Stages: Ordered deployment (Database → App → Proxy)

Prerequisites

Before deploying, gather these values:

A Qovery API token and the IDs of your organization, project and cluster. See Find resource IDs.
A cluster on AWS: the Amazon RDS for PostgreSQL blueprint is available on AWS clusters only.
Basic authentication credentials for the proxy: one htpasswd entry with a SHA-1 password, username:{SHA}.... The Gateway API expects this format, and NGINX accepts it too, so the credentials work whichever of the two routes the traffic.

File Structure

qovery-airbyte/
├── main.tf
├── variables.tf
└── airbyte-values.yaml

variables.tf

variable "qovery_access_token" {
  description = "Qovery API token"
  type        = string
  sensitive   = true
}

variable "qovery_organization_id" {
  description = "Qovery Organization ID"
  type        = string
}

variable "qovery_project_id" {
  description = "Qovery Project ID"
  type        = string
}

variable "qovery_cluster_id" {
  description = "Qovery Cluster ID"
  type        = string
}

variable "airbyte_helm_version" {
  description = "Airbyte Helm chart version"
  type        = string
  default     = "1.7.1"
}

variable "airbyte_service_name" {
  description = "Airbyte service name"
  type        = string
  default     = "Airbyte"
}

variable "qovery_airbyte_web_app_proxy_basic_auth" {
  description = "Basic Auth for Airbyte web app proxy: one htpasswd entry, username:{SHA}..."
  type        = string
  sensitive   = true
}

main.tf

terraform {
  required_providers {
    qovery = {
      source  = "qovery/qovery"
      version = "~> 1.0"
    }
  }
}

provider "qovery" {
  token = var.qovery_access_token
}

# Create environment for Airbyte
resource "qovery_environment" "airbyte" {
  project_id = var.qovery_project_id
  cluster_id = var.qovery_cluster_id
  name       = "airbyte"
  mode       = "PRODUCTION"
}

# Deployment stages: Airbyte, then the proxy. Qovery adds a new stage after the existing ones,
# so both come after the default stages, including the one of the database blueprint.
resource "qovery_deployment_stage" "app" {
  environment_id = qovery_environment.airbyte.id
  name           = "APP"
}

resource "qovery_deployment_stage" "proxy" {
  environment_id = qovery_environment.airbyte.id
  name           = "PROXY"
  is_after       = qovery_deployment_stage.app.id
}

# Create PostgreSQL database for Airbyte: Amazon RDS for PostgreSQL from the service catalog.
# Terraform waits for its deployment, so its outputs exist before the Helm chart is created.
resource "qovery_blueprint" "airbyte_db" {
  environment_id = qovery_environment.airbyte.id
  name           = "airbyte-db"
  blueprint      = "AWS/postgres/17"

  variables = {
    # Also names the RDS instance: it must be unique in the AWS account and region.
    db_name        = "airbyte"
    instance_class = "db.t3.small"
  }
}

locals {
  # Prefix of the output variables of the blueprint, for example QOVERY_OUTPUT_TERRAFORM_ZA1B2C3D4
  airbyte_db_outputs = "QOVERY_OUTPUT_TERRAFORM_Z${upper(split("-", qovery_blueprint.airbyte_db.service_id)[0])}"
}

# Add Airbyte Helm repository
resource "qovery_helm_repository" "airbyte" {
  organization_id       = var.qovery_organization_id
  name                  = "airbyte"
  kind                  = "HTTPS"
  url                   = "https://airbytehq.github.io/helm-charts"
  skip_tls_verification = false
}

# Deploy Airbyte using Helm
resource "qovery_helm" "airbyte" {
  environment_id      = qovery_environment.airbyte.id
  deployment_stage_id = qovery_deployment_stage.app.id
  name                = var.airbyte_service_name
  description         = "Airbyte deployed from its official Helm chart"

  source = {
    helm_repository = {
      helm_repository_id = qovery_helm_repository.airbyte.id
      chart_name         = "airbyte"
      chart_version      = var.airbyte_helm_version
    }
  }

  # Allow Airbyte to create cluster-wide resources
  allow_cluster_wide_resources = true

  # Override default values with airbyte-values.yaml, keyed by file name
  values_override = {
    file = {
      raw = {
        "airbyte-values.yaml" = {
          content = file("${path.module}/airbyte-values.yaml")
        }
      }
    }
  }

  # Database connection details, referenced as qovery.env.<NAME> in airbyte-values.yaml.
  # Qovery replaces each {{...}} with an output of the blueprint when it deploys the chart.
  environment_variables = [
    {
      key   = "DATABASE_HOST"
      value = "{{${local.airbyte_db_outputs}_DB_ADDRESS}}"
    },
    {
      key   = "DATABASE_PORT"
      value = "{{${local.airbyte_db_outputs}_DB_PORT}}"
    },
    {
      key   = "DATABASE_NAME"
      value = "{{${local.airbyte_db_outputs}_DB_NAME}}"
    },
    {
      key   = "DATABASE_USER"
      value = "{{${local.airbyte_db_outputs}_DB_USERNAME}}"
    }
  ]

  secrets = [
    {
      key   = "DATABASE_PASSWORD"
      value = "{{${local.airbyte_db_outputs}_DB_PASSWORD}}"
    }
  ]
}

# Deploy proxy application for Airbyte web app
resource "qovery_application" "airbyte_webapp_proxy" {
  environment_id      = qovery_environment.airbyte.id
  deployment_stage_id = qovery_deployment_stage.proxy.id
  name                = "airbyte-webapp-proxy"

  # Caddy reverse proxy from the external example repository
  git_repository = {
    url    = "https://github.com/evoxmusic/qovery-airbyte.git"
    branch = "main"
  }

  build_mode      = "DOCKER"
  dockerfile_path = "Dockerfile.webappproxy"

  cpu    = 100
  memory = 128

  min_running_instances = 1
  max_running_instances = 1

  ports = [{
    internal_port       = 80
    external_port       = 443
    protocol            = "HTTP"
    publicly_accessible = true
    name                = "http"
  }]

  environment_variables = [
    {
      # Qovery replaces {{...}} with the internal host name of the Airbyte Helm release.
      key   = "AIRBYTE_WEBAPP_INTERNAL_HOST"
      value = "{{QOVERY_HELM_Z${upper(split("-", qovery_helm.airbyte.id)[0])}_HOST_INTERNAL}}-airbyte-webapp-svc"
    },
    {
      key   = "AIRBYTE_WEBAPP_INTERNAL_PORT"
      value = "80"
    },
    {
      key   = "AIRBYTE_WEBAPP_BASIC_AUTH"
      value = var.qovery_airbyte_web_app_proxy_basic_auth
    }
  ]

  # Protect the public endpoint with the credentials of AIRBYTE_WEBAPP_BASIC_AUTH,
  # whether NGINX (ingress) or the Gateway API routes the traffic.
  advanced_settings_json = jsonencode({
    "network.ingress.basic_auth_env_var"     = "AIRBYTE_WEBAPP_BASIC_AUTH"
    "network.gateway_api.basic_auth_env_var" = "AIRBYTE_WEBAPP_BASIC_AUTH"
  })

  healthchecks = {
    readiness_probe = {
      type = {
        http = {
          path   = "/"
          port   = 80
          scheme = "HTTP"
        }
      }
      initial_delay_seconds = 30
      period_seconds        = 10
      timeout_seconds       = 5
      success_threshold     = 1
      failure_threshold     = 3
    }

    liveness_probe = {
      type = {
        http = {
          path   = "/"
          port   = 80
          scheme = "HTTP"
        }
      }
      initial_delay_seconds = 30
      period_seconds        = 10
      timeout_seconds       = 5
      success_threshold     = 1
      failure_threshold     = 3
    }
  }
}

# Creating the services does not deploy them: this resource deploys the environment,
# stage by stage.
resource "qovery_deployment" "airbyte" {
  environment_id = qovery_environment.airbyte.id
  desired_state  = "RUNNING"

  depends_on = [
    qovery_blueprint.airbyte_db,
    qovery_helm.airbyte,
    qovery_application.airbyte_webapp_proxy,
  ]
}

output "airbyte_url" {
  value = "https://${qovery_application.airbyte_webapp_proxy.external_host}"
}

airbyte-values.yaml

Create a airbyte-values.yaml file to customize Airbyte configuration. Qovery replaces each qovery.env.<NAME> with the value of the <NAME> variable of the Helm service when it deploys the chart, as described in Environment Variables in Values.

global:
  serviceAccountName: airbyte-admin
  deploymentMode: oss

  database:
    type: external
    host: qovery.env.DATABASE_HOST
    port: qovery.env.DATABASE_PORT
    database: qovery.env.DATABASE_NAME
    user: qovery.env.DATABASE_USER
    password: qovery.env.DATABASE_PASSWORD

# Use the Qovery database instead of the PostgreSQL of the chart
postgresql:
  enabled: false

# The managed database requires TLS connections
temporal:
  extraEnv:
    - name: POSTGRES_TLS_ENABLED
      value: "true"
    - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
      value: "true"
    - name: SQL_TLS_ENABLED
      value: "true"
    - name: SQL_TLS_DISABLE_HOST_VERIFICATION
      value: "true"

webapp:
  enabled: true
  replicaCount: 1
  service:
    type: ClusterIP
    port: 80

server:
  enabled: true
  replicaCount: 1

worker:
  enabled: true
  replicaCount: 1

airbyte-bootloader:
  enabled: true

Deployment Steps

Clone or Create Files

# Option 1: Clone the external example repository
git clone https://github.com/evoxmusic/qovery-airbyte
cd qovery-airbyte

# Option 2: Create the files manually
# Create main.tf, variables.tf, and airbyte-values.yaml

The files of the repository can differ from the ones on this page.

Set Environment Variables

export TF_VAR_qovery_access_token="your-api-token"
export TF_VAR_qovery_organization_id="your-org-id"
export TF_VAR_qovery_project_id="your-project-id"
export TF_VAR_qovery_cluster_id="your-cluster-id"

Set Basic Authentication

# Generate an htpasswd entry with a SHA-1 password (username:{SHA}...)
export TF_VAR_qovery_airbyte_web_app_proxy_basic_auth=$(printf '%s\n' 'yourpassword' | htpasswd -ni -s admin)

Initialize Terraform

terraform init

Plan and Apply

# Review changes
terraform plan

# Create the resources and deploy the environment
terraform apply

terraform apply waits for the database to be created, then for the deployment of the environment.

Access Airbyte

Open the URL of the proxy and log in with your basic authentication credentials:

terraform output airbyte_url

You can also forward a local port to the proxy. When prompted, select the airbyte-webapp-proxy service:

qovery port-forward --port 8080:80

# Open http://localhost:8080

Port forwarding connects to the proxy directly, without the basic authentication of the public endpoint.

Cleanup

terraform destroy

Key Takeaways

This example demonstrates several advanced Terraform patterns:

Multi-Service Deployment

Deploying multiple interconnected services:

Managed PostgreSQL database
Helm chart application
Proxy application for authentication

Deployment Stages

Ensuring proper deployment order with qovery_deployment_stage resources and the deployment_stage_id of each service. The Terraform service of the database blueprint stays in its default stage, which comes before the stages the configuration adds:

TERRAFORM DEFAULT: PostgreSQL deploys first
APP: Airbyte Helm chart deploys after database
PROXY: Authentication proxy deploys last

Dynamic Configuration

Using Qovery’s database connection details in Helm values:

Outputs of the database blueprint injected in variables of the Helm service with {{...}} interpolation
Variables referenced as qovery.env.<NAME> in airbyte-values.yaml
The internal host of the Helm release injected in the proxy with {{...}} interpolation

External Helm Charts

Integrating third-party Helm repositories:

Adding Airbyte’s official Helm repository
Deploying specific chart versions
Overriding values with custom configuration

Security

Basic authentication on the public endpoint of the proxy
Private database: the blueprint does not make the instance publicly accessible by default
Database password passed as a secret that Qovery builds from the blueprint output, never written in the configuration

Health Checks

Liveness and readiness probes on the proxy:

Initial delay to allow services to start
Regular health checks
Automatic restarts on failure

Resource Management

Setting CPU and memory for each service:

Minimal resources for proxy (100 mCPU, 128 MB)
Managed database with an instance class (db.t3.small) and the default storage of the blueprint
Cluster-wide resource permissions for Airbyte

Troubleshooting

Deployment Takes Too Long

Airbyte deployment typically takes several minutes:

Database must be ready first, and a managed database takes several minutes to create
Helm chart pulls multiple images
Airbyte bootloader initializes the database

If the Helm deployment times out, increase timeout_sec on qovery_helm (600 seconds by default).Check deployment status:

qovery status

Cannot Access Web UI

If you can’t access the Airbyte UI:

Verify all services are deployed
Check proxy application is running
Ensure basic auth credentials are correct
Try port-forwarding locally

Database Connection Issues

If Airbyte can’t connect to the database:

Verify database is in RUNNING state
Check the variables of the Helm service reference the outputs of the database blueprint
Review airbyte-values.yaml configuration
Check Airbyte logs for connection errors

Next Steps

Example Repository

External example repository this page is based on

Airbyte Documentation

Learn more about Airbyte

Helm Configuration

Deploy more Helm charts

Advanced Patterns

Learn advanced Terraform techniques

Getting Started

Examples

Airbyte Deployment

Overview

What You’ll Deploy

Prerequisites

File Structure

variables.tf

main.tf

airbyte-values.yaml

Deployment Steps

Cleanup

Key Takeaways

Troubleshooting

Next Steps

Example Repository

Airbyte Documentation

Helm Configuration

Advanced Patterns

​Overview

​What You’ll Deploy

​Prerequisites

​File Structure

​variables.tf

​main.tf

​airbyte-values.yaml

​Deployment Steps

​Cleanup

​Key Takeaways

​Troubleshooting

​Next Steps

Example Repository

Airbyte Documentation

Helm Configuration

Advanced Patterns

Overview

What You’ll Deploy

Prerequisites

File Structure

variables.tf

main.tf

airbyte-values.yaml

Deployment Steps

Cleanup

Key Takeaways

Troubleshooting

Next Steps