diff --git a/docs/assets/images/do-app-logs.png b/docs/assets/images/do-app-logs.png new file mode 100644 index 000000000..8751bb9b6 Binary files /dev/null and b/docs/assets/images/do-app-logs.png differ diff --git a/docs/assets/images/do-app.png b/docs/assets/images/do-app.png new file mode 100644 index 000000000..14a553835 Binary files /dev/null and b/docs/assets/images/do-app.png differ diff --git a/docs/assets/images/do-kafka-console.png b/docs/assets/images/do-kafka-console.png new file mode 100644 index 000000000..88dbfbfcb Binary files /dev/null and b/docs/assets/images/do-kafka-console.png differ diff --git a/docs/assets/images/do-kafka-event.png b/docs/assets/images/do-kafka-event.png new file mode 100644 index 000000000..f7f23f8bb Binary files /dev/null and b/docs/assets/images/do-kafka-event.png differ diff --git a/docs/guides/do-app-platform.md b/docs/guides/do-app-platform.md new file mode 100644 index 000000000..3ab9fdc8b --- /dev/null +++ b/docs/guides/do-app-platform.md @@ -0,0 +1,72 @@ +# Deploy to DO App Platform + +!!! Warning + This feature is currently available in a RC release. Install `tmctl` like so: + + ```console + curl -sSfL https://raw.githubusercontent.com/triggermesh/tmctl/HEAD/hack/install.sh | PKG_VERSION=v1.2.0-rc.1 sh + ``` + +With a Pipe that is running successfully locally you are ready to deploy it on DO App Platform. + +First, stop the local Pipe to avoid duplicating consumers. + +```console +tmctl stop +``` + +You can verif that all the components are stopped and that no containers are running + +```console +docker ps +``` + +## Generate the App Spec and Create the App + +You can generate the DO App Spec with `tmctl` and pipe it directly to `doctl` like so: + +```console +tmctl dump -p digitalocean +tmctl dump -p digitalocean | doctl apps create --spec - +``` + +!!! Tip + The `tmctl dump` command will generate Kubernetes manifests that you can use on a Kubernetes cluster that runs TriggerMesh. The `tmctl dump -p docker-compose` will generate a Docker compose file. Additional options are available to configure the generated manifests. + + ```console + $ tmctl dump --help + ``` + +## View the App on DO + +Head over to the DO App Platform console, you will see an App named with the name of your local broker. Each TriggerMesh component will be running as separate container like locally. + +![](../assets/images/do-app.png) + +You can select the _Runtime Logs_ tab and you will see the logs of each container. + +![](../assets/images/do-app-logs.png) + + +## Manage the App + +You can manage the App via the DO console like any other App, and you can manage it via the `doctl` client as well. + +```console +doctl apps list +ID Spec Name Default Ingress Active Deployment ID In Progress Deployment ID Created At Updated At +954ab2c0-26cf-4fba-8ef5-de52d8b7c99c do 93b34e5b-439e-4d9b-8ee6-193d481d3904 2023-02-14 20:22:03 +0000 UTC 2023-02-14 20:22:25 +0000 UTC + +doctl apps delete 954ab2c0-26cf-4fba-8ef5-de52d8b7c99c +❯ Are you sure you want to delete this App? yes +``` + +Once you have deleted your App in DO, you can restart your Pipe locally + +```console +tmctl start +2023/02/14 21:31:48 do | Starting broker +2023/02/14 21:31:49 do | Starting do-kafkatarget +2023/02/14 21:31:49 do | Starting do-awssqssource +2023/02/14 21:31:50 do | Starting do-cloudeventstarget +``` \ No newline at end of file diff --git a/docs/guides/do-kafka.md b/docs/guides/do-kafka.md new file mode 100644 index 000000000..215a70c5e --- /dev/null +++ b/docs/guides/do-kafka.md @@ -0,0 +1,183 @@ +# Using a DigitalOcean Kafka Cluster + +With `tmctl` you can create a Kafka Target to produce events into a Kafka topic and with a Kafka Source you can consume events from a Kafka topic. In this guide we are going to show you how to do it with a Kafka cluster on DigitalOcean. + +!!! Tip + Verify that you have configured [auto-completion](../get-started/autocompletion.md) for `tmctl` to help you configure all the source and target parameters easily. + + ```console + $ tmctl completion + ``` + +## Get the Kafka Cluster Connection Details + +Head to the DO console of your Kafka cluster and select the Overview tab as shown below. + +![](../assets/images/do-kafka-console.png) + +Note the `username`, `password`, `host` and `port` and download the CA certificate to a local file. + +## Create an Event Broker + +We start by creating an event broker + +```console +$ tmctl create brokers do +``` + +## Create a Kafka Target + +To produce messages to a Kafka topic you need to setup a Kafka target. + +```console +tmctl create target kafka \ + --bootstrapServers \ + --topic tmctl \ + --auth.username doadmin \ + --auth.password \ + --auth.saslEnable true \ + --auth.tlsEnable true \ + --auth.securityMechanism PLAIN \ + --auth.tls.ca "$(cat /path/to/ca-certificate.crt)" +``` + +You will then be able to _describe_ your Pipe and see the Docker container corresponding to your Kafka producer running. + +```console +tmctl describe +Broker Status +do online(http://localhost:54556) + +Target Kind Expected Events Status +do-kafkatarget kafkatarget * online(http://localhost:54811) + +docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +3dd0f90231c3 gcr.io/triggermesh/kafkatarget-adapter:v1.23.2 "/ko-app/kafkatarget…" 22 seconds ago Up 21 seconds 0.0.0.0:54811->8080/tcp do-kafkatarget +bc7986f80c62 gcr.io/triggermesh/memory-broker:latest "/memory-broker star…" 58 minutes ago Up 58 minutes 0.0.0.0:54556->8080/tcp do-broker + +``` + +!!! Tip + You can also use the SSL connection parameters instead of SASL. + +### Send an event to Kafka + +To send an sample event to your Kafka topic, you can use the `tmctl send-event` command and specify your Kafka target. + +```console +tmctl send-event --eventType io.foo.com '{"hello":"world"}' \ + --target do-kafkatarget +Destination: do-kafkatarget(http://localhost:54811) +Request: +------ +Context Attributes, + specversion: 1.0 + type: io.foo.com + source: triggermesh-cli + id: + datacontenttype: application/json +Data (binary), + { + "hello": "world" + } +------ +Response: OK +``` + +In the DO console if you go to the Insight tab of your Kafka cluster, you should be able to see the message count increase. + +## Create a Kafka Source + +To consume from your Kafka topic (e.g `tmctl`) you need to setup a Kafka source component like so: + +```console +tmctl create source kafka \ + --bootstrapServers \ + --topic tmctl \ + --groupID foobar \ + --auth.username doadmin \ + --auth.password \ + --auth.saslEnable true \ + --auth.tlsEnable true \ + --auth.securityMechanism PLAIN \ + --auth.tls.ca "$(cat /Users/sebgoa/Desktop/ca-certificate.crt)" +``` + +When describing your Pipe you will now see a Kafka source component which emits events of type `io.triggermesh.kafka.event` + +```console +$ tmctl describe +Broker Status +do online(http://localhost:54556) + +Source Kind EventTypes Status +do-kafkasource kafkasource io.triggermesh.kafka.event online(http://localhost:54968) + +Target Kind Expected Events Status +do-kafkatarget kafkatarget * online(http://localhost:54811) +``` + +## Route Kafka Events to a Microservice + +To visualize the Kafka events that you are consuming, you can create a local microservice and route the events to it by creating a `trigger`. + +First let's run a local event visualizer + +```console +tmctl create target --from-image gcr.io/triggermesh/triggermesh-console:v0.0.1 +``` + +And now create the `trigger` to send events of type `io.triggermesh.kafka.event` to it + +```console +tmctl create trigger --eventTypes io.triggermesh.kafka.event \ + --target do-target-service +``` + +When you describe the Pipe you will see the URL of the event visualizer in the _Status_ column, open it with your browser + +```console +tmctl describe +Broker Status +do online(http://localhost:54556) + +Trigger Target Filter +do-trigger-8cbcd2be do-target-service type is io.triggermesh.kafka.event + +Source Kind EventTypes Status +do-kafkasource kafkasource io.triggermesh.kafka.event online(http://localhost:55160) + +Target Kind Expected Events Status +do-kafkatarget kafkatarget * online(http://localhost:54811) +do-target-service service (gcr.io/triggermesh/triggermesh-console:v0.0.1) * online(http://localhost:55042) +``` + +Open the console + +```console +open http://localhost:55042 +``` + +When you send an event to your Kafka target with the `send-event` command, you will see it appear in the microservice. You successfully produced to a Kafka topic, consumed from it and displayed the event in a local microservice. + + +![](../assets/images/do-kafka-event.png) + +You can verify that all the components are running locally + +```console +docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +8a8f2b5073af gcr.io/triggermesh/kafkasource-adapter:v1.23.2 "/ko-app/kafkasource…" 4 minutes ago Up 4 minutes 0.0.0.0:55160->8080/tcp do-kafkasource +f669bb3ddcde gcr.io/triggermesh/triggermesh-console:v0.0.1 "/ko-app/triggermesh…" 18 minutes ago Up 18 minutes 0.0.0.0:55042->8080/tcp do-target-service +3dd0f90231c3 gcr.io/triggermesh/kafkatarget-adapter:v1.23.2 "/ko-app/kafkatarget…" 32 minutes ago Up 32 minutes 0.0.0.0:54811->8080/tcp do-kafkatarget +bc7986f80c62 gcr.io/triggermesh/memory-broker:latest "/memory-broker star…" 2 hours ago Up 2 hours 0.0.0.0:54556->8080/tcp do-broker +``` + +You can stop and start the entire Pipe with + +```console +tmctl stop +tmctl start +``` diff --git a/mkdocs.yml b/mkdocs.yml index 4ae1fb5a1..e165067c6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -149,6 +149,8 @@ nav: - Connecting Clusters: guides/connectingclusters.md - Using Kong with TriggerMesh: guides/kong-ingress.md - Using Kuma Mesh with TriggerMesh: guides/kuma.md + - Using a Kafka Cluster on Digital Ocean: guides/do-kafka.md + - Deploying to Digital Ocean App Platform: guides/do-app-platform.md - Creating a PrivateLink to TriggerMesh Services: guides/triggermesh-privatelink.md - Scaling: guides/scaling.md - Webhook to Slack: guides/webhook-to-slack.md