Connect to Kafka on host from Docker (ksqlDB)

by

Modify Kafka’s server.properties

listener.security.protocol.map=PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_LOCAL:PLAINTEXT

listeners=PLAINTEXT_DOCKER://:29092,PLAINTEXT_LOCAL://localhost:9092

advertised.listeners=PLAINTEXT_DOCKER://host.docker.internal:29092,PLAINTEXT_LOCAL://localhost:9092

inter.broker.listener.name=PLAINTEXT_LOCAL

Update your Compose like so to point at the host rather than itself

version: '3.9'

services:

  # TODO: add schema-registry
  #   environment: 
  #     SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: host.docker.internal:29092
  #   extra_hosts:
  #     - "host.docker.internal:host-gateway"

  # or any other Kafka client 
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.18.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
       KSQL_BOOTSTRAP_SERVERS: host.docker.internal:29092
       ...

(Tested on Mac), Getting /info endpoint of KSQL

http :8088/info
HTTP/1.1 200 OK
content-length: 133
content-type: application/json

{
    "KsqlServerInfo": {
        "kafkaClusterId": "ZH2-h1W_SaivCW0qa8DQGA",
        "ksqlServiceId": "default_",
        "serverStatus": "RUNNING",
        "version": "0.18.0"
    }
}

Replace all host.docker.internal above with the external hostname/IP of the machine, if Kafka is a remote server

More Related Contents:

  1. Kafka access inside and outside docker [duplicate]
  2. Docker Compose wait for container X before starting Y
  3. Using Docker-Compose, how to execute multiple commands
  4. What is the difference between docker-compose ports vs expose
  5. How to mount a single file in a volume
  6. Docker: How to clear the logs properly for a Docker container?
  7. Docker : Can a container A call an executable located on an other container B?
  8. How to specify Memory & CPU limit in docker compose version 3
  9. How to pass arguments within docker-compose?
  10. How do I mount a host directory as a volume in docker compose
  11. What is the difference between ports and expose in docker-compose?
  12. How to use environment variables in docker-compose?
  13. Why the “none” image appears in Docker and how can we avoid it
  14. How do I set hostname in docker-compose?
  15. How to mount local volumes in docker machine
  16. Is it safe to clean docker/overlay2/
  17. How to pass ARG value to ENTRYPOINT?
  18. Using docker-compose with CI – how to deal with exit codes and daemonized linked containers?
  19. How do I configure docker compose to expose ports correctly?
  20. Where is a log file with logs from a container?
  21. docker-compose build environment variable
  22. how do you manage secret values with docker-compose v3.1?
  23. docker-compose: difference between networks and links
  24. How to directly mount NFS share/volume in container using docker compose v3
  25. Substitute environment variables in NGINX config from docker-compose
  26. Connect to another container using Docker compose
  27. Build a single image based on docker compose containers
  28. How to define build-args in docker-compose?
  29. What is the purpose of the file “docker.sock”?
  30. How to stop all containers when one container stops with docker-compose?

Leave a Comment