MSK — Managed Streaming for Kafka
MSK clusters backed by a real Redpanda broker per cluster, promoted to ACTIVE only once it answers Kafka's ApiVersions. Broker scaling, SCRAM and VPC connections are not implemented.
A provisioned cluster starts a real Redpanda broker you can produce to and consume from; without Docker it is metadata only.
Status: ⚠️ Partial
Quick start
export AWS_ENDPOINT_URL=http://localhost:4566
ARN=$(aws kafka create-cluster --cluster-name events \
--kafka-version 3.6.0 --number-of-broker-nodes 1 \
--broker-node-group-info '{"InstanceType":"kafka.m5.large","ClientSubnets":[]}' \
--query ClusterArn --output text)
aws kafka describe-cluster --cluster-arn "$ARN" # wait for ACTIVE
aws kafka get-bootstrap-brokers --cluster-arn "$ARN"
# → { "BootstrapBrokerString": "127.0.0.1:49092" }Any credentials work; with none configured, run eval "$(overcast env)" first
— see Using AWS SDKs and CLI.
What works
| Area | Behaviour |
|---|---|
| Real broker | CreateCluster starts a Redpanda container, ports allocated from MSK_PORT_BASE (default 49092) |
| Honest readiness | CREATING → ACTIVE only once the broker answers a Kafka ApiVersions request — a published port that merely accepts a TCP connection is not enough |
| Failure is terminal | A broker that never answers within 120 seconds settles the cluster in FAILED, with the reason in stateInfo, so aws kafka wait cluster-active stops instead of spinning |
| Bootstrap | GetBootstrapBrokers answers per caller: {cluster}.{region}.kafka.{base}:9092 for a sibling container, the published host port for the host, so one stack output works from both sides |
| v1 and v2 | /v1/ and /api/v2/clusters share one cluster store, so a cluster created through either is visible through both |
| Serverless | CreateClusterV2 accepts a serverless cluster as metadata, immediately ACTIVE |
| VPC placement | A cluster whose ClientSubnets name a VPC has its broker placed on that VPC’s network, so only callers in it can reach the broker |
| Configurations | Create, describe, list, delete, and UpdateClusterConfiguration with currentVersion validation |
| Tags | TagResource, ListTagsForResource, UntagResource on any MSK ARN |
Differences from AWS
| Area | On AWS | Overcast |
|---|---|---|
| Broker count | NumberOfBrokerNodes brokers | Recorded, not honoured — a single Redpanda node whatever the request asks for |
| Broker software | Apache Kafka | Redpanda: wire-compatible with the Kafka protocol, but broker internals, JMX metrics and Kafka-specific admin behaviour differ |
ListKafkaVersions | The versions the service offers | Fixed at 3.6.0, 3.5.1, 3.4.0, 2.8.1 and 2.6.0, whatever the container runs |
| Broker changes | UpdateBrokerCount, UpdateBrokerStorage, UpdateBrokerType, UpdateMonitoring, UpdateSecurity and RebootBroker | All return 501 |
| SASL/SCRAM | The secret-association operations, and an authenticated broker | They return 501; the broker is reachable without authentication |
| VPC connections | CreateVpcConnection and its siblings | They return 501 |
| Encryption, auth, logging and storage settings | Applied to the cluster | encryptionInfo, clientAuthentication, loggingInfo, openMonitoring and a broker group’s storageInfo are discarded — a cluster keeps only instanceType, clientSubnets, securityGroups and brokerAZDistribution |
Gotchas
Important
A cluster name already in use in the region is a ConflictException, and
CreateCluster and CreateClusterV2 share that namespace.
Broker containers are scoped to the instance that created them.
Note
Two Overcasts on one Docker daemon do not disturb each other’s brokers: containers carry the identity of the instance that created them, and the startup sweep and Docker event stream both match on it.
Operations
17 of 30 listed operations are implemented. Per-operation status, notes and AWS API links: MSK operations.