Overcast is alpha — behaviour and APIs may change between releases. Pin your version and read the changelog before upgrading.

overcast

Pipes — Amazon EventBridge Pipes

Quick start, the source, enrichment and target combinations accepted, batching, transformation and partial batch failures, and the fields dropped on decode.

Point-to-point wiring from a source, through an optional Lambda enrichment, to a target — polled and delivered for real.

Status: ⚠️ Partial

Quick start

export AWS_ENDPOINT_URL=http://localhost:4566

SRC=$(aws sqs create-queue --queue-name inbox --query QueueUrl --output text)
SRC_ARN=$(aws sqs get-queue-attributes --queue-url "$SRC" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)
DST=$(aws sns create-topic --name fanout --query TopicArn --output text)

aws pipes create-pipe --name inbox-to-topic \
  --role-arn arn:aws:iam::000000000000:role/pipes \
  --source "$SRC_ARN" --target "$DST"

aws sqs send-message --queue-url "$SRC" --message-body '{"id":1}'
aws pipes describe-pipe --name inbox-to-topic

Any credentials work; with none configured, run eval "$(overcast env)" first — see Using AWS SDKs and CLI.

What works

LegSupportedRefused at CreatePipe/UpdatePipe
SourceDynamoDB Streams, Kinesis streams, SQS queuesEverything else, including Amazon MQ and Kafka
EnrichmentA Lambda function (optional)API destinations, API Gateway, Step Functions
TargetLambda, SQS, SNS, Step Functions, Kinesis, Firehose, EventBridge event busECS tasks, Batch, CloudWatch Logs, Redshift Data, SageMaker, Timestream, API destinations

A combination Overcast cannot run is rejected with a ValidationException naming the offending field, so a pipe is never stored in a state where it would silently do nothing.

AreaBehaviour
BatchingA source produces a JSON array of records. A target with a batch API receives one entry per record; Lambda and Step Functions receive the whole array in one request, as on AWS. BatchSize applies to the polled sources.
EnrichmentInvoked synchronously; its return value replaces the batch. An empty response ("", {}, [], null) drops the batch without invoking the target; [{}] invokes the target with an empty payload.
Input transformationEnrichmentParameters.InputTemplate and TargetParameters.InputTemplate are applied per record, over the JSON-path subset AWS accepts plus the aws.pipes.* reserved variables.
PollingKinesis and SQS sources are polled once per second on the emulator’s clock.
Partial batch responsesA Lambda target’s response is read for batchItemFailures, and only the records it names are redelivered.
LifecycleState transitions (CREATING→RUNNING, STOPPING→STOPPED, …) happen asynchronously with a short delay; setting DesiredState on update triggers one.

Differences from AWS

AreaOn AWSOvercast
FilterCriteriaServer-side filteringRejected outright rather than stored and ignored — filter inside a Lambda enrichment instead
Stream retriesRetry until expiryCapped at 5 retries (an explicit 0 means one attempt); unset means the cap, standing in for AWS’s “retry until the record expires”
Dead-letter payloadA shard/sequence-range failure envelopeThe source records themselves, so the batch is replayable without a second read
DynamoDB Streams batchingBatchSize appliesBus-driven one record at a time, so always a batch of one
SQS dead-letteringSameLeft to the queue’s own RedrivePolicy, as on AWS — DeadLetterConfig does not apply to an SQS source
ParallelizationFactorHonouredStored and never read
LogConfiguration, KmsKeyIdentifierStored and honouredNot modelled — discarded on decode and absent from DescribePipe
RoleArnEnforcedRequired, as AWS requires it, but never evaluated

Once a stream source’s retries are exhausted the batch goes to the source’s DeadLetterConfig SQS queue or SNS topic if one is configured, and is otherwise reported failed with a logged warning while the cursor moves on rather than blocking forever. A DeadLetterConfig naming anything but an SQS queue or SNS topic is refused at create and update time.

Gotchas

Warning

LogConfiguration and KmsKeyIdentifier are dropped, not stored inert. A DescribePipe assertion that round-trips either field will fail.

Only a Lambda target reports partial batch failures. A Step Functions target’s asynchronous StartExecution has no response to read, and a Lambda enrichment’s return value AWS defines as replacing the batch rather than reporting on it. AWS offers no partial-batch reporting for any other target type either.

Operations

All 8 listed operations are implemented. Per-operation status, notes and AWS API links: Pipes operations.