Lambda troubleshooting

Symptom, cause and fix for functions that will not run behind Lambda.

Containers cannot reach the Runtime API

Overcast establishes this address rather than assuming it. At startup it tries each candidate in turn — best first — by binding it and running a throwaway busybox container that connects back, and keeps the first one a container actually reaches:

Verification uses a throwaway ephemeral port rather than the Runtime API port. The causes it catches are port-independent: a Windows firewall rule is per-program, Docker Desktop routes per-address, an --internal bridge severs a path rather than a port. So container_verified=true means the address is reachable, not that a port-specific rule was tested.

The result is remembered per Docker daemon and per control plane in your data directory (runtime-api-host-<network>.json), so only the first startup against a given daemon pays for the probe, and two instances sharing a data directory with different OVERCAST_NETWORK values do not overwrite each other. It is dropped automatically when a container later dies during INIT without having reached the address.

If the probe cannot run at all — an air-gapped host with no cached busybox, or a daemon that refuses container creates — Overcast keeps the ordering above, reports nothing as broken, and logs that the address was chosen without a check.

The startup log says which candidate won:

lambda: Runtime API address   addr=host.docker.internal:9001 mode=docker-internal container_verified=true

When nothing is reachable

/_overcast/health goes degraded with the Runtime API listener in state unreachable, a critical advisory appears on the console’s Metrics & Health page, and one error is logged naming every address tried:

no Runtime API address is reachable from a container — every Lambda invocation will fail during INIT
tried=["gateway 172.19.0.1: could not bind 172.19.0.1", "docker-internal host.docker.internal: wget: download timed out",
       "host 192.168.8.19: wget: download timed out", "wildcard host.docker.internal: wget: download timed out"]

The observed error is the diagnosis: Connection refused means nothing was listening there, a timeout means the packets were dropped on the way in.

Two causes, in order of likelihood:

• A host firewall blocks inbound connections to the Overcast binary. On Windows this is the default for a binary with no allow rule — a freshly built overcast.exe is blocked until someone clicks Allow, and the rule follows the exact path of the binary, so a rebuild somewhere else is blocked again. Add an allow rule for that path, or set LAMBDA_RUNTIME_API_HOST=host.docker.internal to pin the candidate that does not take the filtered path, or run Overcast itself as a container (overcast start --docker --mount-docker-socket), which puts it on the same network as the functions and takes the host route out of the picture.
• The _control network was created internal. An internal network has no route off the bridge at all. Overcast warns at startup when the existing network’s isolation differs from what it wants, and rebuilds it itself when nothing is attached. When something is, run overcast network reset overcast_control — it stops the containers Overcast started, disconnects the ones it did not and leaves them running, then rebuilds the network to spec. Add --dry-run to see the plan first. Or point this instance at its own network with OVERCAST_NETWORK.

To check it by hand, against the network Overcast starts containers on:

docker run --rm --network overcast_control busybox \
  wget -T 4 -qO- http://192.168.1.20:4566/_overcast/health

The exit-139 signature

An unreachable address used to present only as this, and the number points at the wrong subsystem — the Node runtime segfaults because invocation/next never answers, not because the init binary is broken:

lambda container exited during INIT   runtime_api=192.168.1.20:54254 runtime_api_connections=0
runtime_api_mode=host runtime_api_container_verified=false
container_output=[overcast-init] runtime API proxy error: GET /2018-06-01/runtime/invocation/next: dial tcp 192.168.1.20:54254: i/o timeout

runtime_api_connections=0 beside runtime_api_container_verified=false is the signature. When the probe established the failure in advance, the Runtime.InitError returned to the caller carries the same explanation.

Working out why a warm environment went away

GET /_overcast/lambda/instances lists one entry per execution environment, and the lambda:InstanceEvicted event carries an evictedReason:

Related

• Lambda — quick start and what works
• Lambda limitations — concurrency, runtimes, logging, VPC placement
• Lambda examples — hot reload, container images, layers, extensions
• CloudWatch Logs — where function output lands