Troubleshooting

SSH Issues

Connection Timeout


Error: Connection timed out after 30000ms

Usually the server is unreachable, a firewall is blocking port 22, or the hostname is wrong.


ping server.example.com
ssh -v user@server.example.com
ssh user@server "sudo iptables -L -n | grep 22"

Authentication Failed


Error: All configured authentication methods failed


chmod 600 ~/.ssh/id_ed25519
ssh -i ~/.ssh/id_ed25519 user@server
cat ~/.ssh/authorized_keys       # on the server
jiji --verbose server exec "echo ok" -H server1

ProxyJump Issues


Error: Could not establish proxy connection through bastion.example.com


ssh -J bastion.example.com user@internal-server
cat ~/.ssh/config


ssh:
  user: deploy
  proxy: deploy@bastion.example.com

Too Many Authentication Failures


Error: Received too many authentication failures

SSH gives up after too many offered keys, usually because your agent is holding more keys than the server will tolerate. Limit what’s offered:


ssh:
  user: deploy
  keys_only: true
  keys:
    - ~/.ssh/deploy_key


ssh-add -D                  # clear every key from the agent
ssh-add ~/.ssh/deploy_key   # load only the one jiji should use

About `jiji server exec`

A plain command runs on every host -H/--hosts matches, concurrently by default (--sequential for one at a time). An interactive session - no command given, or --interactive - is bound to one local terminal, so it requires -H to resolve to exactly one server. -S/--services isn’t accepted. Interactive sessions automatically downgrade to non-interactive with a warning when stdin/stdout isn’t a real terminal.

Registry Issues

401 Unauthorized


Error: unauthorized: authentication required


jiji secrets print
cat .env | grep GITHUB_TOKEN
jiji registry login
jiji --host-env registry login   # if the token is a host env var, not in .env

403 Forbidden


Error: denied: permission denied

Usually the token is missing a scope, the repository is private, or the username doesn’t match the token owner. For GHCR, confirm the token has write:packages.

Push Failed


docker ps | grep jiji-registry   # for a local registry
df -h
curl -I https://ghcr.io

Local Registry Not Accessible


Error: Failed to connect to localhost:31270


curl http://localhost:31270/v2/          # {} means it's up
 
# On the remote server during deployment
netstat -tlnp | grep 31270

If the remote side can’t reach the tunnel, check sshd_config on that server for AllowTcpForwarding yes. GatewayPorts is not required because Jiji binds the forwarded listener to the server’s 127.0.0.1 address.

Deployment Issues

Health Check Timeout


Error: Health check failed after 60s

The health check runs directly against the candidate container’s own backend address, so a timeout here means the application itself isn’t answering yet, not a routing problem.


jiji service logs -S api --since 5m
jiji server exec "docker ps -a" -H server1
jiji server exec "docker exec <container> curl localhost:3000/health" -H server1


proxy:
  healthcheck:
    path: /health
    deploy_timeout: 120s

Container Crashes


jiji server exec "docker logs <container>" -H server1
jiji server exec "dmesg | grep -i oom" -H server1
jiji server exec "docker inspect <container> | grep -A20 Env" -H server1

Deployment Hangs


jiji lock status
jiji lock release          # if a stale lock is the cause
jiji --verbose deploy

Deploy Was Killed or Lost Its Connection Mid-Deploy

If jiji deploy itself dies (network drop, Ctrl-C, CI job killed) after the new container starts but before its health check finishes, the container is left running with no CLI process left to finish the cutover. jiji-agent notices this on its own next reconcile tick or restart and replays the same health check the deploy would have used, rather than assuming a running container is a healthy one:


jiji network catalog -H server1       # the leftover deployment shows as candidate
jiji network diagnostics -H server1   # shows why it hasn't been promoted, if it hasn't

If the check passes, the agent finishes the cutover itself within a few seconds, no further action needed.
If it keeps failing, the container is left alone (not serving traffic, not removed) until it starts passing or you clean it up yourself (jiji service restart -S <service> or jiji service remove -S <service> followed by a fresh jiji deploy). The previous deployment keeps serving the whole time.

Container Issues

Volume Mount Permission Denied


Error: Permission denied: /data/app


jiji server exec "ls -la /data/app" -H server1
jiji server exec "docker exec <container> id" -H server1        # container's uid/gid
jiji server exec "sudo chown -R 1000:1000 /data/app" -H server1

Port Already in Use


Error: Bind for 0.0.0.0:3000 failed: port is already allocated


jiji server exec "sudo lsof -i :3000" -H server1
jiji server exec "docker ps | grep 3000" -H server1
jiji server exec "docker stop <container-id>" -H server1

Or map to a different host port in deploy.yml:


services:
  api:
    ports:
      - "3001:3000"

Network Issues

Interface, systemd unit, and path names are all derived per project - run jiji network plan to get the exact values for your config, and substitute them for <wireguard_interface>/<slug> below. See Network Reference for the full mapping.

WireGuard Not Connecting

jiji-agent brings up the WireGuard interface itself at startup and repairs it if torn down externally - there is no separate wg-quick@ systemd unit to check.


jiji server exec "sudo wg show <wireguard_interface>" -H server1
jiji server exec "sudo systemctl status jiji-agent-<slug>" -H server1
jiji server exec "sudo journalctl -u jiji-agent-<slug> -n 50" -H server1
jiji network plan
jiji network setup

DNS Not Resolving


jiji server exec "sudo systemctl status jiji-agent-<slug>" -H server1
jiji server exec "docker exec <container> getent hosts myapp-api.jiji" -H server1
jiji server exec "sudo journalctl -u jiji-agent-<slug> -n 50" -H server1
jiji network catalog

DNS is answered from the replicated catalog and returns only reachable, healthy Active deployment addresses.

Container Can’t Resolve External Hostnames

A service that calls a normal internet API (or does anything else that needs a public hostname to resolve) fails with something like Name or service not known or getaddrinfo failed, even though .jiji names resolve fine:


jiji server exec "docker exec <container> getent hosts myapp-api.jiji" -H server1  # works
jiji server exec "docker exec <container> getent hosts api.example.com" -H server1  # fails

A service container’s resolv.conf only ever has the project’s own DNS address as its nameserver; anything outside the .jiji zone is forwarded to network.dns_forwarders (default 1.1.1.1/8.8.8.8, see the Configuration Reference). If external lookups still fail, check that the configured forwarders are actually reachable from the server (not just from your workstation):


jiji server exec "sudo journalctl -u jiji-agent-<slug> -n 50" -H server1
jiji server exec "dig @1.1.1.1 api.example.com" -H server1

If your network blocks outbound DNS to public resolvers, set network.dns_forwarders to a resolver the server can actually reach (a home router, internal DNS server, Pi-hole, etc.).

Containers Can’t Communicate


jiji server exec "ip route show dev <wireguard_interface>" -H server1
jiji server exec "docker exec <container> getent hosts myapp-api.jiji" -H server1
jiji server exec "docker exec <container> ping <leased-address>" -H server1

Build Issues

Build Context Too Large

Add a .dockerignore:


node_modules
.git
*.log
.env*

Dockerfile Not Found


services:
  api:
    build:
      context: .
      dockerfile: docker/Dockerfile

Build Arguments Not Working

args: is a mapping of build-arg names to values, not a list of KEY=value strings:


services:
  api:
    build:
      context: .
      args:
        NODE_ENV: production
        VERSION: "1.2.3"

Proxy Issues

502 Bad Gateway

Usually the container isn’t running, port doesn’t match what the container actually listens on, or the container hasn’t passed its health check yet.


jiji server exec "docker ps" -H server1
jiji proxy logs --since 5m
jiji server exec "docker port <container>" -H server1

Routing Not Working


jiji proxy logs --grep "route"
jiji server exec "curl -H 'Host: myapp.example.com' localhost:80" -H server1

SSL Certificate Errors


jiji proxy logs --grep "ssl\|cert\|tls" --grep-options "-E"

Confirm ssl: true is actually set on the service’s proxy: config, and that DNS for the hostname points at the server.

Performance Issues

Slow Deployments

Enable build caching (builder.cache: true), keep .dockerignore tight, order Dockerfile layers so dependencies install before source is copied, or set builder.remote to offload builds to a dedicated host.

High Memory Usage


jiji server exec "docker stats --no-stream" -H server1
jiji server exec "free -h" -H server1
jiji service prune

Debugging Commands


jiji --verbose deploy
jiji server exec "systemctl list-units 'jiji-*'" -H server1
jiji server exec "docker ps --filter 'label=jiji.managed=true'" -H server1
jiji server exec "docker inspect <container>" -H server1
jiji audit --lines 20
jiji audit --status failed

Agent Version Mismatches

“Agent rejected request … incompatible protocol or schema”

A node is running a different jiji-agent version than the rest of the project’s mesh. Mixed versions are rejected outright rather than partially joining - upgrade every host’s agent (reinstall via jiji server setup) to the same version before retrying.

Common Error Messages

“Configuration already exists at …”

jiji init found a config file already at that path. It prompts to overwrite; answer no and edit the existing file instead if that’s what you meant to do.

“Configuration validation failed with N error(s)”

The listed errors are specific (missing field, invalid CIDR, duplicate host, etc.) - run with --verbose for more detail, or lint the YAML directly:


yamllint .jiji/deploy.yml

“Could not acquire the deployment lock”

Another deploy, service restart, service rollback, or an explicit maintenance lock is active. A process interrupted before cleanup may also leave a stale lock.


jiji lock status
jiji lock release          # once you've confirmed it's safe

Registry push/pull fails with “denied” or “unauthorized”

The image wasn’t pushed, or the wrong registry/tag is configured.


docker images | grep myproject
jiji build
jiji registry login

“Container … already exists but is not Jiji’s registry”

Something else is using the local registry’s container name or port. Remove or rename that container, or change builder.registry.port.

Getting Help

Still stuck? Include the following when asking for help or filing an issue:

jiji version
Operating system (yours and the target server’s)
The exact error message
Your deploy.yml, with secrets redacted
Steps to reproduce


jiji --verbose <command>    # re-run with full detail
jiji audit --status failed  # recent failures, per server

GitHub Issues - search first, then open a new one
Discord - for questions and discussion