Fix: FastAPI 401 Unauthorized on AWS ECS after 48 Hours

The Issue

In high-availability AWS environments, a common “phantom” bug occurs where a FastAPI application, successfully authenticated against AWS Cognito or a custom JWT provider, begins returning 401 Unauthorized responses after several days of uptime.

The application logic remains unchanged, and the same tokens work in development environments. This usually points to a synchronization issue between the AWS host instance (EC2 or ECS Fargate) and the token’s exp (Expiration) or iat (Issued At) claims, or an expiration of the underlying IAM Role credentials used to fetch public keys from the AWS Identity and Access Management (IAM) service.

Exact Error Log

When the failure occurs, the FastAPI console logs or CloudWatch Logs will show the following traceback:

INFO:     10.0.x.x:54321 - "GET /api/v1/protected-route HTTP/1.1" 401 Unauthorized
ERROR:    fastapi_auth: Authentication failed: Signature has expired.
Traceback (most recent call last):
  File "auth/dependencies.py", line 42, in get_current_user
    payload = jwt.decode(token, PUBLIC_KEY, algorithms=["RS256"])
  File "/usr/local/lib/python3.11/site-packages/jwt/api_jwt.py", line 156, in decode
    return self._validate_claims(payload, merged_options, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/jwt/api_jwt.py", line 167, in _validate_claims
    self._validate_exp(payload, now, leeway)
jwt.exceptions.ExpiredSignatureError: Signature has expired

Alternatively, if the system clock has drifted backwards:

jwt.exceptions.ImmatureSignatureError: The token is not yet valid (iat_claim)

Diagnostic Checklist

1. Check Clock Drift on the Host: Connect to the EC2 instance or the ECS Task (via ECS Exec) and verify the system time against an atomic clock.

   # Check current time and sync status
   timedatectl status

2. Verify AWS Secrets Manager Rotation: If you are using AWS Secrets Manager to store your JWT_SECRET, check the “Rotation” tab in the AWS Console. Ensure the FastAPI app is fetching the secret on a schedule, rather than only once at startup.
3. Inspect Task Metadata Endpoint: If running on ECS, ensure the task is not losing connectivity to the Amazon ECS Task Metadata Endpoint, which provides the IAM credentials needed to verify tokens if using Cognito.

   curl ${ECS_CONTAINER_METADATA_URI_V4}/task

4. Validate JWT Leeway: Check if your Python jose or PyJWT implementation has a leeway configured. AWS instances can drift by several seconds per day if NTP is misconfigured.

The Fix

1. Implement Clock Synchronization (EC2/ECS Host)

Ensure the Amazon Time Sync Service is running. This is critical for JWT validation. For EC2-based deployments, configure chrony.

# Install chrony
sudo yum install chrony -y

# Start the service
sudo systemctl start chronyd
sudo systemctl enable chronyd

# Verify that the Amazon server (169.254.169.123) is being used
chronyc sources -v

2. Update FastAPI Authentication Logic

In your FastAPI dependency, add a leeway to the jwt.decode method. This allows for a small margin of error (e.g., 60 seconds) between the token issuer’s clock and the AWS instance’s clock.

from fastapi import HTTPException, status
import jwt

ALGORITHM = "RS256"
# Allow 60 seconds of clock drift
LEEWAY = 60

def get_current_user(token: str):
    try:
        payload = jwt.decode(
            token,
            PUBLIC_KEY,
            algorithms=[ALGORITHM],
            leeway=LEEWAY
        )
        return payload
    except jwt.ExpiredSignatureError:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Token expired"
        )

3. Standardize the Dockerfile for AWS

Ensure the container environment inherits the correct timezone and utilizes the host’s clock properly. Do not hardcode timezones inside the image.

FROM python:3.11-slim

# Ensure system dependencies for time management are present
RUN apt-get update && apt-get install -y \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -r requirements.txt

# Use uvicorn with a worker timeout for AWS ALB compatibility
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--keep-alive-timeout", "65"]

4. IAM Role & Credential Refresh (For Fargate)

If the error persists after 48 hours, ensure your application is not caching the IAM Role session longer than the AWS STS (Security Token Service) allows. If using the AWS SDK inside FastAPI to fetch keys, use the standard boto3 session management which handles refreshes automatically:

# ECS Task Definition Snippet
TaskRoleArn: arn:aws:iam::123456789012:role/FastApiAppRole
ContainerDefinitions:
  - Name: fastapi-app
    Environment:
      - Name: AWS_DEFAULT_REGION
        Value: "us-east-1"

By combining NTP synchronization on the host and JWT leeway in the application code, you eliminate the “drift” that causes authentication to fail after 48 hours of continuous uptime.