Long-Running Job Design

Concept

Long-running job design is the set of patterns for work that exceeds request/response limits — OCR, large-file processing, batch embedding, and RAG indexing — using async APIs, state tables, and durable workers.

Problem it solves. Synchronous HTTP has timeouts (Cloud Run request timeouts, load-balancer limits). Long-running job design moves heavy work off the request path so clients get fast acknowledgements and reliable, resumable processing.

Limitations

• Service quotas and limits apply — always check current documentation
• Regional availability and launch stage vary by feature
• Cost scales with usage; model it before committing

Interactive architecture

flowchart TD
  C[Client] -->|1. submit| API[FastAPI]
  API -->|2. validate| API
  API -->|3. create job record| DB[(State table)]
  API -->|4. publish| PS((Pub/Sub))
  PS -->|5. process| W[Worker: Cloud Run Job]
  W -->|6. update state| DB
  W -.7. retry / DLQ.-> DLQ((DLQ))
  W -->|8. persist result| DB
  C -->|9. poll status| API
  classDef gcp fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  class API,DB,PS,W,DLQ gcp;

flowchart TD
  C[Client] -->|1. submit| API[FastAPI]
  API -->|2. validate| API
  API -->|3. create job record| DB[(State table)]
  API -->|4. publish| PS((Pub/Sub))
  PS -->|5. process| W[Worker: Cloud Run Job]
  W -->|6. update state| DB
  W -.7. retry / DLQ.-> DLQ((DLQ))
  W -->|8. persist result| DB
  C -->|9. poll status| API
  classDef gcp fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  class API,DB,PS,W,DLQ gcp;

How it works

The sections below break Long-Running Job Design down into request/event flow, authentication and authorization, the internal actions Google Cloud performs, and the failure and monitoring signals you should watch in production.

1. 1 · Request or event arrivesEntry point (load balancer, trigger, or API)

   Traffic reaches Long-Running Job Design through a defined entry point with TLS termination and, where relevant, Cloud Armor protection.

   ⚠ 429/5xx if quotas or backends are exhausted 📈 Request count, latency, and error-rate metrics 💰 Ingress is generally free; egress and requests are billed
2. 2 · AuthenticationIAM / identity layer

   The caller's identity (user, service account, or federated workload) is verified before any work is done.

   ⚠ 401/403 on invalid or missing credentials 📈 Cloud Audit Logs (Admin & Data Access)
3. 3 · AuthorizationIAM policy evaluation

   IAM checks whether the principal holds a role granting the required permission on the resource, honoring inheritance and deny policies.

   ⚠ 403 PERMISSION_DENIED 📈 Policy Analyzer, audit logs
4. 4 · Service performs the workLong-Running Job Design

   The service executes the requested action, reading or writing data and emitting structured logs and traces.

   ⚠ Retryable vs non-retryable errors; watch for partial failure 📈 Structured logs, Cloud Trace spans 💰 Compute/storage/IO billed per the pricing model
5. 5 · Response / acknowledgementCaller

   A result is returned synchronously, or an acknowledgement/notification is emitted for asynchronous flows.

   📈 SLI dashboards, error budgets

Hands-on example

Cloud Console

1. Open the Cloud Console and navigate to the Long-Running Job Design section
2. Enable the required API(s) for your project
3. Create the resource with least-privilege settings
4. Verify it appears and note its identifiers

Code

# Set project context
gcloud config set project MY_PROJECT

# Enable required APIs (replace with the service's API)
gcloud services enable SERVICE.googleapis.com

# Create / describe the resource (see per-service docs for exact flags)
# gcloud <group> create NAME --region=us-central1

# Verify
# gcloud <group> describe NAME --region=us-central1

# Uses Application Default Credentials (no service-account keys).
# pip install google-cloud-<service>
from google.cloud import <service>  # replace with the concrete client

def main() -> None:
    client = <service>.Client()
    # ... call the API using the client, letting ADC handle auth ...
    print("connected")

if __name__ == "__main__":
    main()

terraform {
  required_providers {
    google = { source = "hashicorp/google", version = "~> 6.0" }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

# resource "google_..." "this" {
#   name     = "example"
#   # least-privilege, private-by-default configuration
# }

Troubleshooting

• 403 → check IAM roles and the caller's identity
• Connectivity failures → check VPC, firewall, and Private Google Access
• Unexpected cost → check billing exports and Recommender

Production considerations

What interviewers expect you to know

Common mistakes

• Granting overly broad IAM roles (e.g. project-level Owner/Editor)
• Using service-account keys where Workload Identity Federation would work
• Exposing services publicly when they should be private
• Choosing the wrong compute or storage primitive for the access pattern
• Retrying without backoff, or ignoring idempotency
• Ignoring quotas, dead-letter handling, and structured logging
• Leaving demo resources running and ignoring data-transfer costs
• Mixing production and development in one project

Knowledge check