Vertex AI (Generative AI)

Concept

Vertex AI is Google Cloud's unified ML and generative-AI platform. For GenAI it provides managed foundation models (the Gemini family and others via Model Garden), embeddings, grounding, safety controls, evaluation, and — through the Google Gen AI SDK — a consistent API for text, multimodal input, streaming, structured output, and function/tool calling.

Problem it solves. It lets you build GenAI applications without hosting models yourself, with enterprise controls: IAM, VPC-SC, data residency, private connectivity, safety filters, and observability. It unifies model access, embeddings, vector search, and evaluation behind one platform.

Limitations

• Per-model context-window, region availability, and quota limits
• Token cost and latency scale with prompt/response size
• Model versions change — pin versions and keep a config-driven registry
• Safety filters can block outputs; you must handle blocked responses

Interactive architecture

flowchart LR
  APP[FastAPI on Cloud Run] --> ADC[ADC / Workload Identity]
  ADC --> VX[Vertex AI endpoint]
  VX --> GEM[Gemini model]
  VX --> SAFE[Safety filters]
  APP --> EMB[Embeddings API]
  EMB --> VS[(Vector Search / AlloyDB AI)]
  APP --> TRC[Cloud Trace + token metrics]
  VX -.tool call.-> TOOL[Your functions / RAG retriever]
  classDef gcp fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  class APP,VX,GEM,EMB,VS,TRC,TOOL,ADC,SAFE gcp;

flowchart LR
  APP[FastAPI on Cloud Run] --> ADC[ADC / Workload Identity]
  ADC --> VX[Vertex AI endpoint]
  VX --> GEM[Gemini model]
  VX --> SAFE[Safety filters]
  APP --> EMB[Embeddings API]
  EMB --> VS[(Vector Search / AlloyDB AI)]
  APP --> TRC[Cloud Trace + token metrics]
  VX -.tool call.-> TOOL[Your functions / RAG retriever]
  classDef gcp fill:#e8f0fe,stroke:#4285f4,color:#174ea6;
  class APP,VX,GEM,EMB,VS,TRC,TOOL,ADC,SAFE gcp;

How it works

Your app authenticates with ADC and calls a Vertex AI endpoint. Requests are authorized by IAM, run through safety filters, and billed by input/output tokens (or per-character/per-image depending on the model). Streaming, tool calling, and structured output are request options, not separate services.

1. 1 · App builds a requestYour service (Gen AI SDK)

   You construct a prompt with system instructions, optional tools, and generation config (temperature, max tokens, response schema).

   💰 Larger prompts = more input tokens
2. 2 · AuthenticationADC / Workload Identity

   The SDK uses Application Default Credentials; on Cloud Run/GKE this is a workload identity — no keys.

   ⚠ 403 if the SA lacks roles/aiplatform.user 📈 Audit logs
3. 3 · Vertex AI authorizes & routesVertex AI

   IAM authorizes the call; the request routes to the requested model + version in the requested region.

   ⚠ 404/400 on unknown model/version; 429 on quota 📈 Quota metrics
4. 4 · Safety & generationModel + safety filters

   The model generates (optionally streaming). Safety filters may block or annotate output; tool calls pause generation for your app to execute the tool and return results.

   ⚠ Blocked responses (safety), truncation at max tokens 📈 Finish reason, safety ratings
5. 5 · Response handlingYour service

   Parse text or structured JSON, record input/output token counts, attach citations if grounding was used, and stream to the client.

   📈 Token counts, latency, cost per response

Hands-on example

Cloud Console

1. Cloud Console → Vertex AI → enable the Vertex AI API
2. Open Vertex AI Studio to prototype prompts and copy generated code
3. Model Garden → browse available managed and open models
4. Grant the runtime service account roles/aiplatform.user

Code

PROJECT=my-project
gcloud config set project "$PROJECT"
gcloud services enable aiplatform.googleapis.com

# Grant the Cloud Run runtime SA permission to call Vertex AI
gcloud projects add-iam-policy-binding "$PROJECT" \
  --member "serviceAccount:rag-api-sa@$PROJECT.iam.gserviceaccount.com" \
  --role roles/aiplatform.user

# Quick REST smoke test with an access token (model id is configuration-driven)
LOC=us-central1
MODEL=${GENAI_MODEL:-gemini-2.0-flash}  # keep model ids in config, not hard-coded
curl -s -X POST \
  "https://$LOC-aiplatform.googleapis.com/v1/projects/$PROJECT/locations/$LOC/publishers/google/models/$MODEL:generateContent" \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"Explain Cloud Run in one sentence."}]}]}'

# pip install google-genai pydantic
# Model ids are configuration-driven — never hard-code them permanently.
import os
from pydantic import BaseModel
from google import genai
from google.genai import types

MODEL = os.getenv("GENAI_MODEL", "gemini-2.0-flash")

# Vertex AI mode uses ADC (no API keys) — set these env vars in Cloud Run:
#   GOOGLE_GENAI_USE_VERTEXAI=true, GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION
client = genai.Client()

# 1) Simple + streaming generation
def stream_answer(question: str):
    for chunk in client.models.generate_content_stream(
        model=MODEL,
        contents=question,
        config=types.GenerateContentConfig(temperature=0.2, max_output_tokens=1024),
    ):
        yield chunk.text

# 2) Structured JSON output enforced by a schema
class Answer(BaseModel):
    summary: str
    confidence: float

def structured(question: str) -> Answer:
    resp = client.models.generate_content(
        model=MODEL, contents=question,
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_schema=Answer,
        ),
    )
    # Token accounting for cost tracking
    print("tokens:", resp.usage_metadata.total_token_count)
    return Answer.model_validate_json(resp.text)

# 3) Function / tool calling
def get_weather(city: str) -> str:
    return f"Sunny in {city}"

def with_tools(question: str) -> str:
    resp = client.models.generate_content(
        model=MODEL, contents=question,
        config=types.GenerateContentConfig(tools=[get_weather]),  # SDK auto-invokes
    )
    return resp.text

# 4) Embeddings for RAG
def embed(texts: list[str]) -> list[list[float]]:
    r = client.models.embed_content(
        model=os.getenv("EMBED_MODEL", "text-embedding-005"), contents=texts)
    return [e.values for e in r.embeddings]

# See the gcloud tab for a full curl example; the endpoint pattern is:
# POST https://{LOCATION}-aiplatform.googleapis.com/v1/projects/{PROJECT}/locations/{LOCATION}/publishers/google/models/{MODEL}:generateContent

resource "google_project_service" "vertex" {
  service            = "aiplatform.googleapis.com"
  disable_on_destroy = false
}

resource "google_project_iam_member" "vertex_user" {
  project = var.project_id
  role    = "roles/aiplatform.user"
  member  = "serviceAccount:${google_service_account.rag_api.email}"
}

Troubleshooting

• 403 → runtime SA missing roles/aiplatform.user
• 429 → hitting per-model quota; request an increase or add backoff/routing
• Empty/blocked output → check finish_reason and safety ratings; adjust prompt or thresholds
• 404 model → wrong model id/version/region; verify against Model Garden and your registry
• High latency → stream responses, trim context, and cache repeated prompts

Production considerations

What interviewers expect you to know

Common mistakes

• Hard-coding model ids so the app breaks when a version is deprecated
• Using API keys instead of ADC/Workload Identity
• Ignoring usage_metadata, so token cost is invisible until the bill arrives
• Not handling blocked/truncated responses (finish_reason)
• Sending huge contexts when retrieval would be cheaper and more accurate
• No fallback model or backoff, so a 429 spike takes the feature down
• Logging full prompts/outputs with PII and no redaction
• Leaving Vector Search index endpoints deployed (continuous billing)

Comparisons

Knowledge check