Host an Existing Model (Hugging Face / Replicate)

Take an open model (weights from Hugging Face, or a schema from a Replicate model), wrap it in an APIPod FastTaskAPI service, and run it directly on Socaity's GPUs. The model runs in your own container, not proxied through a third party. This page walks the full path: write the service, build the image, push it, and deploy it from the dashboard.

The pattern is the same whether you start from Hugging Face or Replicate: you wrap an existing open model in an APIPod FastTaskAPI service that loads the weights and runs inference, then host that container on a Socaity GPU. Nothing is proxied. The model executes inside your image.

• From Hugging Face. Pull weights with transformers (LLMs) or diffusers (image models) by model id. This is the most direct route for an open model.
• From Replicate. A Replicate model publishes an openapi_schema. Mirror that input schema 1:1 on your /predictions endpoint, then fill the inference from the underlying open weights (almost always the same model on Hugging Face).

socaity build runs docker build, so you need a working Docker install before you deploy, not only the Python package.

• Docker. Install Docker Desktop (or Docker Engine) and confirm the daemon is running (docker info succeeds). Without it, socaity build fails immediately.
• Disk space. The CUDA base image plus model layers is large, so budget ≥ 30 GB free. Docker stores images on its own disk; on Docker Desktop that defaults to the system drive. For big ML images, move it under Settings → Resources → Advanced → Disk image location to a roomy volume.
• Architecture (Apple Silicon). RunPod GPUs run linux/amd64. On an Apple-Silicon (arm64) Mac you must cross-build: docker build --platform linux/amd64 …. A native arm64 image will not run on the GPU. The cross-build uses emulation and is slow.
• Hugging Face access. Gated weights (e.g. Gemma, Llama) require accepting the license on the model page and setting an HF_TOKEN environment variable at deploy time. See Secrets.
• Docker Hub. A Docker Hub account (or any registry Socaity can pull from), and a docker login before you push.

Create a FastTaskAPI service with a single @app.task_endpoint(path="/predictions"). Load the weights from Hugging Face: transformers for LLMs, diffusers for image models. Build the pipeline once at container start (module scope) so the cold-start cost is paid per container, not per request.

# main.py
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from fast_task_api import FastTaskAPI

# Pass ONLY title/summary — see "Pin the framework" below.
app = FastTaskAPI(title="my-llm", summary="Self-hosted open LLM")

# Load weights once at container start (module scope), not per request.
MODEL_ID = "google/gemma-2-9b-it"  # any open / gated Hugging Face model id
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=os.getenv("HF_TOKEN"))
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    token=os.getenv("HF_TOKEN"),
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

@app.task_endpoint(path="/predictions")
def predict(prompt: str, max_new_tokens: int = 256) -> dict:
    """Mirror the upstream model's input schema 1:1."""
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=max_new_tokens)
    text = tokenizer.decode(out[0], skip_special_tokens=True)
    return {"output": text}

if __name__ == "__main__":
    app.start()

Pin fast-task-api in pyproject.toml and pass only title and summary to the FastTaskAPI(...) constructor.

[project]
name = "my-llm"
version = "0.1.0"
dependencies = [
    # Pin below 1.3 — 1.3.x forwards extra constructor kwargs to
    # object.__init__ and crashes the container at startup.
    "fast-task-api>=1.2.9,<1.3",
    "transformers",
    "torch",
    "accelerate",
]

Scan the project, then build. socaity scan writes apipod-deploy/apipod.json with the detected framework stack; socaity build renders a Dockerfile and runs docker build. The default entrypoint is main.py.

# 1. Detect the framework stack -> apipod-deploy/apipod.json
socaity scan

# 2. Render the Dockerfile and run docker build.
#    Default entrypoint is main.py.
socaity build

# On an arm64 Mac, cross-build for the GPU architecture:
docker build --platform linux/amd64 -t your-namespace/your-image:v1 .

Authenticate to your registry, then push the tagged image. There is no apipod push command, so use the standard Docker tooling.

# Authenticate first — push fails without it.
docker login

# Push the tagged image to your registry.
docker push your-namespace/your-image:v1

Open the socaity.ai dashboard and start the Deploy a Service flow:

1. Source: choose Docker image and paste the fully-qualified image reference (your-namespace/your-image:tag). Add Docker Hub credentials for private images.
2. GPU: pick the smallest GPU that fits the model in VRAM (see sizing below). Bigger than needed only costs more.
3. Container disk: set it large enough for the weights the container downloads at boot. A multi-billion-parameter model can be tens of GB on disk; undersizing this fails the first boot.
4. Environment: add env vars under Advanced settings (e.g. HF_TOKEN for gated weights). Never bake secrets into the image.
5. Name: the service name must be unique. Deploy.

Size by parameters × bytes-per-parameter, then leave headroom for activations and KV cache. As a rule of thumb the weights should fit in ≤ 80% of VRAM.