# LatentSync 1.0

> Re-sync a talking-head video's mouth to a new audio track with audio-conditioned diffusion — the catalog's lowest-priced lip sync at $0.014 per output second.

## Overview

- **Endpoint**: `https://queue.modelrunner.run/bytedance/latentsync`
- **Model ID**: `bytedance/latentsync`
- **Category**: video-to-video
- **Kind**: inference
- **Tags**: latentsync, bytedance, lipsync, lip-sync, video-to-video, dubbing, video-dubbing, talking-head, re-voice, audio-driven, mouth-sync, open-source

## Pricing

- **Price**: $0.014 per output second

## Request Lifecycle

This model runs on the ModelRunner **asynchronous queue API** — a single POST does not return the output. Every call requires an `Authorization: Key $MODEL_RUNNER_KEY` header. Run three steps:

1. **Submit** — `POST https://queue.modelrunner.run/bytedance/latentsync` with a JSON body holding the input fields at the top level. The body may also include a reserved top-level `metadata` object — a flat string map (max 16 keys, key ≤64 / value ≤512 chars) stored on the request for your own tagging. It is never sent to the model; filter your request history with `GET https://queue.modelrunner.run/requests?metadata=<url-encoded JSON>` (exact key=value matches, AND-ed). The response carries request handles only (no output yet):

```json
{
  "status": "IN_QUEUE",
  "request_id": "<21-char id>",
  "status_url": "https://queue.modelrunner.run/bytedance/latentsync/requests/<request_id>/status",
  "response_url": "https://queue.modelrunner.run/bytedance/latentsync/requests/<request_id>",
  "cancel_url": "https://queue.modelrunner.run/bytedance/latentsync/requests/<request_id>/cancel"
}
```

2. **Poll status** — `GET <status_url>` until `status` is `COMPLETED`. Possible values are `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`. A `FAILED` request responds with HTTP 400 and an `error` field.

3. **Read result** — `GET <response_url>`. Returns the finished request, including the generated `output`:

```json
{ "id": "<request_id>", "status": "COMPLETED", "output": ..., "input": ... }
```

The JavaScript and Python SDKs below perform steps 2–3 for you. In any language without an SDK (Swift, Go, Kotlin, etc.) you must implement the polling loop and the final result fetch yourself — see the cURL example for the full flow.

### Input Schema

- **`seed`** (`integer`, _optional_):
  Random seed. 0 (the default) draws a fresh random seed on every run, so repeated calls with identical inputs differ; any positive integer is used as given for a repeatable run.
  - Default: `0`

- **`audio_url`** (`string`, _required_):
  URL of the speech track the speaker should appear to say (MP3, AAC, WAV or M4A). The result runs for the shorter of this track and the video, trimmed down to a whole multiple of 0.64 seconds — supply audio slightly shorter than the video and expect the last fraction of a second to be cut.

- **`video_url`** (`string`, _required_):
  URL of the source video — an MP4 with one clearly visible, front-facing speaker in every frame; the run fails on any frame where no face is detected. The result keeps this video's frame size, and its identity, lighting and background; only the mouth region is regenerated. Video running past the end of the audio is discarded, so trim it to roughly the audio's length.

- **`guidance_scale`** (`number`, _optional_):
  Strength of the audio conditioning during diffusion. The default of 1 leaves classifier-free guidance off; values above 1 switch it on. The model's own demo exposes 1–3.5, though the field accepts up to 10.
  - Default: `1`
  - Range: `0` to `10`

### Output Schema

_No `Output` schema properties are available._

## Default Example

**Input**

```json
{
  "seed": 0,
  "audio_url": "https://media.modelrunner.ai/lZfxwe6ZN6ZikXQhVtiq7.wav",
  "video_url": "https://media.modelrunner.ai/NLaN6i1oQvbd8Mn4oEDXs.mp4",
  "guidance_scale": 1
}
```

**Output**

```json
"https://media.modelrunner.ai/f6p9cpEJKMXJa8LXBhywO.mp4"
```
## Usage Examples

### cURL

The queue API is asynchronous: submit the request, poll `status_url` until it is `COMPLETED`, then read the result from `response_url`. Requires `jq`.

```bash
# 1. Submit the request (returns request handles, not the output)
SUBMIT=$(curl --silent --request POST \
  --url https://queue.modelrunner.run/bytedance/latentsync \
  --header "Authorization: Key $MODEL_RUNNER_KEY" \
  --header "Content-Type: application/json" \
  --data '{
  "seed": 0,
  "audio_url": "https://media.modelrunner.ai/lZfxwe6ZN6ZikXQhVtiq7.wav",
  "video_url": "https://media.modelrunner.ai/NLaN6i1oQvbd8Mn4oEDXs.mp4",
  "guidance_scale": 1
}')

STATUS_URL=$(echo "$SUBMIT" | jq -r '.status_url')
RESPONSE_URL=$(echo "$SUBMIT" | jq -r '.response_url')

# 2. Poll until the request leaves the queue / in-progress state
while true; do
  STATUS=$(curl --silent --url "$STATUS_URL" \
    --header "Authorization: Key $MODEL_RUNNER_KEY" | jq -r '.status')
  echo "Status: $STATUS"
  case "$STATUS" in
    COMPLETED) break ;;
    FAILED|CANCELLED) echo "Request $STATUS"; exit 1 ;;
  esac
  sleep 1
done

# 3. Read the finished request, including the generated output
curl --silent --url "$RESPONSE_URL" \
  --header "Authorization: Key $MODEL_RUNNER_KEY"
```

### JavaScript

```javascript
import { modelrunner } from "@modelrunner/client";

const result = await modelrunner.subscribe("bytedance/latentsync", {
  input: {
  "seed": 0,
  "audio_url": "https://media.modelrunner.ai/lZfxwe6ZN6ZikXQhVtiq7.wav",
  "video_url": "https://media.modelrunner.ai/NLaN6i1oQvbd8Mn4oEDXs.mp4",
  "guidance_scale": 1
}
});

console.log(result.data);
```

### Python

```python
import asyncio
import modelrunner_ai

async def main():
    response = await modelrunner_ai.submit_async(
        "bytedance/latentsync",
        arguments={
  "seed": 0,
  "audio_url": "https://media.modelrunner.ai/lZfxwe6ZN6ZikXQhVtiq7.wav",
  "video_url": "https://media.modelrunner.ai/NLaN6i1oQvbd8Mn4oEDXs.mp4",
  "guidance_scale": 1
}
    )
    result = await response.get()
    print(result["output"])

asyncio.run(main())
```

## Additional Resources

- [Playground](https://modelrunner.ai/models/bytedance/latentsync)
- [OpenAPI Schema](https://modelrunner.ai/models/bytedance/latentsync/openapi.json)
- [LLM Instructions](https://modelrunner.ai/models/bytedance/latentsync/llms.txt)
- [GitHub](https://github.com/bytedance/LatentSync)
- [License](https://huggingface.co/chunyu-li/LatentSync)
- [Weights](https://huggingface.co/chunyu-li/LatentSync)
- [Paper](https://arxiv.org/abs/2412.09262)