What is reference to video?

Reference to video works by letting the model read your pictures while it generates, without placing any of them in the clip. Each picture says what something looks like; the prompt says what happens. The model then generates every frame from noise, as in text to video, holding the faces, shapes and colors it was shown.

That is the difference from image to video, where your picture is the first frame and the camera starts exactly there. With references, the opening is the model’s choice, so a close-up, a wide shot and a reverse angle can all show the same person. It is the most common way to get a consistent character across the shots of a film.

Under the hood: where the reference pictures go

The open VACE model, which names the task R2V, documents one design. Each reference picture is compressed by the same encoder as the video and placed in front of the clip’s frames, marked as a frame to keep. The model generates the clip beside them, and the reference frames are cut away after decoding.

Because the references sit in the same sequence as the frames, the attention that keeps frame 10 agreeing with frame 11 also keeps both agreeing with your picture. Nothing in the method copies the picture’s pose or background: the model learns from training pairs which parts to keep.

The Phantom paper names the balance the task depends on: text and pictures steer together. When the pictures weigh too much, their content leaks in, such as the pose, the background or the light of the photo; with several subjects, the model can mix them up.

How to write a prompt for reference to video

• Name each reference and cite it where it acts: “@mia lifts @lamp from the shelf” tells the model which picture plays which part.
• Describe the action, the camera and the light, not the subject’s looks: the picture already gives those, and a second description can fight it.
• Give each picture one job. A photo of a woman holding a lamp is two subjects in one picture; crop it, or the model may keep them together.
• Say where the shot takes place even with a location reference, so the model knows it is the setting and not a backdrop to paste.

All three take pictures, and they use them in different ways. The question is whether your picture should be a frame of the clip or a description of something in it.

Which should I use? If the clip must open on a frame you have, use image to video. If you need the same face or product in shots that each start somewhere new, use references: the camera can start anywhere.

A reference is only as useful as what it shows clearly. DreamBench asks you to tag every picture as one of four kinds, because each one tells the model something different:

A product counts as a prop. A logo too, though its letters are the detail models keep least well.

References steer edits as well as new shots. In this Morph edit, the words cite three cats’ character sheets, a living room and its floor plan by name, and the clip keeps its dance while the cats and the room change.

Original

With Morph

Change the background to @livingroom, laid out as in @floorplan, but swap the 3 cats for @yickshuncat, @oreo and @mango.

This is video to video with references rather than reference to video, since the motion comes from a real clip. The references do the same job in both: they say who and what appears, and nothing else.

Each model maker names the feature its own way, and each sets its own limit on pictures. What their official pages said when we read them:

• Short films and series in which the same characters appear in every scene.
• Product ads: one product photo, many shots of it in use, its label kept.
• Mascots and brand characters that must look the same in every post.
• Putting a pet, a toy or a drawn character into new scenes.
• Keeping one location, a shop or a flat, through all the shots that happen there.
• Storyboards that move, with the cast already looking as cast.

• Copy and paste: the reference’s pose, background or light comes through whole, so every shot looks like the photo. A plain background and even light help.
• Mixed-up subjects: two people of similar build and coloring can trade faces or clothes. Make them different, and cite each by name where it acts.
• Fine detail: small text on a label, a logo’s letters and patterned fabric are kept less well than shapes and colors.
• Unseen sides: a reference of a face from the front leaves the back of the head to the model. A sheet with several views gives it more to keep.
• Too many pictures: each model takes only so many on a clip, and past that some are left out.
• Drift over time: a long or busy shot can still lose a detail, a failure of temporal consistency.

Why doesn’t my character look like my reference? Usually the picture shows too much at once: a small face, harsh light, other people or a busy background. Use a sharp picture of the subject alone, keep the prompt from describing their looks again, and keep the shot short.

A reference of a real person can put that person in a scene they were never in. Use your own face, or someone’s who has said yes to this use, and label a video that could be taken for real footage.

• DreamBench’s Terms ask for informed permission before a real person’s likeness or voice is depicted with AI, and prohibit unauthorized real-person depictions, impersonation and false endorsements.
• Every uploaded picture is checked before any credits are used: nudity, gore and self-harm are refused, and so is a photorealistic person who appears to be under 18.
• Seedance doesn’t accept real human faces. A reference that shows one is refused, and DreamBench says so and suggests another video model for those scenes.
• Model makers set rules of their own: MiniMax’s subject reference, for one, takes a single picture of a person’s face.

Every video model on DreamBench uses your references. On four of them, the pictures go to the video model with each clip. On the rest, an image model first draws the clip’s opening still from them, and the video model animates that still:

When a clip’s model takes fewer pictures than its scene has, your own uploads are kept first. Seedance 2.5 takes the most; Veo 3.1 the fewest, so a scene with many people and props fits better on Seedance, if none of them is a real face.

In the composer you describe the film and add up to 8 reference pictures. Each gets a name and a tag: character, creature, location or prop. The name becomes its handle, so the brief can cite it, as in “@mia opens the box and lifts out @lamp”; typing @ in the brief lists the handles.

DreamBench then draws a reference sheet for every character, place and prop the film needs. A sheet made from your picture is drawn of the exact subject it shows, and a picture whose name says it is a sheet is used as it is.

• A character sheet: the face close up, the whole body from the front and back, and a three-quarter view, the same face in every panel, on plain gray.
• A prop sheet: the object from the front and at three-quarters.
• A location sheet: the place alone, empty of people and props, so none of them is copied from it.

Each clip carries the sheets of what is in it, and DreamBench writes the shot prompts that cite them. In Edit Scenes, each scene shows its references: remove one, add another of the film’s sheets or upload your own, up to as many as that scene’s model takes.

You can start from a tool too. After the product photo to video tool makes a clip, Continue as a story opens the composer with your photo as a @product reference and a brief that cites it.

The first 4 reference pictures are in a video’s base price, and each one past them adds the price of the sheet it needs. A sheet you pick from an earlier film adds nothing, since it is used as it is. The composer shows the total in credits before you generate.