AI video editing
What is video inpainting?
Video inpainting is filling a chosen region of a video with new content that fits the rest of every frame. It is how AI removes an object, a person, a logo or a caption from a clip: the hole is filled with the background that was probably there, steady as the camera and scene move.
By the DreamBench team. Updated .
Video inpainting at a glance
At a glance
- Input
- A video and the region to fill: drawn as a mask, or named in words.
- Output
- The video with that region replaced, frame by frame.
- Used for
- Removing objects, people, wires, logos and captions; repairing damaged footage.
- Easiest
- Small objects, a steady camera, a simple background that is visible at some point.
- Hardest
- Large areas, busy backgrounds, and things that are never seen behind the object.
How video inpainting works
AI removes an object from a video in two steps. First, a segmentation model such as SAM 2 outlines the object once and follows it through every frame. Then the region is filled: real background copied from frames where it was visible, and generated where it never was, so the fill moves with the camera.
Source video
The clip you upload: its motion, timing and camera stay the backbone.
Find the region
The model locates what you named and follows it through every frame.
Generate the change
New pixels for that region only: a new face, an added object, an empty background.
Blend over time
Light, shadow and motion are matched so the change holds from frame to frame.
Edited video
Everything you didn’t name is meant to look as it did.
Mask or instruction
There are two ways to say what should go. In classic visual-effects tools you mark it: draw a mask on one frame and let a tracker follow it through the rest. In newer generative editors you name it in words, “the cyclist on the left”, and the model finds it in every frame.
A mask is exact but takes work, and a tracker that slips leaves a trace. Words are quick, but the model decides where the edges are, so check them.
Borrowing from other frames
Unlike a photo, a video often shows what is behind an object, a few frames earlier or later, when the object or the camera has moved. Flow-guided methods such as ProPainter track how every pixel moves between frames and copy the real background into the hole wherever it was seen.
Generating what was never seen
Where the background was never visible, the model has to invent it, the way image inpainting does for a still. Generative video models do this for the whole clip at once, so the invented part moves with the camera and keeps its texture.
What video inpainting is used for
- Removing passersby, cars or crew from a shot that can’t be filmed again.
- Cleaning up wires, rigs, boom microphones and markers in visual effects.
- Taking out a logo or sign you don’t have the rights to show.
- Repairing scratches, dust and damaged areas in old film.
- Making room in a scene before something new is added to it.
- Obscuring faces or license plates for privacy, by replacing them rather than blurring.
Video inpainting and related terms
| What it does | Result | |
|---|---|---|
| Video inpainting | Fills a region inside the frame | Something removed or replaced |
| Video outpainting | Extends the frame beyond its edges | A wider or taller video |
| Rotoscoping | Outlines a subject through every frame | A mask to cut it out |
| Background removal | Separates the subject from its background | The subject on transparency |
Before you remove something from a video
Most failed removals could have been predicted from the clip. Check these before you start:
| Good sign | Warning sign | |
|---|---|---|
| Size | The object covers a small part of the frame | It fills a third of the frame or more |
| Background | The background behind it is seen at some point in the clip | It is never uncovered, so it has to be invented |
| Movement | The object or the camera moves, revealing what is behind | Both stay still, so no frame shows the background |
| Texture | Sky, walls, grass, water at a distance | Faces, text, crowds, fine patterns |
| Traces | No shadow or reflection | A shadow, reflection or splash that must go too |
Name the traces in your instruction: “remove the cyclist and her shadow”. A model removes what you name, not what you meant.
When video inpainting fails
- Shadows and reflections left behind: remove the person, keep their shadow. Include them in what you ask to remove.
- A blur or smear where the object was, on large regions with nothing to borrow from.
- Flicker in the filled area, when each frame is filled slightly differently: a temporal consistency failure.
- Lost detail on moving cameras and busy textures such as foliage, crowds and water.
A removal that looks right on one frame and wrong on the next is the usual failure. Either the fill is regenerated a little differently each frame and flickers, or the tracker loses the object when something passes in front of it and the object comes back for a few frames: SAM 2’s authors note that it can lose track after a long occlusion. A shorter clip, a steadier camera and a region that stays clear of other moving things give the tracker less to lose.
Don’t remove watermarks, credits or captions you don’t have the rights to remove.
Removing something from a video on DreamBench
In Morph, choose Remove something, upload a clip of 4 to 30 seconds and say what should go. Morph fills the area and keeps the rest of the clip and its motion. For a still picture, the object remover does the same job.
| Figure | What it means for your clip | |
|---|---|---|
| Clip | 4 to 30 seconds in one edit | A longer video is edited in pieces. A clip with real people in it has a shorter limit, and Morph says so when the part you picked is too long. |
| How you say it | In words, with no mask to draw | Name the object, its place, and its shadow and reflection: the model finds it in every frame. |
| Sound | Kept from the clip or generated to match the edit | You choose when you set up the edit; kept, it is put back under the edited picture. |
| Price | Credits from your plan | As for every Morph edit. |
| For a photo | The AI object remover | The same job on a still, from 7 credits. |
Questions
Can AI remove a person from a video?
Yes, AI can remove a person from a video with video inpainting. It works best when the person is small in the frame, the camera is steady, and the background behind them is visible at some point in the clip. Name their shadow and reflection too, or they stay behind.
What is the difference between inpainting and outpainting?
Inpainting fills a region inside the frame, for example where an object was. Outpainting adds new content outside the frame’s edges to make it wider or taller.
What is the difference between image and video inpainting?
Video inpainting has to fill the same area in every frame so that it holds still as the clip plays. It can also borrow the real background from frames where it was visible, which a single photo can’t.
Why is there a blur or a ghost where I removed something?
A blur means the model had too little to go on: a large region, a background it never saw, or fast motion. Try a shorter clip, a smaller region, or a shot where the background is visible. A faint outline that comes and goes is the tracker slipping or the shadow and reflection left behind; name them in the instruction.
Does removing an object change the rest of the video?
It shouldn’t: a good removal touches only the region you named. Generative editors can shift small details nearby, so compare the result with the original before you use it.
Can AI remove a watermark or caption from a video?
Technically it is the same job as any removal, and the result is cleanest on a steady shot with a plain background behind the mark. Only remove marks you have the rights to remove: a platform’s watermark on your own clip is yours to take off; a stock or creator watermark is not.
Can video inpainting remove the background from a video?
No. Inpainting fills a region with background; removing the background is the opposite job, separating the subject on every frame, which is rotoscoping or video background removal. DreamBench’s background remover works on photos.
How long a clip can I remove something from?
On DreamBench, Morph edits 4 to 30 seconds of a clip in one run; a longer video is cut into parts and each is edited on its own. A clip that shows real people is edited on a lane with a shorter limit, and Morph tells you when the part you picked is too long. Most generative editors work on seconds, not minutes, because every frame of the edit is generated together.
Does removing an object from a video keep the sound?
Yes, if you want it to. On DreamBench you choose whether to keep the clip’s own sound when you set up the edit; kept, it is put back under the edited picture, and otherwise Morph generates sound to match the edit. The removal itself touches only the picture, so a voice or music that was in the clip is still there when you keep it.
Do I need to draw a mask to remove something from a video?
Not in Morph: you name what should go, “the cyclist on the left and her shadow”, and the model finds it in every frame. Classic visual-effects tools work the other way, with a mask drawn on one frame and tracked through the rest, which is exact but takes work. Words are quicker, and the model decides where the edges are, so check them in the result.
Related
DreamBench tools
Terms
- AI video editingChanging what is in a real video with a generative model, while its motion and timing stay.
- InpaintingFilling a chosen area of a picture with new content, the way AI removes objects and text.
- Temporal consistencyHow well an AI video keeps things the same from frame to frame, without flicker or drift.
- OutpaintingExtending a picture past its edges with new content that matches the scene.
- Background removalSeparating a photo’s subject from what is behind it, with segmentation or matting models.
Sources
- SAM 2: Segment Anything in Images and VideosarXiv, read on
- ProPainter: Improving Propagation and Transformer for Video InpaintingarXiv, read on
DreamBench is an independent platform and isn’t affiliated with or endorsed by the companies named on this page.