Lilycut: the starting requirements

The AI video editor has a name now: Lilycut. If it needs a command-line interface, I’ll call that lily.

I’m giving Lilycut its own project page, while keeping the original proposal in the Ideas backlog. These are the requirements I want to start building from. I haven’t tested the editing pipeline on my footage yet, so this is the plan for what it should do. I’ll update this post as I build it and learn what works.

The original idea came from recording work like stripping and sanding a floor. I could have half an hour of footage or eight hours, and I want to turn that into something people would actually watch without going through the whole recording again myself.

Three versions of the work

I want Lilycut to make three different edits from the same footage:

30 seconds: Where I started, a few good moments of the work, and how it came out.
Two minutes: The main stages in order, with repetitive work compressed and enough context to understand the process.
Ten minutes: More of the technique, explanation, mistakes, decisions, and how I dealt with problems.

Each length needs its own editing decisions. It should choose what belongs in that version and decide what to skip or speed up. I should be able to change the target length, and it should tell me if there isn’t enough useful footage to fill it.

Import the recordings and keep the originals

I should be able to create a project, add one or more recordings, put them in order, and describe what I was doing. MP4 and MOV are the initial input targets, with a clear error when a particular file or codec isn’t supported.

Lilycut should make smaller copies for analysis and preview, keeping the original files unchanged. The finished video should be rendered from those originals. Source timestamps need to stay accurate across files and processing chunks so every selected shot can be traced back to the recording.

The project should save its analysis, markers, edit versions, and export settings. If I move the recordings, I should be able to point it at their new location.

Figure out what happened

The editor needs to recognize stages, visible progress, useful action, and before-and-after moments. For a floor, that might mean preparation, stripping, sanding, cleanup, and the finished room.

It should flag repetition, inactivity, blocked camera views, and shots that probably aren’t useful. It also needs to work when I’m not talking or when a machine drowns out the audio.

The approach I want to test is a broad scan followed by a closer look at promising sections. The result should be a timestamped map of the project that I can inspect and correct. If it’s uncertain about what happened, it should make that uncertainty visible.

Let me help while I’m recording

I want a configurable phrase such as “mark that” to draw attention to something. It should look before the phrase as well as after it. A starting window would be the previous 30 seconds and the following 60 seconds, with a way to adjust that.

Natural commentary should help too. “Switching to a finer grit,” “that didn’t work,” and “here’s the result” all give context for the images. I shouldn’t have to narrate continuously.

The transcript and detected markers need to be editable because speech recognition around machinery will make mistakes. I should also be able to mark a range manually as something to keep or leave out. The marker phrase itself should normally be left out of the finished audio.

Put together a first cut

The default edit should follow the order the work happened. It should combine normal-speed shots with accelerated sections where the work repeats. Important spoken explanations should stay at normal speed.

Every selected shot should have a source range and a short reason for why it was chosen. The story needs to come from the recording. If I didn’t capture the finished result, the editor should tell me that rather than imply it has footage it doesn’t have.

Simple cuts and modest audio fades are enough to start. I want it to make good selections before spending time on effects.

Make corrections easy

I want a preview and a simple sequence of shots with thumbnails, descriptions, and source times. I should be able to trim, remove, reorder, and change the speed of a shot, then undo those changes.

I should be able to ask for revisions such as “less sanding,” “show more of the mistake,” or “keep the reveal.” Shots I lock need to survive those revisions. Each revision should be saved as a separate version so I can go back.

If the shots I require won’t fit the requested length, Lilycut should explain the conflict. It shouldn’t silently throw away something I told it to keep.

Export something usable

The first export target is a 1080p landscape MP4 with useful original audio, simple level controls, and fades. Accelerated machinery noise or speech should be muted by default, with a way to review and change that choice.

The export needs valid cut points, the expected duration, and synchronized audio and video. The editable project stays available after the video is rendered.

Keep processing time and cost under control

The analysis should be saved and reused for all three lengths. Asking for another edit shouldn’t mean analyzing eight hours of footage from scratch.

If cloud processing is used, Lilycut should show which media will be uploaded, estimate the cost before starting, and record actual usage where the provider makes that available. I want a configurable budget, with processing paused before more billable work would exceed it and some allowance for estimation uncertainty.

Long jobs need visible progress for preparation, analysis, edit planning, and rendering. I should be able to cancel and resume from checkpoints without losing completed analysis.

Where I’ll start building

The initial direction is a local application that manages files and renders on my computer. A browser interface may be enough for the first version. Cloud and local analysis are both options; I haven’t settled on a model or provider.

The analysis provider should be replaceable. Importing footage, transcribing it, finding useful moments, planning an edit, validating that plan, and rendering it should be separate stages with saved intermediate results.

The edit plan should record each shot’s source file, start and end time, speed, audio settings, position, selection reason, and whether I locked it. The application needs to validate those decisions before it renders anything.

If I add a CLI, lily is the intended short command. Other tools already use that name, so installation should detect an existing command and leave it alone. lilycut can also be available as the full command.

The first milestone

Start with a representative 20–30-minute recording that contains a few stages of a project. Produce a timestamped map and a two-minute edit. Then review what it missed, what it chose badly, how long it took, and what it cost.

Once that works, add the review controls, saved projects, spoken markers, the other output lengths, and reliable recovery for long jobs. Then test four-hour and eight-hour recordings, multiple files, silent footage, loud machinery, and interrupted processing.

I’ll consider the first release useful when:

I can import footage and export a reviewed two-minute video without opening another editor.
The edit represents the major stages that were recorded and uses a before-and-after comparison when the footage supports one.
Every selected segment points to a valid source range, and the audio stays synchronized.
My corrections and locked shots survive regeneration and reopening the project.
Additional output lengths reuse the existing analysis unless a closer inspection is needed.
Interrupted analysis resumes without repeating completed chunks.
Silent or noisy recordings still work, and long jobs finish within the budget or pause with an explanation.

I’ll measure how much hands-on review it takes as well as processing time and cost. That first test should tell me what targets are realistic.

Things that can wait

Portrait and square exports, automatic reframing, captions, music, voiceover, 4K export, local model support if the prototype starts in the cloud, multi-camera synchronization, 360-degree reframing, and export into another editor can come later.

A full professional timeline editor, generated replacement footage, team collaboration, mobile capture, and automatic social publishing are outside the initial scope.

The next update should be what happened when I tried this on real footage. Progress and related posts will be on the Lilycut project page.