GitHub Park

ClipSketch AI: Turn Video Moments into Hand-Drawn Stories

ClipSketch AI is an end-to-end productivity tool designed for video creators, social media managers, and content enthusiasts.

More than just a video player, ClipSketch AI is an AI-driven content creation workspace. It can parse video links from Bilibili and Xiaohongshu, allowing you to pinpoint and label exciting moments with frame-level accuracy. By integrating Google Gemini's latest multimodal large model, it can instantly transform these moments into beautifully illustrated hand-drawn style storyboards and automatically generate viral copy tailored for social media platforms like Xiaohongshu.

Powerful Video Capture

  • Multi-Source Import: Supports parsing shared links from Bilibili and Xiaohongshu (including short links and mixed text posts).

  • HD Playback: Features adaptive layout optimization for both vertical (9:16) and widescreen videos.

  • Precise Control: Supports keyboard shortcuts (Spacebar for play/pause, Arrow keys for frame-by-frame/smart step navigation).

Frame-Level Labeling System

  • Millisecond Precision: Accurately capture every exciting moment.

  • Hotkey Marking: Press the T key to quickly add a label.

  • Data Export: Export timeline labels as a TXT file or export marked frames as a ZIP image pack.

AI Art Studio (Powered by Gemini)

  • Smart Illustration: Uses the gemini-3-pro-image-preview model to combine multiple marked frames into a coherent, cute hand-drawn style storyboard.

  • Social Copy Generation: Leverages gemini-3-pro-preview to automatically generate three distinct styles of promotional copy based on the visual content (Emotional Storytelling, Practical Tutorial, Concise & Punchy).

  • Character Integration: Upload custom characters/avatars for AI to seamlessly blend into the storyboard scene.

  • Cover Generation: Creates high-quality vertical video covers based on selected copy and original footage.

  • Batch Refinement: Supports batch generation and optimization of storyboard frames (configurable to use Batch API for cost savings).

Full-Platform Compatibility

  • Responsive Design: Perfectly adapts to PC widescreens, iPad tablets, and mobile vertical screens.

  • Mobile Optimization: Automatically switches to a vertical split-screen layout on phones for more intuitive operation.

links:

Google Gemini API Key

clipsketch-ai.vercel.app

Visit RanFeng/clipsketch-ai to access the source code and obtain more information.