Unified-Multimodal Transformer Pipeline for Political Content Creation: TikTok Reel Generator (Under-development)
Welcome to the Unified-Multimodal Transformer for Political Content Generation repository! This powerful tool harnesses the latest advancements in artificial intelligence and machine learning ๐ง to create engaging and high-impact TikTok reels ๐น from political videos. By leveraging the power of transformers and multimodal learning, our tool is able to identify the most interesting and click-worthy segments of a video, automatically edit them, and generate a TikTok reel that is both captivating and informative ๐คฏ.
output_video.mp4
- (Development under progress) Re-train the UMT model https://github.com/TencentARC/UMT Unified-Multi Modal Transformer Model on political content to extract high click potential content โ .
- Implement auto-posting on social channels.
- Add the feature to constantly monitor channels for interview/podcast length videos ๐บ.
- Perform video cropping, automated to 9:16 aspect ๐๏ธ.
- Integrate Speech to Text engine Whisper (English, Urdu, Hindi, etc.) + Integrate ASR model for clear speech cutting ๐ฃ๏ธ.
- Generate --.srt files ๐.
- Chop SRT to < 8 words, 2 lines per frame ๐.
- Animate srt on top of the video in Python ๐.
- Add Instagram style filters. A range of them ๐.
- An interface to download videos to upload on accounts โฌ๏ธ.
- Wrap up the app into Gradio ๐.
This was a project that I did to understand the construction of transformers and a key application of it in automation of political campaigning ๐ฃ. This tool utilizes state-of-the-art Unified-Multimodal Transformers, which are capable of processing and understanding multiple forms of data simultaneously ๐. This allows the model to analyze and comprehend the intricacies of political content, such as speech, text, and visual cues, resulting in a deeper understanding of the video's context and potential impact on viewers ๐ฏ.
One of the key features of our tool is its ability to edit the extracted video segments just like a person would ๐งโ๐ป. This means that the final TikTok reels will have a professional and polished look, making them more appealing to viewers and increasing the chances of your content going viral ๐.
This tool coupled with other technologies available such as LLMs can potentially turn the game for political campaigning by automating spread of content. The training costs for the transformer to identify the segments is too high, so this project has been paused and made public for now, and I plan to build this further a bit later.
- Setup an Azure VM ๐ฅ๏ธ
- Install and setup Docker ๐ณ
- Run the following commands
pip3 install -r requirements.txt
All mediapipe missing files can be found here.
Compiled AutoFlip Docker Image can be found at this link