D-Soft: Research and Implement Image-to-Video application

• Approach methods
• SOTA models
• Model Architecture
• Fine-Tuning
• Applications
• OpenAI API

1. Variational Autoencoder (VAE): Encode images to a compressed size, then decode back to the original size, while learning the distribution of the data

2. Generative Adversarial Network (GAN): They have two parts (the Generator and the Discriminator) that help each other get better. The Generator learns to make data that looks real, and the Discriminator learns to tell the difference between real and fake data.

3. Flow-based Generative Model: Create new data that’s similar to the data they were trained on and then calculate how likely a certain output is

4. Auto-Regressive Model: Model the conditional probability of each pixel given previous pixels. Then use the probability distribution to generate new data

5. Diffusion Model: Systematically and slowly destroy struture in data distribution though an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data.

1. Stable video diffusion (SDM):

2. Laten flow diffusion model (LDM):

3. Cascade diffusion model (CDM):

1. Encoder: Extract features from the input image, reduce the spatial information, and compress the image into a smaller size.

2. Decoder: Upsample the features to the original size, and restore the spatial information.

3. Skip Connections: Connect the encoder and decoder layers to preserve the spatial information.

4. Output Layer: Produce the final segmentation map with the sampe spatial dimension as the input image.

The UNet can take more information in the form of embeddings

• Time embedding: Related to the timestep and noise level

• Context embedding: Control the content of the generated image

1. Download Pretrained Model:

2. Load Pretrained Model

3. Fine Tuning

4. Sampling Function

5. Generation

We can follow this tutorial to fine-tune the model: Colab

• DALL-E: Link

• Image GPT: Link

• Mid Journey: Link

• Gen-2 Gen: Link

• Authencation: Controls on access to API endpoint services and resources

UNET: Convolutional Networks for Biomedical Image Segmentation [Paper]

High Resolution Image Synthesis and Semantic Manipulation with Conditional GANs [Paper]

Stable Video Diffusion [Paper]

Latent Flow Diffusion Models [Paper]

Denoise Diffusion Probabilistic Models [Paper]