E-Commerce AI Video Generation Benchmark

Product visualization plays a crucial role in e-commerce success, yet creating high-quality product videos remains a significant challenge. Recent advancements in AI video generation technology offer promising solutions.

We evaluated leading AI platforms’ capabilities in generating product demonstration videos:

Benchmark results

Methodology

Products used

• OpenAI Sora
• Kling AI KLING 1.5
• Runway Gen3 Alpha Turbo
• Hailuo AI I2V-01-live

All tests were performed in December/2024.

Test Image Classification and Objectives

Our study utilized three distinct categories of product images, each designed to test specific capabilities of AI video generators:

White Background Products

Purpose: Evaluate dual capabilities

1. Basic manipulation: Product movement and rotation in a neutral setting

2. Environmental adaptation: Integration of products into new contexts

Test focus: AI’s ability to maintain product integrity while adding or changing environments.

Contextual Product Images

Purpose: Assess environmental animation capabilities

1. Scene-to-video conversion accuracy

2. Maintenance of existing lighting and atmosphere

3. Adding dynamic elements to established setting

Test focus: AI’s ability to bring static environmental product shots to life.

Multi-Product Scenes

Purpose: Test complex product relationships and interactions

1. Inter-product physical interactions

2. Consistent scale maintenance

3. Group movement dynamics

4. Collective lighting effects

Test focus: AI’s ability to handle multiple products while maintaining individual integrity and natural interactions.

This three-category approach enables us to evaluate not only individual product rendering and environment creation, but also the AI’s capability to manage complex multi-product scenarios, providing a more complete assessment of real-world e-commerce applications.

Our evaluation metrics are:

Prompt Compliance: (3 points)

• Consistency between prompt requirements and generated output for the product

• Consistency between prompt requirements and generated output for the environment

• Consistency between prompt requirements and generated output for the camera and shooting.

Physical Accuracy: (3 points)

• Adherence to real-world physics

• Accuracy of object interactions (surface contact, movement)

• Lighting and shadow behavior

Product Integrity: (4 points)

• Consistency in product appearance throughout the video

• Preservation of product / brand-specific features and details

• Maintenance of product proportions and scale

• Texture, color and material rendering accuracy

Each generated video is rated out of 10 based on these metrics.

Dataset: We used stock images from pexels.

An example

Image:

Prompt: Pure white background, soft studio lighting. Smooth 360-degree rotation, starting and ending with front view. Keep bag centered and maintain consistent rotation speed.

Output of OpenAI Sora:

This video is rated as 6/10 due to these issues:

• Prompt compliance: It failed to demonstrate consistency between prompt requirements and the generated output in terms of product appearance, environment rendering, and camera movements. (-3 points)
• Preservation of product / brand-specific features: The side clips and the rings on the front are distorted as the point of view is rotated. (-1 point)

What are the issues of AI video generators?

We tried these video production tools to promote a product on e-commerce sites using only its photograph and a prompt, but the outputs showed us that this was not possible.

In most cases, these AI tools could not:

• Accurately convey the product’s features, brand-specific details, size, color, texture, etc., to the buyer.
• Generate a video that is 100% compatible to the prompt.

Tips: To address these issues, we recommend enhancing prompts and contextualizing AI video generators through LLM fine-tuning, contextual RAG, or Agentic RAG.

FAQ

Further reading

Discover more on generative AI capabilities, use cases and tools by checking out:

External sources