First stream up with AI video. Now to work on configuration and automation!

Example 2

24 Frames @ 12 FPS -> 48 Frames @ 24 FPS (x2)

Before FILM

After FILM

Example

24 Frames @ 8 FPS -> 72 Frames @ 24 FPS (x3)

Before FILM

After FILM

Hugging face space - https://huggingface.co/spaces/fffiloni/ControlVideo

Here are some huggingface spaces to test it:

https://huggingface.co/spaces/fffiloni/video_frame_interpolation https://huggingface.co/spaces/jbilcke-hf/video_frame_interpolation

this is also in the default install of deforum on auto1111

Database here - https://huggingface.co/spaces/jbilcke-hf/media-server/raw/main/database.json

Example -

{
  "version": 1,
  "startAtShotId": "",
  "sequences": [
    {
      "sequenceId": "d344637d-247f-4dd1-8700-02741f36a94a",
      "skip": false,
      "lastGenerationAt": "",
      "videoPrompt": "3D animation, blender renderings, simulations showcasing a variety of different effects",
      "audioPrompt": "electronic goal music from the 90s",
      "tags": [
        "trailer",
        "cgi"
      ],
      "channel": "main",
      "shots": [
        {
          "shotId": "fa3b74d1-f27f-4fc4-a5f3-f29dd0d2750e",
          "index": 0,
          "lastGenerationAt": "",
          "videoPrompt": "stunning 3D video of rippling water waves, reflected on hovering metal beads in an underwater setting. Rendered in Blender with subtle, whaling ambient light, captured in 8K UHD.",
          "audioPrompt": ""
        },
        {
          "shotId": "f89e600c-91e8-401d-a870-3286b779e5a3",
          "index": 1,
          "lastGenerationAt": "",
          "videoPrompt": "high-quality 3D animation of a cloth-like structure falling in a windless room, displaying the realistic physics of fabric movement. Rendered in ultra-detailed Blender, shot in 4K Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "333aae1c-db8c-424e-bb05-d7161952beea",
          "index": 2,
          "lastGenerationAt": "",
          "videoPrompt": "splendid video showing the fluid simulation of liquids as they interact with static and moving objects, set in a minimalistic, clean environment. Captured with 4K Cinema4D, showcases Blender render.",
          "audioPrompt": ""
        },
        {
          "shotId": "a19a3e22-9009-449c-a011-6d6cdc660477",
          "index": 3,
          "lastGenerationAt": "",
          "videoPrompt": "detailed 3D animation of cellular automata, using simple elements to create complex patterns in a uniform space. Exhibits award-winning lighting techniques and Blender rendering, shot in Cinema4D studio quality.",
          "audioPrompt": ""
        },
        {
          "shotId": "694cb3a9-7f51-4f7e-86df-7ae56d34e433",
          "index": 4,
          "lastGenerationAt": "",
          "videoPrompt": "fabulous Blender rendering of a high-speed camera capture of metal beads impacting a flat surface, displaying brilliant n-body collision physics. Captured at a ripe evening with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "d59ec76d-4769-44ab-9aea-09362764d35d",
          "index": 5,
          "lastGenerationAt": "",
          "videoPrompt": "breathtakingly crisp 3D animation featuring a fluid mechanism, where treacle-thick liquid pours into a large vat. Beautifully lit by diffused, soft light. Rendered in Blender, captured at 8K UHD with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "e170953f-4913-428d-a551-53128920e4c8",
          "index": 6,
          "lastGenerationAt": "",
          "videoPrompt": "surreal yet hyper-realistic 3D animation of cloth simulation, showcasing the realism of Blender's material simulation capabilities. With studio lighting illuminating the scene, shot at close-up with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "1dcca91d-c9a4-4ed2-b011-1a55d990ca46",
          "index": 7,
          "lastGenerationAt": "",
          "videoPrompt": "contemporary fine-art 3D animation showcasing the stunning interaction between thousands of metal beads under n-body simulation. Rendered with precision in Blender and captured at a high detail with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "a9296116-b5e5-43ef-914e-d75c7ad2750e",
          "index": 8,
          "lastGenerationAt": "",
          "videoPrompt": "Blender-rendered 3D animation of a glass cube falling into a pool of liquid, captured to showcase fluid dynamics and refraction of light effects. Caught under a soft diffused light with Cinema4D.",
          "audioPrompt": ""
        },
        {
          "shotId": "6c8e7622-268e-491a-a6bc-0ad5b927c7c0",
          "index": 9,
          "lastGenerationAt": "",
          "videoPrompt": "impressive 3D animation, of cellular automata, forming fascinating patterns in a confined environment. The geometrically exact elements are rendered with Blender. Shot at dusk, in crisp 8K UHD with Cinema4D.",
          "audioPrompt": ""
        }
      ]
    },
{

Some notes from here: https://huggingface.co/spaces/jbilcke-hf/AI-WebTV/discussions/8

The main code of the webtv is located inside the media-server :

manual steps:

• human input to write a short paragraph describing a multi-shot video sequence
• manual submit it to GPT-4 to generate a list of video captions for each shot (the system instructions are extracts from a stable diffusion guide)
• commit the captions to the playlist database

Inside the media-server space (generation process running in the background):

• for each prompt in the database
• generate a silent 3 seconds video clip with Zeroscope V2 576w (hosted on Hugging Face Spaces)
• upscale the clip with Zeroscope V2 XL (also a HF Space)
• perform frame interpolation with FILM (also a HF Space)
• storage in the Persistent Storage of the media-server Space

Inside the media-server space (streaming process running in the foreground):

• for each video file in the persistent storage folder
• add it to a new FFmpeg playlist (it’s just a .txt file)
• broadcast it over the RTMP protocol using FFmpeg (in FLV format)
• diffusion of the stream using node-media-server

Inside the AI-WebTV space:

• display the stream using mpegts.js
• this doesn’t work on iPhone, but now there is also a Twitch mirror

Do you see your instance on https://lemmy.world/instances or other servers /instances? That’s typically a good sign that your server is federated.

A new instance doesn’t federate on its own. It needs to discover the other servers. Use something like https://lemmyverse.net/ to make links with your home server and visit the other instances. Subscribe to a few and you’ll start getting those posts.

No worries! All tests welcome in this thread. Do you think it’s your server host? What’s the output of the “date” command?

This is the way.

That makes sense!

That’s very strange. Once a post gets federated, can a user not delete it? Can it only be removed with a purge from the instance admin?

It’s interesting that your post says “in 1 hour” as the timestamp. What would be causing that?

Still seeing it on my end.

I see 5 hours ago. That’s interesting that it would put a timestamp before the thread was posted.

Thank you everyone! Amazing to see this working with all the other instances. Honestly surprised I didn’t have to do too much config after using this post as a rough guide. Just took time for everything to start downloading properly, especially with the chaos of the July 1st exodus.

Is there any other way??

Thank you!

test comment