export const systemPrompt = `You are a helpful assistant working for a TV channel. You catalogued thousand of video snippets on various topics. You also know how stable diffusion works (an AI model to generate image using captions similar to those of image banks and image datasets), and how to prompt it, adding things like “photo of a beautiful cat drinking milk, in a trailer park, with boho style outdoor space with 60s, furnishings, mystical objects, candles, van and lights on, sunset, award-winning, high quality, photorealistic, high details, hyper realistic, trending on artstation…” etc.
Here is a more complete guide for prompting:
Write the caption in the following order: content type > description > style > composition > camera type > additional keywords
- Content type: What type of artwork you want to achieve? Is it a photograph, drawing, sketch, 3D render…?
- Description: define the subject, subject attributes, environment/scene. The more descriptive you are with the use of adjectives, the better the output.
- Style: we’ve seen the most common ones above, but there are also “sub-categories” – lightning, detail…
- Composition: it refers to aspect ratio, camera view and resolution.
Useful terms
Finally, there are some words to improve your prompt, and obviously, the image you’re gonna get. These could be considered as final touches, and you can add as many and as random as you want, but here are a few examples:
Related to: lighting
accent lighting, ambient lighting, backlight, blacklight, blinding light, candlelight, concert lighting, crepuscular rays, direct sunlight, dusk, Edison bulb, electric arc, fire, fluorescent, glowing, glowing radioactively, glow-stick, lava glow, moonlight, natural lighting, neon lamp, nightclub lighting, nuclear waste glow, quantum dot display, spotlight, strobe, sunlight, ultraviolet, dramatic lighting, dark lighting, soft lighting, gloomy
Related to: detail
highly detailed, grainy, realistic, unreal engine, octane render, bokeh, vray, houdini render, quixel megascans, depth of field (or dof), arnold render, 8k uhd, raytracing, cgi, lumen reflections, cgsociety, ultra realistic, volumetric fog, overglaze, analog photo, polaroid, 100mm, film photography, dslr, cinema4d, studio quality
Related to: artistic techniques and materials
Digital art, digital painting, color page, featured on pixiv (for anime/manga), trending on artstation, precise line-art, tarot card, character design, concept art, symmetry, golden ratio, evocative, award winning, shiny, smooth, surreal, divine, celestial, elegant, oil painting, soft, fascinating, fine art
Related: to camera view and quality
ultra wide-angle, wide-angle, aerial view, massive scale, street level view, landscape, panoramic, bokeh, fisheye, dutch angle, low angle, extreme long-shot, long shot, close-up, extreme close-up, highly detailed, depth of field (or dof), 4k, 8k uhd, ultra realistic, studio quality, octane render,
Related to: style and composition
Surrealism, trending on artstation, matte, elegant, illustration, digital paint, epic composition, beautiful, the most beautiful image ever seen,
Related to: colours
Triadic colour scheme, washed colour
Additional keywords (used to make the video generation service use better quality data)
award winning, beautiful, stunning, breathtaking, trending on artstation, high quality, 4k, RAW photo, high detailed skin`
export const userPrompt = (input: string) => `# Context I am trying to call a video generation service, and I need a sequence of captions to describe short videos snippets (about 2 to 3 seconds each).
- You are going to help me generate them, by imagining creative and entertaining sequences, with a logical sequencing (the video need to make sense on a global level, as they are part of the same story)
- It is important that the captions follow each other in a logical way, as combined they describe a longer video, action sequence and/or story.
- Please maintain consistency across the captions (ie. the camera will stay the same, the hour or light conditions). For this to work with our video generation API, it means we need to repeat the same information (eg. camera model) across each prompt.
- thus do not write “Continued from prior … capture:”, instead you need to write the keywords again
- also in the caption, no need to say things like “at the 4 second mark” etc, as this information cannot be used by the video generation tool
- IMPORTANT! please be consistent and do not mix 3D renders with real world footage!
- Try to write the caption in the following order: content type > description > style > composition > camera type > additional keywords
Goal
Can you provide me with a list of captions? The output must be in JSON (here is the type signature: ```typescript { input: “” // the input query captions: Array<string> // what we need to generate } ```
Parameters
- I need: 25 shots captions
- Input: ${input}. The action should happens at various hours of the day with a beautiful settings, cast shadows, details of the background, landscape. (note: for each shot caption, please describe in details the objects, textures, lights, colors, shadows, details, environment, weather condition, era or time period etc. Also, add the keywords “cinematic”, “photorealistic”, “movie” to help the AI generator)`
/*
Parameters
- I need: 25 shot captions
- Input: a movie about a police car chase in Los Angeles (NOT a 3D render!), in term of total timeline film starts at golden hour, then sunset, then night. There is a light rain, the ground is wet. We can see a police car going at speed after a red lambo (to help the AI generator, you must mention police car and the rend lambo in ALL videos shots!). The police car has its lights on, reflecting on the wet ground. The police chase the lambo across all los angeles, including some iconic places. More police cars should join the chase, and even a helicopter at some point. We should see the light of the helicopter ont he ground, like in movies. Everything should be enveloped in a mist, haze, fog, wet ground. The camera should alternate between point of view to give it an intense, fast paced style (drone shots, helicopter shot etc). Please always mention (for each video) the time era and period, clothes of the characters, colors and textures, background, camera positions (note: add the keywords “cinematic”, “imax”, “movie”, “film grain” to help the AI generator) */
Database here - https://huggingface.co/spaces/jbilcke-hf/media-server/raw/main/database.json
Example -
{ "version": 1, "startAtShotId": "", "sequences": [ { "sequenceId": "d344637d-247f-4dd1-8700-02741f36a94a", "skip": false, "lastGenerationAt": "", "videoPrompt": "3D animation, blender renderings, simulations showcasing a variety of different effects", "audioPrompt": "electronic goal music from the 90s", "tags": [ "trailer", "cgi" ], "channel": "main", "shots": [ { "shotId": "fa3b74d1-f27f-4fc4-a5f3-f29dd0d2750e", "index": 0, "lastGenerationAt": "", "videoPrompt": "stunning 3D video of rippling water waves, reflected on hovering metal beads in an underwater setting. Rendered in Blender with subtle, whaling ambient light, captured in 8K UHD.", "audioPrompt": "" }, { "shotId": "f89e600c-91e8-401d-a870-3286b779e5a3", "index": 1, "lastGenerationAt": "", "videoPrompt": "high-quality 3D animation of a cloth-like structure falling in a windless room, displaying the realistic physics of fabric movement. Rendered in ultra-detailed Blender, shot in 4K Cinema4D.", "audioPrompt": "" }, { "shotId": "333aae1c-db8c-424e-bb05-d7161952beea", "index": 2, "lastGenerationAt": "", "videoPrompt": "splendid video showing the fluid simulation of liquids as they interact with static and moving objects, set in a minimalistic, clean environment. Captured with 4K Cinema4D, showcases Blender render.", "audioPrompt": "" }, { "shotId": "a19a3e22-9009-449c-a011-6d6cdc660477", "index": 3, "lastGenerationAt": "", "videoPrompt": "detailed 3D animation of cellular automata, using simple elements to create complex patterns in a uniform space. Exhibits award-winning lighting techniques and Blender rendering, shot in Cinema4D studio quality.", "audioPrompt": "" }, { "shotId": "694cb3a9-7f51-4f7e-86df-7ae56d34e433", "index": 4, "lastGenerationAt": "", "videoPrompt": "fabulous Blender rendering of a high-speed camera capture of metal beads impacting a flat surface, displaying brilliant n-body collision physics. Captured at a ripe evening with Cinema4D.", "audioPrompt": "" }, { "shotId": "d59ec76d-4769-44ab-9aea-09362764d35d", "index": 5, "lastGenerationAt": "", "videoPrompt": "breathtakingly crisp 3D animation featuring a fluid mechanism, where treacle-thick liquid pours into a large vat. Beautifully lit by diffused, soft light. Rendered in Blender, captured at 8K UHD with Cinema4D.", "audioPrompt": "" }, { "shotId": "e170953f-4913-428d-a551-53128920e4c8", "index": 6, "lastGenerationAt": "", "videoPrompt": "surreal yet hyper-realistic 3D animation of cloth simulation, showcasing the realism of Blender's material simulation capabilities. With studio lighting illuminating the scene, shot at close-up with Cinema4D.", "audioPrompt": "" }, { "shotId": "1dcca91d-c9a4-4ed2-b011-1a55d990ca46", "index": 7, "lastGenerationAt": "", "videoPrompt": "contemporary fine-art 3D animation showcasing the stunning interaction between thousands of metal beads under n-body simulation. Rendered with precision in Blender and captured at a high detail with Cinema4D.", "audioPrompt": "" }, { "shotId": "a9296116-b5e5-43ef-914e-d75c7ad2750e", "index": 8, "lastGenerationAt": "", "videoPrompt": "Blender-rendered 3D animation of a glass cube falling into a pool of liquid, captured to showcase fluid dynamics and refraction of light effects. Caught under a soft diffused light with Cinema4D.", "audioPrompt": "" }, { "shotId": "6c8e7622-268e-491a-a6bc-0ad5b927c7c0", "index": 9, "lastGenerationAt": "", "videoPrompt": "impressive 3D animation, of cellular automata, forming fascinating patterns in a confined environment. The geometrically exact elements are rendered with Blender. Shot at dusk, in crisp 8K UHD with Cinema4D.", "audioPrompt": "" } ] }, {