Prepare

requestAnimationFrame

requestAnimationFrame is a web standard API. By passing a callback as an argument, the callback is executed when the browser is ready to redraw.

If redrawing is performed in a loop process without using requestAnimationFrame, there is a possibility that redrawing will be performed at a timing when it is not possible to draw. In other words, there is a possibility of iteration of the redrawing process not being displayed on the screen.

Also, if multiple requestAnimationFrame callbacks are enqueued in the same frame, they will be executed in the same frame. The following is from mozilla docs. timestamp refers to the end time of the previous drawing frame The timestamp refers to the end time of the previous drawing frame.

When multiple callbacks queued by requestAnimationFrame() begin to fire in a single frame, each receives the same timestamp even though time has passed during the computation of every previous callback’s workload.

How remotion renders

In the following, you will read about how to render videos for remotion based on the implementation of remotion v4.0.81.

Sequence diagram

The following is a sequence diagram of rendering by remotion. ¹

The following are characteristic points

• Whether the headless browser is ready to render a frame or not is determined using requestAnimationFrame.
• The API for taking screenshots does not exist in JavaScript ², so the CDP (Chrome DevTools Protocol) Page.captureScreenshot
  • To execute Page.captureScreenshot when the browser is ready to render, determine the execution timing as follows
    • On a headless browser running React, set the global variable (remotion_renderReady) to have a flag indicating whether or not the browser is ready to render.
    • The waitForReady function will send remotion_renderReady to the renderReady. To allow the React side to execute a continueRender callback with requestAnimationFrame, the polling loop should be executed by requestAnimationFrame.

sequenceDiagram
    participant renderer
    participant puppeteer
    participant browser
    participant remotion_renderReady
    participant react

    loop screenshot a frame
      Note over renderer,react: set the next frame
        renderer->>puppeteer: set the next frame
        puppeteer->>browser: remotion_setFrame
(Runtime.callFunctionOn)
        browser->>remotion_renderReady: delayRender
        remotion_renderReady->>remotion_renderReady: set false
        remotion_renderReady->>browser: done
        browser->>react: set frame context
        browser->>remotion_renderReady: continueRender
with requestAnimationFrame
        browser->>puppeteer: done
        puppeteer->>renderer: done
        par rerendering the next frame
          react->>react: update frame context and 
rerender user-defined components
          remotion_renderReady->>remotion_renderReady: set true
with renderable timing
        end
        
      Note over renderer,react: wait ready state
        renderer->>puppeteer: wait ready
        puppeteer->>browser: onRaF
(Runtime.callFunctionOn)
        loop until remotion_renderReady being true
          browser->>remotion_renderReady: check value
with requestAnimationFrame
          remotion_renderReady->>browser: 
        end 
        browser->>puppeteer: done
        puppeteer->>renderer: ready
        
      Note over renderer,react: take screenshot
        renderer->>puppeteer: take screenshot
        puppeteer->>browser: Page.captureScreenshot(CDP)
        browser->>puppeteer: screenshot
    end
  

Summary

remotion is a very practical framework that enables accurate frame-by-frame video rendering by using requestAnimationFrame and React’s Context.
Although not covered in this article, it also provides a mechanism for embedding video and audio so that they will play correctly on the rendered video.