webrtc_codecs_capture (webrtc_codecs v0.1.0)

View Source

Capture raw pictures or audio from a camera, a microphone, a file or a test source, through the ffmpeg executable running as a port.

{ok, Cam} = webrtc_codecs_capture:start({camera, 0}, #{width => 640, height => 480, frame_rate => 30}),
receive
    {webrtc_capture, Cam, #{format := i420} = Picture} -> webrtc:send_media(Pc, Mid, Picture)
end.

The capture process owns the device. It sends each picture (webrtc_codecs:picture/0) or chunk of audio (webrtc_codecs:audio/0, without a timestamp) to its owner as {webrtc_capture, Pid, Raw}, then {webrtc_capture, Pid, eof} or {webrtc_capture, Pid, {error, Reason}} when the source ends, and stops. It stops with its owner.

When the owner has more than max_queue messages waiting, what is captured is dropped and counted, so a slow owner costs frames, not memory.

With the encode option FFmpeg also encodes the video, and the owner gets encoded frames to pass to webrtc:send_frame/4, with no codec plugin:

{ok, Cam} = webrtc_codecs_capture:start({camera, 0}, #{encode => #{codec => vp8, bitrate => 1000000}}),
receive
    {webrtc_capture, Cam, #{data := Frame, timestamp := Ts, keyframe := Key}} ->
        webrtc:send_frame(Pc, Mid, Frame, #{timestamp => Ts, keyframe => Key})
end.

Summary

Types

An encoded frame, as webrtc:send_frame/4 takes it: timestamp is in 90 kHz units.

What to capture: a device by its index or its name in devices/0, a file FFmpeg can read, or a source FFmpeg generates (a moving test picture, a 440 Hz tone).

Functions

The cameras and microphones FFmpeg sees. On macOS the terminal or the application needs the camera and microphone permissions to open them; on Linux the user has to be in the video and audio groups.

Start capturing. The process is linked to the caller.

Pictures, chunks of audio or encoded frames captured so far, and how many of them were dropped.

Stop capturing and release the device.

Types

device()

-type device() :: #{index := non_neg_integer(), name := binary()}.

encode()

-type encode() ::
          #{codec := vp8 | vp9 | av1 | h264,
            bitrate => pos_integer(),
            keyframe_interval => pos_integer(),
            encoder => string(),
            encoder_options => [string()]}.
  • codec: vp8, vp9, av1 or h264
  • bitrate: in bits per second, default 1000000
  • keyframe_interval: frames between two key frames, default 60. FFmpeg cannot be asked for a key frame while it runs, so a peer that loses a frame waits for the next one.
  • encoder: the FFmpeg encoder, default libvpx, libvpx-vp9, libsvtav1, and for H.264 h264_videotoolbox on macOS and libopenh264 elsewhere
  • encoder_options: FFmpeg options for the encoder, replacing the real time ones this module sets for the default encoders

encoded_frame()

-type encoded_frame() ::
          #{codec := vp8 | vp9 | av1 | h264,
            data := binary(),
            timestamp := non_neg_integer(),
            keyframe := boolean()}.

An encoded frame, as webrtc:send_frame/4 takes it: timestamp is in 90 kHz units.

options()

-type options() ::
          #{owner => pid(),
            width => pos_integer(),
            height => pos_integer(),
            frame_rate => pos_integer(),
            sample_rate => pos_integer(),
            channels => pos_integer(),
            frame_ms => pos_integer(),
            max_queue => non_neg_integer(),
            realtime => boolean(),
            encode => encode(),
            input_options => [string()],
            ffmpeg => file:filename_all()}.
  • owner: the process that gets the media, default the caller
  • width, height, frame_rate: of pictures, default 640, 480, 30; width and height are even
  • sample_rate, channels, frame_ms: of audio, default 48000, 1, 20
  • max_queue: messages the owner may have waiting before capture drops, default 10
  • realtime: pace a file or a test source at its own rate, default true; with false FFmpeg reads as fast as it can, and what the owner does not take in time is dropped
  • encode: have FFmpeg encode the video and give encoded frames, see encode/0
  • input_options: FFmpeg options placed before the input, to replace the ones this module derives for a device
  • ffmpeg: path of the executable, default the one in PATH

source()

-type source() ::
          {camera, non_neg_integer() | binary()} |
          {microphone, non_neg_integer() | binary()} |
          {file, video | audio, file:filename_all()} |
          {test, video | audio}.

What to capture: a device by its index or its name in devices/0, a file FFmpeg can read, or a source FFmpeg generates (a moving test picture, a 440 Hz tone).

Functions

devices()

-spec devices() -> {ok, #{video := [device()], audio := [device()]}} | {error, term()}.

The cameras and microphones FFmpeg sees. On macOS the terminal or the application needs the camera and microphone permissions to open them; on Linux the user has to be in the video and audio groups.

start(Source, Opts)

-spec start(source(), options()) -> {ok, pid()} | {error, term()}.

Start capturing. The process is linked to the caller.

stats(Pid)

-spec stats(pid()) -> #{captured := non_neg_integer(), dropped := non_neg_integer()}.

Pictures, chunks of audio or encoded frames captured so far, and how many of them were dropped.

stop(Pid)

-spec stop(pid()) -> ok.

Stop capturing and release the device.