Working With Dynamic Video Sources in Savant

It is difficult to work efficiently with multiple video sources in a computer vision pipeline. The system must handle their connection, disconnection, and outages, negotiate codecs, and spin corresponding decoder and encoder pipelines based on known hardware features and limitations.

That is why Savant promotes plug-and-play technology for video streams, which takes care of the nuances related to source management, automatic dead source eviction, codec negotiation, etc. Developers do not care about how the framework implements that – just attach and detach new sources on demand.

The article demonstrates how to dynamically attach and detach sources to a pipeline with plain Docker.

To fully understand how Savant works with adapters, please first read the article related to the Savant protocol in the documentation. In this article, we will show how to connect and disconnect a source from a running module without diving into how it works internally.

In our samples, we use the “docker-compose” to simplify the execution and help users quickly launch the code. However, this often causes misunderstandings among those who do not understand Docker machinery well. So, let us begin by “decomposing” a sample. Savant supports Unix domain sockets and TCP/IP sockets for component communication, and we will try both.

Sample Decomposing

We will use the key point detection sample, which you can find in the samples/keypoint_detection directory. Let us take a quick look at the YAML:

version: "3.3"
services:

  video-loop-source:
    image: ghcr.io/insight-platform/savant-adapters-gstreamer:latest
    restart: unless-stopped
    volumes:
      - zmq_sockets:/tmp/zmq-sockets
      - /tmp/video-loop-source-downloads:/tmp/video-loop-source-downloads
    environment:
      - LOCATION=https://eu-central-1.linodeobjects.com/savant-data/demo/shuffle_dance.mp4
      - DOWNLOAD_PATH=/tmp/video-loop-source-downloads
      - ZMQ_ENDPOINT=dealer+connect:ipc:///tmp/zmq-sockets/input-video.ipc
      - SOURCE_ID=video
      - SYNC_OUTPUT=True
    entrypoint: /opt/savant/adapters/gst/sources/video_loop.sh
    depends_on:
      module:
        condition: service_healthy

  module:
    image: ghcr.io/insight-platform/savant-deepstream:latest
    restart: unless-stopped
    volumes:
      - zmq_sockets:/tmp/zmq-sockets
      - ../../cache:/cache
      - .:/opt/savant/samples/keypoint_detection
    command: samples/keypoint_detection/module.yml
    environment:
      - MODEL_PATH=/cache/models/keypoint_detection
      - DOWNLOAD_PATH=/cache/downloads/keypoint_detection
      - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
      - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
      - METRICS_FRAME_PERIOD=1000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  always-on-sink:
    image: ghcr.io/insight-platform/savant-adapters-deepstream:latest
    restart: unless-stopped
    ports:
      - "554:554"    # RTSP
      - "1935:1935"  # RTMP
      - "888:888"    # HLS
      - "8889:8889"  # WebRTC
    volumes:
      - zmq_sockets:/tmp/zmq-sockets
      - ../assets/stub_imgs:/stub_imgs
    environment:
      - ZMQ_ENDPOINT=sub+connect:ipc:///tmp/zmq-sockets/output-video.ipc
      - SOURCE_ID=video
      - STUB_FILE_LOCATION=/stub_imgs/smpte100_1280x720.jpeg
      - DEV_MODE=True
      - FRAMERATE=25/1
    command: python -m adapters.ds.sinks.always_on_rtsp
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  zmq_sockets:

In the code, you can see that containers use the Unix domain sockets for communication. Those sockets must share the same file space, which is defined with:

volumes:
  zmq_sockets:

The zmq_sockets declaration is not a filesystem path; it is a docker volume and when you invoke docker compose, it mangles the name with the compose parent directory to avoid name collision.

The volume is a preferred way for compose but for independent docker containers launched with the docker run command, you must create a volume separately by running the docker volume create command.

docker volume create mysockets

After the volume is created, let us decompose the module:

 module:
    image: ghcr.io/insight-platform/savant-deepstream:latest
    restart: unless-stopped
    volumes:
      - zmq_sockets:/tmp/zmq-sockets
      - ../../cache:/cache
      - .:/opt/savant/samples/keypoint_detection
    command: samples/keypoint_detection/module.yml
    environment:
      - MODEL_PATH=/cache/models/keypoint_detection
      - DOWNLOAD_PATH=/cache/downloads/keypoint_detection
      - ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc
      - ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc
      - METRICS_FRAME_PERIOD=1000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

to:

docker run --gpus all -it --rm \
   -v mysockets:/tmp/zmq-sockets \
   -v $(pwd)/../../cache:/cache \
   -v $(pwd)/.:/opt/savant/samples/keypoint_detection \
   -e MODEL_PATH=/cache/models/keypoint_detection \
   -e DOWNLOAD_PATH=/cache/downloads/keypoint_detection \
   -e ZMQ_SRC_ENDPOINT=router+bind:ipc:///tmp/zmq-sockets/input-video.ipc \
   -e ZMQ_SINK_ENDPOINT=pub+bind:ipc:///tmp/zmq-sockets/output-video.ipc \
   -e METRICS_FRAME_PERIOD=1000 \
   ghcr.io/insight-platform/savant-deepstream:latest \
     samples/keypoint_detection/module.yml

The module must run normally, transitioning to the “RUNNING” state:

 INFO  insight::savant::gstreamer::runner                   > The pipeline is initialized and ready to process data. Initialization took 0:00:03.917970.
 INFO  insight::savant::healthcheck::status                 > Setting module status to ModuleStatus.RUNNING.
 INFO  insight::savant::utils::zeromq                       > Starting ZMQ source.

Now, it does not serve streams because no adapter is connected. Let us connect the adapter.

The source adapter declaration in the compose looks like as follows:

  video-loop-source:
    image: ghcr.io/insight-platform/savant-adapters-gstreamer:latest
    restart: unless-stopped
    volumes:
      - zmq_sockets:/tmp/zmq-sockets
      - /tmp/video-loop-source-downloads:/tmp/video-loop-source-downloads
    environment:
      - LOCATION=https://eu-central-1.linodeobjects.com/savant-data/demo/shuffle_dance.mp4
      - DOWNLOAD_PATH=/tmp/video-loop-source-downloads
      - ZMQ_ENDPOINT=dealer+connect:ipc:///tmp/zmq-sockets/input-video.ipc
      - SOURCE_ID=video
      - SYNC_OUTPUT=True
    entrypoint: /opt/savant/adapters/gst/sources/video_loop.sh
    depends_on:
      module:
        condition: service_healthy

In the plain docker, it will be:

docker run -it --rm \
   -v mysockets:/tmp/zmq-sockets \
   -v /tmp/video-loop-source-downloads:/tmp/video-loop-source-downloads \
   -e LOCATION=https://eu-central-1.linodeobjects.com/savant-data/demo/shuffle_dance.mp4 \
   -e DOWNLOAD_PATH=/tmp/video-loop-source-downloads \
   -e ZMQ_ENDPOINT=dealer+connect:ipc:///tmp/zmq-sockets/input-video.ipc \
   -e SOURCE_ID=video \
   -e SYNC_OUTPUT=True \
   --entrypoint /opt/savant/adapters/gst/sources/video_loop.sh \
   ghcr.io/insight-platform/savant-adapters-gstreamer:latest

After the launch, the module will report that a new source is connected:

 INFO  insight::savant::utils::zeromq                       > Starting ZMQ source.
 INFO  insight::savant::savant_rs_video_decode_bin          > Adding branch with source video
 INFO  insight::savant::savant_rs_video_decode_bin          > Branch with source video added
 INFO  insight::savant::savant_rs_video_demux               > Created new src pad for source video: src_video.
 INFO  insight::savant::keypoint_detection                  > Added source video

If you stop the adapter, the module will report after a while that the stream is no longer available:

nvstreammux: Successfully handled EOS for source_id=0
 INFO  insight::savant::savant_rs_video_decode_bin          > Removing branch with source video
 INFO  insight::savant::savant_rs_video_decode_bin          > Branch with source video removed
 INFO  insight::savant::keypoint_detection                  > Resources for source video has been released.

Thus, you can connect and disconnect an arbitrary number of streams dynamically and simultaneously. You must only provide a unique SOURCE_ID for each of them.

I do not decompose the AO-RTSP component, you can do it by yourself using the same techniques.

Host Directory Instead Of Volume

You can replace the docker volume with a host directory. Just replace:

-v mysockets:/tmp/zmq-sockets

with

-v /tmp/mysockets:/tmp/zmq-sockets

TCP/IP Sockets Instead Of Volume

You can use network sockets to connect components over the network. Let us modify our docker commands:

docker run --gpus all -it --rm \
   -v $(pwd)/../../cache:/cache \
   -v $(pwd)/.:/opt/savant/samples/keypoint_detection \
   -e MODEL_PATH=/cache/models/keypoint_detection \
   -e DOWNLOAD_PATH=/cache/downloads/keypoint_detection \
   -e ZMQ_SRC_ENDPOINT=router+bind:tcp://0.0.0.0:12345 \
   -e ZMQ_SINK_ENDPOINT=pub+bind:tcp://0.0.0.0:12346 \
   -e METRICS_FRAME_PERIOD=1000 \
   -p 12345:12345 \
   -p 12346:12346 \
   ghcr.io/insight-platform/savant-deepstream:latest \
     samples/keypoint_detection/module.yml

And the source adapter command:

docker run -it --rm \
  -v /tmp/video-loop-source-downloads:/tmp/video-loop-source-downloads \
  -e LOCATION=https://eu-central-1.linodeobjects.com/savant-data/demo/shuffle_dance.mp4 \
  -e DOWNLOAD_PATH=/tmp/video-loop-source-downloads \
  -e ZMQ_ENDPOINT=dealer+connect:tcp://192.168.100.126:12345 \
  -e SOURCE_ID=video \
  -e SYNC_OUTPUT=True \
  --entrypoint /opt/savant/adapters/gst/sources/video_loop.sh \
  ghcr.io/insight-platform/savant-adapters-gstreamer:latest

Do not forget to replace the host’s IP with the correct one.