Savant 0.4.0 is out. This release focuses on system usability, interfacing, advanced computer vision, and video analytics. It is based on the state-of-the-art DeepStream 6.4. Savant moves towards the frontier of creating omnipresent computer vision and video analytical systems working in hybrid mode on the edge and in the data centers.
Unlike commonly used computer vision frameworks like PyTorch, TensorFlow, OpenVINO/DlStreamer, and DeepStream, Savant offers its users not only inference and image manipulation tools but also advanced architecture for building distributed edge/datacenter computer vision applications communicating over the network. Thus, Savant users focus on computer vision but do not reinvent the wheel, productizing their pipelines. So, what is new in Savant v0.4.0?
Developer-friendly “-extra” Docker Images For X86/dGPU and Jetson
Savant is a Docker-first runtime. Previously, we provided only production-friendly docker images, so developers were required to install their favorite toolkits by themselves. People who are into computer vision know how challenging it can be to fight incompatible versions of libraries. Thus, we decided to deliver -extra images easing developers’ lives. Extra images contain the following, in addition to basic components:
| Component | x86 Version | Jetson Version | Notes |
|---|---|---|---|
| PyTorch | 2.3.0 | 2.3.0 | With CUDA support |
| Torchaudio | 2.3.0 | 2.3.0 | |
| Torchvision | 0.18.0 | 0.18.0 | |
| TensorRT | 8.6.1 | 8.6.2 | |
| Torch2TRT | 0.5.0 | 0.4.0 | |
| ONNX | 1.16.0 | 1.16.0 | |
| ONNX Runtime | 1.17.1 | 1.17.0 | With CUDA support |
| PyCUDA | 2024.1 | 2024.1 | |
| Pandas | 2.2.2 | 2.2.2 | |
| Polars | 0.20.26 | 0.19.12 | |
| Scikit-learn | 1.4.2 | 1.4.2 | |
| JupyterLab | 4.2.0 | 4.2.0 |
People often use Savant to prepare and validate datasets and evaluate pipeline business metrics. Thus, we included most components seasoned data scientists and machine learning engineers require.
AWS Kinesis Video Streams Integration
In this release, Savant introduces two adapters that allow video streams to be sent to AWS KVS and consumed from it. Now, it is very easy to build the system by utilizing the power of KVS to inject streams, read them continuously on-demand, and process the video with Savant.
To help users understand KVS integration, we prepared a sample that shows how to pass video data and metadata from the edge to the KVS to the data center processing module.
Always-On RTSP Sink Now Supports HEVC
The AO-RTSP sink supported only the H.264 codec in previous releases. This codec limited the use of the adapter with high-definition streaming, requiring advanced codecs like HEVC. Nvidia GPUs can now encode 8K images, so we implemented HEVC in AO-RTSP. Now, you can opt-in for HEVC if you need it.
Always-On RTSP Sink Now Supports REST API
In previous releases, the AO-RTSP configuration could be defined only during launch. Thus, users could not enable or disable playback for streams based on their needs. REST API enables selective behavior, allowing users to control what streams must be cast and what is ignored. It enables efficient utilization of computing and traffic resources.
Savant customers implementing on-demand playback for end-user needs desperately wanted this feature.
AO-RTSP Enhancement For Time-Synchronized Video Playback
By default, AO-RTSP maps ingress frames to egress streams as soon as received without considering encoded PTS/DTS labels. This feature allows buffering ingress frames to ensure the video is processed according to the timestamps.
Savant users who want to display the pipeline processing results do not often use this feature. However, time synchronization is crucial when displaying a video with cinematographic quality, precisely matching the original one without micro lags or freezes.
USB Camera Source Supports MJPEG
In this release, we removed the old USB source adapter, which supported only RAW image capturing, and enhanced the universal FFmpeg source adapter to support MJPEG.
We wrote the article to show how to consume MJPEG with the FFmpeg source adapter. We also implemented a sample demonstrating work with an MJPEG USB camera.
FFmpeg Source Raw Pixel Format Enhancement
L4T USB cameras and raw image/video sources can provide images in multiple-pixel formats. However, Savant supports only a limited number of them. The FFmpeg source adapter automatically converts unsupported pixel formats to RGB24 in this release.
Stream Quarantine Feature
Certain adapters can intentionally or unintentionally deliver inconsistent or unsupported streams to computer vision systems. Thus, in 0.4.0, we introduce a new feature that allows Savant modules to block those streams, placing them into quarantine temporarily.
This feature allows pipelines to endure situations that previously caused failures because the video decoding subsystem could not process the stream. Instead of crashing, we add those streams to a blacklist.
VP8, VP9 Decoding, And MKV Support
We extended the codec support in this release. The source adapters can now process VP8 and VP9-encoded streams and read MKV containers. We also wanted to implement AV1 encoding support but met the problem in DeepStream, so this feature will be postponed until the resolution is achieved.
Synchronized Video Streams Support
There are situations when multiple video sources have time-based synchronization, and those frames must be delivered into the processing pipeline together. In 0.4.0, we extended the Savant protocol to support so-called time-synchronized batch messages combining video packets from multiple sources. Examples when one needs that are:
- Stereo-vision systems (two or more USB/CSI2 cameras are time-synchronized);
- Multiple RTSP stream time-based synchronization (all RTSP cameras are synchronized with RTCP NTP time).
This feature is required for computer vision pipelines to process images from various sources, such as self-driving systems, robotics, and camera calibration.
Replay: Video Archiving and Re-Streaming Component
In this release, we present a new component, Replay, which allows video streams with their metadata to be collected and re-streamed with RESTful API.
Replay is an advanced storage providing features required for non-linear computer vision and video analytics:
- collects video from multiple streams (archiving with TTL eviction);
- provides a REST API for video re-streaming to Savant sinks or modules;
- supports time-synchronized and fast video re-streaming;
- supports configurable video re-streaming stop conditions;
- supports setting minimum and maximum frame duration to increase or decrease the video playback speed;
- can fix incorrect TS in re-streaming video streams;
- can look backward when the video stream is re-streamed;
- can set additional attributes to retrieved video streams;
- can work as a sidecar or intermediary service in Savant pipelines.
How Replay Is Implemented
Replay is implemented with Rust and RocksDB. It delivers video with low latency and high throughput. Replay can be deployed on edge devices and in the cloud on ARM or X86 devices.
Usage Sample
We implemented a sample demonstrating how to ingest a video with the file source adapter into the Replay database and re-restream it to the AO-RTSP sink for playback.
Replay License
Replay is provided under the BSL-1.1 license, meaning that organizations must acquire a commercial license to use it in production but do not need licenses for non-production use.
We offer various license types:
- Sponsored licenses for non-commercial users,
- Affordable perpetual licenses for end-user organizations;
- Annual licenses for service and solution providers that allow distribution of the software containing Replay.
Please fill out the form to acquire the license and consult on Replay.
Future Releases
We will release several minor updates with features and fixes in the 0.4.x branch. By the end of the summer, we plan to release Savant 1.0.0 (probably based on DeepStream 7.0 with WSL2 support).
Do not hesitate to contact us on Discord to get more information about Savant and to progress with community help quickly.