Vizario H264RTSP Player for Unity3D

This package provides a H264/AAC player in Unity3D receiving data over RTSP.

To repeat what is the main purpose of this package:

Vizario H264RTSP Player is a lightweight library to enable live RTSP streaming of H264 and AAC bitstreams to iOS or Android devices (respectively standalone apps or Editor Win64 and OSX) from a local PC or remote server. The library allows you to register as an RTSP client to a server and play back the video content within a texture, respectively audio over the native audio device.

The library is developed towards playback of live data and is essentially the playback-counterpart of the video casting asset:

https://assetstore.unity.com/packages/tools/video/vizario-h264rtsp-196278

The main purpose of this asset is to enable simple streaming of H264/AAC live streams from an RTSP server to a client, decoding und playing back as efficiently as possible. It was **NOT** designed to do perfect audio/video synchronization, **NOR** to become a generic player framework like LibVLC.

However, the libraries use the native audio devices for audio output (if desired). In other words, the AAC decoder is indeed a full player without the need to mess with device audio setups.

General Note:

I want to emphasize again that I’m really trying to maintain this plugin for all platforms that are out there and with the maximum of flexibility and reliability, but as with every other software package, it is your own responsibility to evaluate, before using it in a productive environment.

A lot of people send emails to me, but they don’t even have a clue how an IP network is set up. You don’t have to be an expert in video encoding technology to use the setup, but a minimal understanding of how individual devices are hooked up through cables and IP addresses is a must. Here are a few more general notes.

  • WIFI Network Standards have different bandwidth and WIFI networks are crowded with devices and packet collisions occur. Check https://en.wikipedia.org/wiki/IEEE_802.11 to understand, why it might or might not work depending on the load. This directly translates also to routers and wired setups. You can’t run four 25MBit streams over a 100MBit RJ45 ethernet connection without any loss.
  • Video Encoding (and the plugin) is complex and Unity adds to it. You cannot expect any plugin to do the same as VLC or gstreamer does. Why? Because Unity is a layer between the data that is decoded by a hardware unit, and the surface you render to. All the data has to go a specific route between the decoder (on C++) and the renderer (in C#). To give you an idea, for 1920×1080 at 60fps you need to copy 1920*1080*4*60 bytes = 500MB per second across some memory barriers.
  • There is a ton of things that can go wrong, and following my experience it is sometimes “somewhere”. This includes e.g. network ports that are not freed automatically by .NET or just after a while, once you stop the streaming. These are things I can’t fix, because it is on the operating system layer. Similarly, every network setup is different and I neither know yours nor your setup.
  • This plugin is not built for telepresence in the sense that you do audio-video chatting like Skype. You can implement it, but it is explicitly NOT built for that. Go use WebRTC.
  • I’m maintaining this package in my spare time. I can do fixes and look into certain problems, but for most of the questions I receive I can just explain the framework and the reasons for certain decisions I made during development.
  • Generally, I would advise the following strategy concerning the use of the plugin:
    • Does it work out of the box? Yes? Great, move on… No? What does not work? Is it the receiving, the decoding or the rendering?
    • Use Local mode to check if the rest of the pipeline works (and in 99% of cases it does). In that case it’s about receiving the stream. Check this part (and the sender and try the mediamtx engine that gives you some hints).

Notes on Unity 6

In general the plugin works as expected, but I encountered two errors (they might or might not appear for you

  • .The plane standard material does not work on the URP in Unity 6. You need to change the plane material to Lit.
  • It might be (don’t know why) that the MainThreadDispatcher might get lost. You need to assign it to the main camera.
  • If you have troubles pointing to AAR files during building for Android, just don’t use AAR. AAR files are containers for Android libraries (naively speaking). They encapsulate multiple architecture libs, similar to iOS or OSX dylib files (which have multiple architectures like silicon or intel).
    • Rename the AAR files to ZIP and unzip.
    • Copy out the library in the jni/<abi> folder that you need (likely you only use arm64-v8a most of the time)
    • Assign the proper architecture setting in the player settings in Unity to the abi value

If you find any more issues, please send me a note.

Quick FAQ:

  • Q: Does it do RTMP?
  • A: No and will likely not do so any time soon.
  • Q: Does it work within VR setups?
  • A: Yes, it was tested to work within Oculus Quest 2/3 and Oculus Quest Pro.
  • Q: Does it work with H265/AC3?
  • A: H265 is in the making (will be included in v1.2). It does not work (and will not work any time soon) with AC3.
  • Q: It still takes a couple of seconds to show up, although it seems to have a very low latency then and audio is there almost instantly. Why’s that?
  • A: The way H264 and H265 works is using keyframes (one full frame) and differential encoding from that frame on to the next keyframe. Keyframes are inserted usually dynamically or at a fixed rate, usually something between 5 and 10 seconds. The decoder can only start if it receives a keyframe, which (depending on the sender) arrives when it arrives and that might take some seconds (and everything before that is discarded). Audio AAC can be decoded for received frames independently, therefore audio is there almost instantly.
  • Q: I’m doing everything correct, but my stream (which has a very large resolution, admittedly) stutters.
  • A: As mentioned above, there are some limits to what is doable and also reasonable, that have nothing to do with the functionality of the plugin itself. For example, a 4k video stream at 60fps will EASILY max out your decoder on mobile platforms. At 16:9 aspect ratio, this is
    4096*2304*4*60 bytes per second to display. This is 2.2GB on memory to be copied per second! Please consider the capabilities of your device with respect to what you want to achieve. Honestly, if the plugin can’t do it (considering the overhead included by Unity itself), then likely no other plugin or software can do it.
  • Q: My H264/H265 streams stutter and seem to go forth and back on a per frame basis.
  • A: You need to provide a stream without B-frames. The decoder cannot deal with B-frames right now. B-frames mean “backward-in-time” frames. For offline encoded videos, there is forward and backward encoding. For live streaming, backward info does not make sense.
  • Q: Is there a version for the Apple Vision Pro?
  • A: Yes and no. I have a preliminary version, but (thank you Apple) there are a few issues with this one. Send me an Email if you are interested in testing it.
  • Q: Can I stream multiple videos at the same time?
  • A: Yes, but there is a physical limit to the number of hardware decoder units. There is no software fallback, so as soon as you reach that limit (at 4 usually), creating a new player will fail.
  • Q: Is it built for 100% reliable streaming over a couple of hours or longer?
  • A: Certainly not, and no other player (not even VLC) can guarantee this. RTSP is a relatively simplistic protocol without fail safe reconnect or other features. The best the plugin can do is stop gracefully. You have to take care about restarting yourself.
  • Q: Is there a test package available before I buy it?
  • A: No, but if you send me a nice email with some info about your use case, I can probably make something work.
  • Q: Can I stream from VLC to RTSPPlayer directly?
  • A: It is indeed possible, but it currently only works from the command line (see the PDF in the package for the exact call details). Unfortunately I was not able to get it to work configuring the GUI.

See the current documentation PDF for more information.

Advanced FAQ:

  • Q: How does it compare to VLC?
  • A: It was never designed to do anything like VLC. If you need a full media player, you are better off with an other package
  • Q: Can I stream multiple streams?
  • A: Yes, but probably it is not working in the version you have. See this video for a working example:
  • Q; How does the sprite rendering work?
  • A: See this video for an example:
  • Q: A feature I need is not working or implemented. What should I do?
  • A: Send me an email! I don’t update the package every time I make bug-fixes or implement new stuff.