Loading repository data…
whitphx / repository
Real-time video and audio processing on Streamlit
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
Handling and transmitting real-time video/audio streams over the network with Streamlit 
Sister project: streamlit-fesion to execute video filters on browsers with Wasm.
It converts your voice into text in real time. This app is self-contained; it does not depend on any external API.
It applies a wide variety of style transfer filters to real-time video streams.
(Online demo not available)
You can create video chat apps with ~100 lines of Python code.
MediaPipe is used for pose estimation.
$ pip install -U streamlit-webrtc
See also the sample pages, pages/*.py, which contain a wide variety of usage.
See also "Developing Web-Based Real-Time Video/Audio Processing Apps Quickly with Streamlit".
Create app.py with the content below.
from streamlit_webrtc import webrtc_streamer
webrtc_streamer(key="sample")
Unlike other Streamlit components, webrtc_streamer() requires the key argument as a unique identifier. Set an arbitrary string to it.
Then run it with Streamlit and open http://localhost:8501/.
$ streamlit run app.py
You see the app view, so click the "START" button.
Then, video and audio streaming starts. If asked for permissions to access the camera and microphone, allow it.
When the app sends local camera or microphone input, webrtc_streamer() shows camera and microphone toggle buttons next to the Start/Stop button. These controls let users turn their outgoing camera or microphone track on and off without stopping the WebRTC session.
Set media_toggle_controls=False to hide these toggle buttons.
from streamlit_webrtc import webrtc_streamer
webrtc_streamer(key="example", media_toggle_controls=False)
When a user turns off the camera or microphone with these buttons, the WebRTC track stays active. As described in MDN's MediaStreamTrack.enabled documentation, disabled audio tracks send silence, and disabled video tracks send black frames; the session does not stop or renegotiate.
Next, edit app.py as below and run it again.
from streamlit_webrtc import webrtc_streamer
import av
def video_frame_callback(frame):
img = frame.to_ndarray(format="bgr24")
flipped = img[::-1,:,:]
return av.VideoFrame.from_ndarray(flipped, format="bgr24")
webrtc_streamer(key="example", video_frame_callback=video_frame_callback)
Now the video is vertically flipped.
As an example above, you can edit the video frames by defining a callback that receives and returns a frame and passing it to the video_frame_callback argument (or audio_frame_callback for audio manipulation).
The input and output frames are the instance of av.VideoFrame (or av.AudioFrame when dealing with audio) of PyAV library.
You can inject any kinds of image (or audio) processing inside the callback. See examples above for more applications.
You can also pass parameters to the callback.
In the example below, a boolean flip flag is used to turn on/off the image flipping.
import streamlit as st
from streamlit_webrtc import webrtc_streamer
import av
flip = st.checkbox("Flip")
def video_frame_callback(frame):
img = frame.to_ndarray(format="bgr24")
flipped = img[::-1,:,:] if flip else img
return av.VideoFrame.from_ndarray(flipped, format="bgr24")
webrtc_streamer(key="example", video_frame_callback=video_frame_callback)
Sometimes we want to read the values generated in the callback from the outer scope.
Note that the callback is executed in a forked thread running independently of the main script, so we have to take care of the following points and need some tricks for implementation like the example below (See also the section below for some limitations in the callback due to multi-threading).
The following example is to pass the image frames from the callback to the outer scope and continuously process it in the loop. In this example, a simple image analysis (calculating the histogram like this OpenCV tutorial) is done on the image frames.
threading.Lock is one standard way to control variable accesses across threads.
A dict object img_container here is a mutable container shared by the callback and the outer scope and the lock object is used at assigning and reading the values to/from the container for thread-safety.
import threading
import cv2
import streamlit as st
from matplotlib import pyplot as plt
from streamlit_webrtc import webrtc_streamer
lock = threading.Lock()
img_container = {"img": None}
def video_frame_callback(frame):
img = frame.to_ndarray(format="bgr24")
with lock:
img_container["img"] = img
return frame
ctx = webrtc_streamer(key="example", video_frame_callback=video_frame_callback)
fig_place = st.empty()
fig, ax = plt.subplots(1, 1)
while ctx.state.playing:
with lock:
img = img_container["img"]
if img is None:
continue
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
ax.cla()
ax.hist(gray.ravel(), 256, [0, 256])
fig_place.pyplot(fig)
The callbacks are executed in forked threads different from the main one, so there are some limitations:
st.* such as st.write()) do not work inside the callbacks.global keyword does not work expectedly in the callbacks.webrtc_streamer() accepts on_video_ended and on_audio_ended arguments — zero-argument callables that fire when the corresponding input media track ends (the user clicks "STOP", closes the page, or the connection drops). They are the recommended hook for tearing down per-session resources that the frame callbacks allocated, such as worker threads, model handles, file writers, queues, or st.session_state entries.
import streamlit as st
from streamlit_webrtc import webrtc_streamer
def video_frame_callback(frame):
# ... process the frame, possibly initializing per-session state on first call ...
return frame
def on_video_ended():
st.session_state.pop("my_session_resource", None)
webrtc_streamer(
key="example",
video_frame_callback=video_frame_callback,
on_video_ended=on_video_ended,
)
These callbacks run on aiortc's asyncio loop — not Streamlit's main thread — so the same caveats as the frame callbacks apply: st.* calls do not work inside them, and shared state must be mutated in a thread-safe way (e.g. with a threading.Lock, a queue, or a threading.Event).
When using the class-based API, override VideoProcessorBase.on_ended() / AudioProcessorBase.on_ended() instead — they fire at the same lifecycle point.
Factory helpers such as create_video_source_track(), create_audio_source_track(), create_video_sink_track(), create_audio_sink_track(), and create_pcm_audio_source_track() cache their returned objects in st.session_state by key so they survive Streamlit reruns. This is usually what you want: widget changes and reruns keep using the same media track.
By default, factory-created source and sink tracks are scoped to the active WebRTC session. When that session ends, for example when the user clicks STOP, closes the page, or the connection drops, the cached object is stopped and removed from st.session_state. The next WebRTC session with the same key gets a fresh object.
from streamlit_webrtc import create_video_source_track
video_track = create_video_source_track(
callback=video_source_callback,
key="video-source",
)
To keep a factory-created object alive across multiple WebRTC sessions in the same Streamlit session, opt out with lifecycle_scope="streamlit-session":
video_track = create_video_source_track(
callback=video_source_callback,
key="video-source",
lifecycle_scope="streamlit-session",
)
lifecycle_scope applies to source/sink factory helpers and create_pcm_audio_source_track(). I
Selected from shared topics, language and repository description—not editorial ratings.
Kartik-Aggarwal /
Using only Image Processing Techniques, detecting vehicles in real time video, creating bounding box, and tracking them
31/100 health