Aiortc to Create a Remote Web Camera With an Android Phone
Fleetingaiortc, android webcam, ip camera
Using the code from https://github.com/jlaine/aiortc/tree/master/examples/server, keeping only what I need.
This setup involves 4 programs:
- the signaling server, using peerjs, to make two web applications communicate, listening to 192.168.1.46:9999/peerjs
- a web server, using aiortc to receive some camera flux, listening to 192.168.1.245:9999/offer
- a web application, that will send its flux to the aiortc application, hereafter called the camera,
- a web application to control the camera provider, hereafter called the controller,
setup
mkdir -p ~/test/android_webcam
cd ~/test/android_webcam
pdm init --non-interactive
The only protocol that web applications can use is http (aiohttp). On top of that, I want to bootstrap an RTC communication (aiortc).
Is this setup, The web application will communicate from localhost to 192.168.1.245:9999. This Cross-Origin Resource Sharing is prevented by the browser by default. Therefore we will make the server explicitly allow this connection using a middleware (aiohttp-cors).
pdm add aiohttp aiortc aiohttp-cors
Adding packages to default dependencies: aiohttp, aiortc, aiohttp-cors
0:00:09 🔒 Lock successful.
Changes are written to pyproject.toml.
STATUS: Resolving packages from lockfile...
All packages are synced to date, nothing to do.
0:00:00 🎉 All complete! 0/0
code that runs on the server, to save the video
You may have to patch aiortc/contrib/media.py to change stream = self.__container.add_stream("libx264", rate=30) into stream = self.__container.add_stream("libx264", rate=300)
Otherwise, the video will stop streaming with errors like
[libx264 @ 0x34b4180] non-strictly-monotonic PTS
[libx264 @ 0x34b4180] non-strictly-monotonic PTS
[mp4 @ 0x3b1bbc0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 352768 >= 352768
This is a known issue, see https://github.com/aiortc/aiortc/issues/580, https://github.com/aiortc/aiortc/issues/546 and https://github.com/aiortc/aiortc/issues/331
#!/usr/bin/env python3
# -*- coding:utf-8 -*-
import argparse
import asyncio
import json
import logging
import os
import subprocess
import uuid
import numpy as np
from aiohttp import web
import aiohttp_cors
import cv2
from aiortc import MediaStreamTrack, RTCPeerConnection, RTCSessionDescription
from aiortc.contrib.media import MediaBlackhole, MediaPlayer, MediaRecorder, MediaRelay
from aiortc.mediastreams import MediaStreamError
from aiohttp import web
pcs = set()
relay = MediaRelay()
ROOT = os.path.dirname(__file__)
logger = logging.getLogger("pc")
logging.basicConfig(level=logging.INFO)
class VideoSave:
def __init__(self):
self.width, self.height = 640, 480
self.fps = 30
fourcc = cv2.VideoWriter_fourcc(*'VP80')
self.out = cv2.VideoWriter('output.webm', fourcc, self.fps, (self.width, self.height))
# self.process = subprocess.Popen([
# 'ffmpeg',
# '-y', # Overwrite output file if it exists
# '-f', 'rawvideo',
# '-vcodec', 'rawvideo',
# '-pix_fmt', 'bgr24',
# '-s', f'{self.width}x{self.height}', # Frame size
# '-r', str(self.fps), # Frame rate
# '-i', '-', # Input comes from stdin
# '-c:v', 'libvpx', # Use VP8 codec for WebM
# '-b:v', '1M', # Bitrate
# '-f', 'webm',
# "output2.webm",
# ], stdin=subprocess.PIPE)
async def save(self, track):
print("start saving frames")
while True:
try:
frame = await track.recv()
except MediaStreamError:
print("Ended getting the stream")
return
img = frame.to_ndarray(format="bgr24")
img = cv2.resize(img, (self.width, self.height))
self.out.write(img) # cv2.cvtColor(np.array(frame.to_image()), cv2.COLOR_RGB2BGR))
#self.process.stdin.write(img.tobytes())
def start(self, track):
print("starting save")
asyncio.ensure_future(self.save(track))
def stop(self):
print("Saving the video")
self.out.release()
#self.process.stdin.close()
#self.process.wait()
async def on_shutdown(app):
coros = [pc.close() for pc in pcs]
await asyncio.gather(*coros)
pcs.clear()
async def offer(request):
print("offer")
params = await request.json()
offer = RTCSessionDescription(sdp=params["sdp"], type=params["type"])
pc = RTCPeerConnection()
pc_id = "PeerConnection(%s)" % uuid.uuid4()
pcs.add(pc)
saver = VideoSave()
def log_info(msg, *args):
logging.info(pc_id + " " + msg, *args)
log_info("Created for %s", request.remote)
@pc.on("connectionstatechange")
async def on_connectionstatechange():
log_info("Connection state is %s", pc.connectionState)
if pc.connectionState == "failed":
await pc.close()
pcs.discard(pc)
@pc.on("track")
def on_track(track):
log_info("Track %s received", track.kind)
# send it back
# pc.addTrack(relay.subscribe(track))
saver.start(track)
@track.on("ended")
async def on_ended():
log_info("Track %s ended", track.kind)
saver.stop()
# handle offer
await pc.setRemoteDescription(offer)
# send answer
answer = await pc.createAnswer()
await pc.setLocalDescription(answer)
return web.Response(
content_type="application/json",
text=json.dumps(
{"sdp": pc.localDescription.sdp, "type": pc.localDescription.type}
),
)
def main():
app = web.Application()
app.on_shutdown.append(on_shutdown)
app.router.add_post("/offer", offer)
cors = aiohttp_cors.setup(app, defaults={
"*": aiohttp_cors.ResourceOptions(
allow_credentials=True,
expose_headers="*",
allow_headers="*",
)
})
for route in list(app.router.routes()):
cors.add(route)
web.run_app(
app, access_log=None, host="0.0.0.0", port=9999,
)
if __name__ == "__main__":
main()
code that runs on the camera phone
<html>
<head>
<meta charset="UTF-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<script defer src="https://konubinix.eu/ipfs/bafkreic33rowgvvugzgzajagkuwidfnit2dyqyn465iygfs67agsukk24i?orig=https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
<script src="https://konubinix.eu/ipfs/bafybeihp5kzlgqt56dmy5l4z7kpymfc4kn3fnehrrtr7cid7cn7ra36yha?orig=https://cdn.tailwindcss.com/3.4.3"></script>
<script src="https://unpkg.com/peerjs@1.5.2/dist/peerjs.min.js"></script>
<script>
document.addEventListener('alpine:init', () => {
Alpine.data('app', () => ({
pc: null,
message: "",
video: null,
dc: null,
peer: null,
conn: null,
peerId: null,
current_call: null,
wakelock: null,
width: {min: 320, max: 5000, value: 640},
currentwidth: null,
currentFrameRate: null,
configure: true,
bodyClass: "",
async init () {
this.video = document.getElementById("video")
this.peer = new Peer("camera", {host: "192.168.1.46", port: 9999, path: "/peerjs/"});
this.peer.on('open', (id) => {
this.message += `connected to peerjs with id ${id}`
this.peerId = id
});
if(! 'mediaDevices' in navigator) {
alert( 'Your browser does not support media devices.' );
}
await this.setupMediaStream()
this.peer.on('connection', async (conn) => {
this.message += "connected to the controller"
this.conn = conn
this.message += "calling the controller"
this.current_call = this.peer.call("control", this.mediaStream)
this.conn.on("open", async () => {
conn.send(JSON.stringify({type: "configure", value: this.pc === null ? true : false}))
})
conn.on('data', async (data) => {
this.message += `got ${data}`
const order = JSON.parse(data)
if(order.action === "start") {
await this.start(this.width)
conn.send(JSON.stringify({"type": "answer", "message": "started"}))
}
else if(order.action === "stop") {
await this.stop()
conn.send(JSON.stringify({"type": "answer", "message": "stopped"}))
}
else if(order.action === "setupMediaStream")
{
this.width.value = order.width
await this.setupMediaStream()
conn.send(JSON.stringify({"type": "answer", "message": "mediaStreamUpdate"}))
}
else {
this.message("Unrecognized")
}
});
});
try{
navigator.wakeLock.request("screen").then(
(w) => {
this.wakelock = w;
this.message += "wakelock acquired"
}
).catch((err) => {
this.message += `${err.name}, ${err.message}`;
});
} catch (err) {
this.message += err.toString()
};
},
createPeerConnection() {
this.pc = new RTCPeerConnection();
this.pc.addEventListener('track', (evt) => {
this.video.srcObject = evt.streams[0];
});
},
async setupMediaStream() {
this.mediaStream = await navigator.mediaDevices.getUserMedia(
{
video: {
width: { ideal: this.width.value },
height: { ideal: this.width.value },
frameRate: 30,
facingMode: {exact: "environment"},
},
audio: false,
}
);
// window.ms = this.mediaStream
this.currentwidth = this.mediaStream.getVideoTracks()[0].getSettings().width
this.currentFrameRate = this.mediaStream.getVideoTracks()[0].getSettings().frameRate
this.video.srcObject = this.mediaStream
if(this.conn !== null){
this.current_call.close()
this.current_call = this.peer.call("control", this.mediaStream)
}
},
async start() {
this.configure = false
this.bodyClass = "text-slate-700 bg-black"
this.createPeerConnection()
this.mediaStream.getTracks().forEach((track) => {
this.pc.addTrack(track, this.mediaStream);
});
await this.negociate()
},
async negociate() {
this.message += "\nneg start"
var offer = await this.pc.createOffer();
await this.pc.setLocalDescription(offer);
try{
var resp = await fetch('http://192.168.1.245:9999/offer', {
body: JSON.stringify({
sdp: offer.sdp,
type: offer.type,
}),
headers: {
'Content-Type': "application/json"
},
method: "POST"
});
} catch (err) {
alert(err.toString())
alert(JSON.stringify(err));
throw err
};
this.message += "\nneg sent"
var answer = await resp.json()
await this.pc.setRemoteDescription(answer);
this.message += "\nconnected"
},
async stop() {
// if (this.wakelock != null) {
// this.wakelock.release().then(() => {
// this.wakelock = null;
// // alert("released");
// }).catch((err) => {
// alert(`${err.name}, ${err.message}`);
// });
// }
// this.current_call.close()
this.pc.getTransceivers().forEach(function(transceiver) {
if (transceiver.stop) {
transceiver.stop();
}
});
this.pc.getSenders().forEach(function(sender) {
sender.track.stop();
});
// close peer connection
setTimeout(async () => {
this.pc.close();
this.pc = null
this.message += "\nall closed"
this.conn.send(JSON.stringify({"type": "answer", "message": "allclosed"}))
this.bodyClass = ""
this.configure = true
setTimeout(async () => {
this.message += "connecting back"
await this.setupMediaStream()
this.current_call = this.peer.call("control", this.mediaStream)
}, 500)
}, 500);
},
}))
})
</script>
</head>
<body x-data="app" :class="bodyClass">
<div>
<!-- <button @click="start">Start</button> -->
<!-- <button @click="stop">Stop</button> -->
<div x-text="peerId"></div>
<div x-text="message"></div>
<div x-show="configure">
<span x-text="width.value"></span>
<span x-text="currentwidth"></span>
<span x-text="currentFrameRate"></span>
<div class="justify-center sm:px-20 p-15 h-screen" x-data="{show: true}">
<button class="text-white bg-blue-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800" @click="show = !show ; $refs.video.scrollIntoView()" x-text="show ? 'Hide' : 'Show'"></button>
<video id="video" @dblclick="$event.target.requestFullscreen()" x-show="show" class="object-scale-down max-h-full" x-ref="video" poster="https://d1tobl1u8ox4qn.cloudfront.net/2018/05/3814a83b-a2d4-4dfd-8945-f1cd003eb16f-1920x1080.jpg" autoplay="true" playsinline="true"></video>
</div>
</div>
</div>
</body>
</html>
code that runs on the controlling interface, possibly another phone
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<script defer src="https://konubinix.eu/ipfs/bafkreic33rowgvvugzgzajagkuwidfnit2dyqyn465iygfs67agsukk24i?orig=https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
<script src="https://konubinix.eu/ipfs/bafybeihp5kzlgqt56dmy5l4z7kpymfc4kn3fnehrrtr7cid7cn7ra36yha?orig=https://cdn.tailwindcss.com/3.4.3"></script>
<script src="https://unpkg.com/peerjs@1.5.2/dist/peerjs.min.js"></script>
<script>
document.addEventListener('alpine:init', () => {
Alpine.data('app', () => ({
peer: null,
peerid: null,
conn: null,
message: "",
video: null,
configure: true,
width: {min: 320, max: 6000, value: 640},
async init() {
this.video = document.getElementById('videoElement')
this.peer = new Peer("control", {host: "192.168.1.46", port: 9999, path: "/peerjs/"})
this.peer.on('open', (id) => {
this.message += `\nconnected to peerjs with id ${id}`
this.peerid = id
});
this.peer.on('call', (call) => {
this.message += "in call"
call.answer();
call.on('stream', (stream) => {
this.message += "getting stream"
this.video.srcObject = stream;
});
});
this.connect()
},
connect() {
this.conn = this.peer.connect("camera")
this.conn.on('data', async (data) => {
this.message += `\nreceived ${data}`
const message = JSON.parse(data)
if(message.type === "answer")
{
if(message.message === "started")
{
this.configure = false
}
else if(message.message === "allclosed")
{
this.configure = true
}
}
else if(message.type === "configure")
{
this.configure = message.value
}
else
{
this.message += `I don't deal with message of type ${message.type}`
}
})
this.conn.on("open", async () => {
this.message += "\nconnected to the camera"
})
},
async start() {
this.conn.send(JSON.stringify({action: "start"}))
this.message += "\nsent start"
},
async stop() {
this.conn.send(JSON.stringify({action: "stop"}))
this.message += "\nsent stop"
},
async setupMediaStream () {
this.conn.send(JSON.stringify({action: "setupMediaStream", width: this.width.value}))
}
}))
})
</script>
</head>
<body>
<div x-data="app">
<div x-text="peerid"></div>
<div>
<button class="text-white bg-blue-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800" @click="connect">Connect</button>
<button class="text-white bg-blue-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800" @click="start">Start</button>
<button class="text-white bg-blue-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800" @click="stop">Stop</button>
</div>
<div x-text="message"></div>
<div x-show="configure">
<span x-text="width.value"></span>
<input class="w-full h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer dark:bg-gray-700"
type="range"
:min="width.min"
:max="width.max"
x-model="width.value"
@input.debounce.500ms="setupMediaStream"
>
</div>
<div class="justify-center sm:px-20 p-15 h-screen" x-data="{show: true}">
<button class="text-white bg-blue-700 hover:bg-blue-800 focus:ring-4 focus:ring-blue-300 font-medium rounded-lg text-sm px-5 py-2.5 me-2 mb-2 dark:bg-blue-600 dark:hover:bg-blue-700 focus:outline-none dark:focus:ring-blue-800" @click="show = !show ; $refs.video.scrollIntoView()" x-text="show ? 'Hide' : 'Show'"></button>
<video x-ref="video" x-show="show" @dblclick="$event.target.requestFullscreen()" class="object-scale-down max-h-full" id="videoElement" poster="https://d1tobl1u8ox4qn.cloudfront.net/2018/05/3814a83b-a2d4-4dfd-8945-f1cd003eb16f-1920x1080.jpg" autoplay></video>
</div>
</div>
</body>
</html>
now what?
In the server project, run
pdm run ./main.py
In the phone running the camera.
python3 -m http.server 9904
Or, if the phone is too old to install termux or whatever install servdroid
Then connect the phone to http://localhost:9904 . It needs to be localhost, because we need to get access to the stream and we cannot use https because the server is accessed over http.
On the controlling device, the only constraint is to use http, because the peerjs signaling server is available over http only.
You can also run the same python command if you’d like.
some more information
I played a bit with that setup.
- stopped working after 3 hours, while trying to discover
where my chicken get out of the fence,
the chromium running on my zte blade s6 froze and I had to try to close it and wait for the “wait or kill” suggestion from android,
In the end, it ran for about 10h, plugged on a 138000 mAh power bank before dying.
Notes linking here
- discover where my chicken get out of the fence
- go2rtc to use an android device as webcam
- use the camera with kivy on android (blog)