First Person View has become popular among RC enthusiasts. FPV is typically implemented with dedicated equipment such as PAL/NTSC analog video cameras, 2.4/5.8 GHz transmitters, and analog video goggles with built-in receivers. Such equipment can of course be mounted on a LEGO model. In this project, instead, we merge radio control and FPV functionality into the LEGO Mindstorms EV3 platform, leveraging its relatively powerful ARM9 system-on-chip, embedded Linux OS, USB capabilities and wireless connectivity. Digital video is received and displayed by a generic smartphone in a Google Cardboard-style headset.
As an illustration of the benefits of a fully digital and integrated system, we overlay telemetry data with just a few lines of Javascript code. In a legacy analog system this would typically require dedicated on-board hardware.
In the current implementation a Bluetooth gamepad connects directly to the EV3 brick, and MJPEG video from USB webcams is also transmitted over Bluetooth (HTTP/TCP/IP/BNEP) with adjustable trade-offs between image quality and frame rate. We discuss ways to improve range, latency and video quality.
As virtual reality applications become mainstream, medical authorities worldwide are developing guidelines for the safe use of immersive displays. See for example: https://www.oculus.com/warnings/.
These recommendations are not to be taken lightly. Low-quality, DIY or poorly configured head-mounted stereoscopic displays can exercise human binocular vision in ways that are completely unnatural, leading to long-lasting discomfort or even injury.
Children under 13 and anyone with impaired vision can still enjoy FPV on regular (non immersive) displays.
The tracked vehicle in the demo video is derived from the TRACK3R and GRIPP3R models from the Mindstorms EV3 base set (31313).
Cameras are mounted on a motorized turret which uses parts from the EV3 Education Expansion set (45560) and an extra EV3 medium motor (45503).
Most USB webcams are supported under Linux nowadays, but many chipsets are not publicly documented and their Linux driver are based on partial reverse-engineering of Windows drivers. Thus, we favour webcams which comply with the USB video device class (UVC) standard. Here is a list of common UVC webcams.
This project requires a webcam with built-in compression; we do not want to process raw video on the EV3 CPU. Inexpensive MJPEG webcams are now available, and it is easy to decode and display MJPEG streams in a web browser.
The webcam should be backward-compatible with USB 1.1, as the Host port on the side of the EV3 brick does not support USB 2.0.
To connect two cameras to the EV3 brick, a small USB hub is required.
We use a pair of Logitech C170 webcams (VGA, MJPEG) which were already known to work with ev3dev (see [AD4M4ST0R]). We stream at 120x160 mono by default to maximize frame rate, and the user can manually request a stereo snapshot at 2x480x640 when the vehicle is stationary.
FPV can be as simple as looking at a video feed on a tablet or smartphone, but a head-mounted display improves the feeling of immersion. While a monoscopic head mount (with left and right screens showing the same image) is sufficient for aerial imagery, a stereoscopic view is definitely useful when driving a ground vehicle in a cluttered environment.
Any clone of Google Cardboard should be suitable for this project. We use the Homido headset which has significant advantages over plain implementations:
- Inter-lens distance adjustable from about 57 mm to 64 mm (to match the user's IPD)
- Three lens-to-screen distance settings (useful for vision-impaired users)
- Adjustable lens-to-eye distance (to maximize the field of view)
- Wide lenses (1.4" vs 1")
- Conical mounts that bring the lenses close to the eyes without pressing against the nose
- Good airflow which prevents fogging of the lenses.
The device must support IP networking over Bluetooth, also known as BNEP or PAN or simply "Internet access" in the Bluetooth settings menu.
We currently use an old Samsung Galaxy S3 phone, but any recent Android device is likely to be suitable. Even iOS might work as well.
At the time of writing the best available device for mobile VR is probably the Samsung Galaxy Note 4 (2560x1440, 515 pixels per inch), but a device with a 1270x720 display is sufficient for a project with VGA cameras.
Our software currently supports the SIXAXIS, DualShock 3 and Move Navigation Controller. Second-hand units should be massively available and inexpensive in the next few years as newer gaming consoles reach the market.
brickfpv consists of small independent C programs (brickrc, v4l2http, sixpair) running on top of [EV3DEV], a Debian-based third-party operating system for the EV3 brick which unlocks its full potential as an embedded Linux platform. The programs can be configured and combined to handle various types of vehicles (tracked, steered), view configurations (single front camera, front/rear cameras, stereo cameras), camera gimbals (pan/tilt, pan only, fixed) and displays (stereoscopic HMD or regular screen).
Browse the source code: http://www.pabr.org/bricks/brickfpv/brickfpv-1.0/
brickrc is a successor to [BRICKHID]
with support for the motor drivers of ev3dev (/sys/class/tacho-motor/*).
Video is simply rendered as JPEG images in a web browser. Javascript code receives images from the EV3 brick over HTTP. CSS transforms rotate and scale the images to account for various camera mounting options.
The PlayStation gamepads require a special pairing procedure over USB before they can be used wirelessly. See [SIXLINUX] for background information.
We use ev3dev-jessie-2014-12-01, apparently the first release with user-friendly IP networking over Bluetooth.
On Linux, installation boils down to flashing a microSD card (1 GB or more):
xzcat ev3dev-jessie-2014-12-01.img.xz | sudo dd bs=4M of=/dev/sdX
(See http://www.ev3dev.org/docs/getting-started/#step-2-copy-the-image-on-to-the-sd-card for details.)
After inserting the microSD card and booting the EV3 brick, we need to connect it to a network. The simplest way is reportedly with a USB-to-ethernet dongle. The least expensive (but not so simple) way is to set up networking over USB between the EV3 and a Linux PC. See http://www.ev3dev.org/docs/getting-started/#step-4-setup-a-network-connection.
Once we have a root shell, we can install a few useful packages:
root@ev3dev:~# apt-get update root@ev3dev:~# apt-get install bluez-hcidump bluez-tools bridge-utils iptables netcat-traditional bc
Finally, we must tell the Bluetooth HID subsystem to let brickrc handle HID peripherals:
root@ev3dev:~# cp /lib/systemd/system/bluetooth.service /etc/systemd/system/bluetooth.service
Using e.g. vi or nano, edit /etc/systemd/system/bluetooth.service
to change the ExecStart line to:
ExecStart=/usr/lib/bluetooth/bluetoothd -P input
Then restart the Bluetooth subsystem: with the new configuration:
root@ev3dev:~# systemctl daemon-reload root@ev3dev:~# systemctl restart bluetooth.service
root@ev3dev:~# wget http://www.pabr.org/bricks/brickfpv/brickfpv-1.0.tar.gz root@ev3dev:~# tar zxvf brickfpv-1.0.tar.gz root@ev3dev:~# cd brickfpv-1.0
We need to pair the PS3 gamepad with the EV3 brick. So, connect the PS3 gamepad to the EV3 brick with a USB mini-B to Type-A cable before proceeding. Then:
root@ev3dev:~/brickfpv-1.0# ./brickfpv.init start root@ev3dev:~/brickfpv-1.0# tail -f /tmp/brickfpv.log
Unplug the gamepad and press its PS button. The four LEDs will flash briefly, then one will turn solid red. At this point the gamepad controls the motors.
In the smartphone:
- Enter the Bluetooth settings menu
- Scan for devices
- Select the device named "ev3dev"
- Select "Use for Internet access"
- Watch for a pop-up dialog on the LCD screen and confirm with the OK button.
- In the web browser, open http://192.168.0.1/
To start brickfpv automatically when the EV3 brick is powered up:
root@ev3dev:~/brickfpv-1.0# cp brickfpv.service /etc/systemd/system/ root@ev3dev:~/brickfpv-1.0# systemctl daemon-reload root@ev3dev:~/brickfpv-1.0# systemctl enable brickfpv.service
brickfpv will automatically pair any gamepad found on the USB bus when it starts. Plug the gamepad, turn the brick on, then wait at least until the LED turns solid green and the LCD shows the brickman menu before unplugging.
Look for error messages in /tmp/brickfpv.log.
Stop any lingering component of brickfpv:
root@ev3dev:~# cd /root/brickfpv-1.0 root@ev3dev:~/brickfpv-1.0# ./brickfpv.init stop
Run sixpair and brickrc manually from the command-line:
root@ev3dev:~/brickfpv-1.0# ./sixpair root@ev3dev:~/brickfpv-1.0# ./brickrc -v ps3_tank.config
then watch for messages when the PS button is pressed.
Check whether the gamepad tries to establish a Bluetooth connection:
root@ev3dev:~# hcidump
Enable IP networking over Bluetooth in NAP (Network Access Point) mode:
root@ev3dev:~# connmanctl tether bluetooth on
Run the video server manually from the command-line:
root@ev3dev:~/brickfpv-1.0# ./streamer.sh 81 "./v4l2http -c /dev/video0 -r 15 --mjpeg -w 160 -h 120 -v" & root@ev3dev:~/brickfpv-1.0# ./httpd.sh &
The traditional way to develop embedded software involves cross-compilation toolchains. For ev3dev, see https://github.com/ev3dev/ev3dev/wiki/Using-brickstrap-to-cross-compile-and-debug.
Alternatively, small programs (such as brickfpv) can be compiled directly on the EV3 brick.
ev3dev ships with compilers and interpreters for several exotic languages, but not for C. So we free some space and install gcc:
root@ev3dev:~# dpkg -P golang-go-linux-arm golang-go golang-src root@ev3dev:~# dpkg -P guile-1.8 guile-1.8-libs root@ev3dev:~# apt-get install gcc libusb-dev libbluetooth-dev
The ev3dev kernel configuration lacks a few features which might benefit this project. To enable them, the kernel (or at least certain loadable modules) must be recompiled.
Again, we can compile the whole Linux kernel on the EV3 brick. The initial build takes some time, but then individual drivers can be patched, recompiled and tested reasonably quickly. The only difficulty is that RAM is too small, so we add swap (requires a microSD card larger than 1 GB):
root@ev3dev:~# lvcreate ev3devVG --size 512M --name swap root@ev3dev:~# mkswap /dev/ev3devVG/swap root@ev3dev:~# swapon /dev/ev3devVG/swap
In theory BlueZ 5.12+ has support for PS3 controllers. For some reason this does not work. This is why brickfpv uses the old sixpair tool and raw L2CAP sockets.
brickfpv uses netcat-traditional and shell scripts as a lightweight HTTP server. This should probably be done with systemd sockets.
As described above, brickfpv merely replicates (and poorly so) the functionality of dedicated RC and FPV systems. However, having integrated all these functions digitally on a single computer, we can do much more:
Image processing: Collision avoidance, range estimation.
Simple autopilot: Navigate to a location that the user has selected on the video display.
- Autonomous behaviour: Visual navigation, mapping, recognizing and picking objects, etc.
For proper stereoscopic viewing, image pairs must be aligned to within a few pixels. Therefore we have to orient the cameras very carefully. Alternatively, cameras can be calibrated and images can be reprojected in software. This process is known as image rectification. Implementations can be found e.g. in [OPENCV]. It might be possible to perform reprojection in real-time on the GPU of a modern smartphone, and even compensate for HMD lens distortion in the same rendering pass.
Trying to stream from two /dev/video devices simultaneously on Linux often fails with ENOSPC (not enough bandwidth). This happens even on USB 2.0, regardless of resolution, frame rate and compression.
For uncompressed formats, modprobe uvc_video quirks=128 sometimes fixes the problem.
For compressed formats, there is currently no simple workaround. This discussion provides a patch. Note that this requires recompiling the uvc_video kernel module.
USB webcams with built-in H.264 compression are now available. On the one hand, H.264 is more efficient than MJPEG, and it handles not only video but also audio. On the other hand, H.264 decoders in smartphones tend to be proprietary and it is unclear whether two synchronized left/right streams could be displayed side by side easily.
According to this discussion, Logitech C920 webcams might be able to stream two Full HD streams at 30 fps over USB 1.1. But a 1 Mbit/s Bluetooth link could only carry two 360x180 streams at 5 fps. It is unclear whether the compression level can be adjusted.
Range could be extended by streaming video over WiFi instead of Bluetooth. Even with two webcams and a WiFi adapter all connected to the USB 1.1 port, bandwidth might be better than in the current implementation.
The Bluetooth joystick would connect to an app running on the smartphone, and commands would be relayed to the EV3 brick over WiFi.
[EV3DEV] Debian on LEGO MINDSTORMS EV3 . http://www.ev3dev.org/.
[AD4M4ST0R] AD4M4ST0R – um rover LEGO. http://ofalcao.pt/blog/pt/2014/ad4m4st0r-um-rover-lego.
[OPENCV] OpenCV (Open Source Computer Vision. http://opencv.org.
[BRICKHID] brickhid - Direct connection between Bluetooth gamepads and LEGO Mindstorms EV3 . http://www.pabr.org/bricks/brickhid/brickhid.en.html .
[SIXLINUX] Using the PlayStation 3 controller in Bluetooth mode with Linux . http://www.pabr.org/sixlinux/sixlinux.en.html .