Week 8. Perception

Perception is organisation, identification and interpretation of sensory (sound, haptic, speech, etc) information to represent and understand the environment.

Visual perception is the ability to interpret the surrounding environment using colour, scotopic, or mesopic vision using light in visible spectrum.

2D Perception	3D Perception
Usually works on RGB	Work on depth maps or point clouds
Provides texture and colour	Usually without texture or colour
Is affected by lighting conditions	Not really affected by lighting conditions
Pixels represent colour	Pixels represent distance

There are different ways we can sense depth:

• Stereo vision: two eyes allow us to increase FOV and perceive depth Computes depth by binocular disparity. (difference in image location of an object as seen by the left and right eyes)
• Structured light: cast a known infra-red pattern on the scene, the camera processes the distorted projected pattern. Observed distortion is used to compute the depth. This has some issues in direct / strong sunlight. (used by Xbox Kinect v1)
• Time of flight: emit infra-red light, compute time it takes to return to the sensor Closer objects will have shorter time of flight than those far away. The LiDAR (light detection and ranging) sensors use this method. (used by Xbox Kinect v2)

Depth registration is the process by which depth and RGB information is registered in order to represent points in the real world. We typically create a 3D RGBD point cloud.

YOLO: Image Recognition (“You Only Look Once”)

YOLO is a real-time object recognition method for recognising multiple objects in a single image. Based on a single Convolutional Neural Network (CNN). It allows you to detect a bounding box around objects and classify them.

YOLOv4 install instructions are included in container and slides.

Working with the camera in ROS and OpenCV2

Subscribe to the camera:

from sensor_msgs.msg import Image
 
# from a webcam
rospy.Subscriber("/usb_cam/image_raw", Image, callback)
 
# from robot's sensors (i.e. HSR)
rospy.Subscriber("/hsrb/head_rgbd_sensor/rgb/image_color", Image, callback)

Create an OpenCV2 image:

from cv_bridge import CvBridge
 
# cv_bridge = CvBridge(...)
 
def callback(img: Image):
	cv_image = cv_bridge.imgmsg_to_cv2(img, desired_encoding='passthrough')

Subscribers / services run on different threads in Python.

As opposed to C++, where it generally runs on one thread. We need to be careful with handling cv_image to avoid race conditions.

Populate usb_cam with dummy data:

roslaunch usb_cam usb_cam-test.launch

We can use a GUI tool to view image outputs:

rosrun rqt_image_view rqt_image_view

Interacting with YOLO for object detection

We first need to create a service type for object detections, this may be empty for now. The response will include all the detections.

# my_pkg/srv/YOLOLastFrame.srv
---
my_pkg/YOLODetection[] detections

We also need to define this new message YOLODetection:

# my_pkg/msg/YOLODetection.msg
string name
float32 confidence
uint32 bbox_x
uint32 bbox_y
uint32 width
uint32 height

Now we can begin to write the service:

# declare the service
rospy.Service('/detect_frame', YOLOLastFrame, yolo_service)
 
# import service classes
from my_pkg.srv import YOLOLastFrame, YOLOLastFrameResponse
from my_pkg.msg import YOLODetection
 
# import cv2
import cv2
 
# import detector
from yolov4 import Detector
 
# initialise YOLO detector
detector = Detector(gpu_id=0, config_path='/opt/darknet/cfg/yolov4.cfg',
					weights_path='/opt/darknet/yolov4.weights',
					lib_darknet_path='/opt/darknet/libdarknet.so',
					meta_path='<PATH>/catkin_ws/src/my_pkg/config/coco.data')
 
# create the service
def yolo_service(request):
	res = YOLOLastFrameResponse()
	if self.cv_image is not None:
		# unpack image width
		cv_width, cv_height, _ = self.cv_image.shape
		
		# reize all images to network size
		img_arr = cv2.resize(self.cv_image, (
			detector.network_width(),
			detector.network_height()
		))
		
		# run YOLO
		detections = self.detector.perform_detect(
			image_path_or_buf=img_arr, show_image=True)
		
		for detection in detections:
			# create detection object
			d = YOLODetection(detection.class_name, detection.class_confidence,
				detection.left_x, detection.top_y,
				detection.width, detection.height)
			
			# convert the bounding box to image space
			d.bbox_x = int(d.bbox_x / detector.network_width() * cv_width)
			d.bbox_y = int(d.bbox_y / detector.network_height() * cv_height)
			d.width = int(d.width / detector.network_width() * cv_width)
			d.height = int(d.height / detector.network_height() * cv_height)
			
			# append to list
			res.detections.append(d)
		
	return res

Further Notes

Point clouds are also useful and widely used for 3D perception. Not covered in module, we can use PointCloud Library already available in ROS for this.

Recording / Playing RealSense data

roslaunch realsense2_camera demo_pointcloud.launch

rosbag record /camera/depth/color/points

rosbag play x.bag /camera/depth/color/points:=/a