December 21, 2023

• Found a Logitech gamepad, including an adapter (and fresh AA-batteries).
• Still no response from the front-end in RobotHub.
• Looked at the Apps in RobotHub. You could create your own app, there are two templates available. One is 'RAE Start' (which gets the name 'Payable Peach Crayfish' in RobotHub_, which contains a script which starts ros (ROS_DOMAIN_ID=30; ros2 launch rae_hw control.launch.py enable_battery_status:=false
• The readme states:
  # Video Streams
  
  This App uploads a DepthAI pipeline to RAE and streams each camera output to RobotHub.
  
  You can easily replace with you desired sockets that what to be streamed in `streams` dictionary. Also for configuring the pipeline you can edit `build_pipeline()`, such as resolution, fps and more.
  
  ## Requirements
  
  - RAE or a RVC3 device.
  
  ## Usage
  
  - Assign devices to the App and launch it, one stream will be started.
• The pipeline is build in src/app.py, which has an Application class with method on_start() and on_stop(). This should init five camera streams (left/right, front/back and high-res color).
• The app initialized the Robot class, which not only contains a ROS2Manager (handling the battery_status), but also a MovementController.
• Added "Hello World" at start, tried to upload this version (seems that still the template was used). Selected the RAE as robot (after restarting the robot, because connection was gone). Installed the app, although I couldn't select the default device (RAE, which is assigned to the Default app). Install is pending, so stopped the default app.
• The Install failed with an unknown error and no installation logs.

December 18, 2023

• Looked if I could get the robot controlled from my Window partition of nb-dual. The robot was connected via USB-C and visible in RobotHub.
• The Default App was running, so started the front-end. Also get a stream while using the Chrome-browser (under Windows at least!).
• With manual control, the arrow keys work.
• No response from the two gamepads. The MangDang controller is described SLAM guide.
• The pairing of the controller is described in Quick Start guide.
• The MangDang controller blinked (on top) before, but doesn't do it again.
• Tried a real joystick (Logitech Attack 3), but also no response.
• At least, I could change the color of the bottom-LEDs via this front-end.
• Lets try to do the same with the native or ROS-interface
• Could login via LAB42. The /home directory has three accounts. The robothub account contains the containers. The node-agent account is empty.
• At October 18 I was experimenting with rae-ros. The default branch is humble, although there is also a rae-robothub_base. Last change is 3 weeks ago.
• The robot is doing strange. On the front-end I saw low battery. I would expect that it receives power from the USB-C, but maybe not enough through an adapter. Will try to charge with my Dell 45W adapter.
• The display indicates that is connected to the robothub, but I don't see that. Also charging via the USB-C of the Dell-adapter doesn't work, I will try it now directly to my laptops USB-port. Could at least login again. Tried on the robot both docker ps and docker images list, but both returned an empty list. Yet, both /home/robothub/containers as /data/docker contains data (both from Oct 18 and Dec 18). The last call was wrong, docker images shows repository luxonis/rae-ros-robot with tag humble.
• Trying to run docker run -it --restart=unless-stopped -v /dev/:/dev/ -v /sys/:/sys/ --privileged --net=host luxonis/rae-ros-robot:humble fails because the device is busy. At the robot side robothub-ctl doesn't work. That is because the docker actually starts. When I exit this I could access /usr/bin/robothub-ctl again.
• Logged in manually via the USB-C as indicated here. Provided the /ect/wpa_supplicant.cfg with the settings of LAB42.
• The docker still gives the same error on the device, but starting ros2 launch rae_hw control.launch.py changes the bottom LEDs to purple. I see in the log also a control_manager and a battery_node running.
• I could start another shell with docker exec -it CONTAINER_ID zsh. Yet, no topics are visible. But in the environment no ROS_DOMAIN is specified, only ROS_LOCALHOST_ONLY=0.
• Checked ROS2 Environment configurations. Tried to set export ROS_DOMAIN_ID=. Set the same domain_id as the last number of my ipv4 address. Now ros2 topic list works:
  /audio_in
  /audio_out
  /battery_status
  /cmd_vel
  /diff_controller/odom
  /diff_controller/transition_event
  /dynamic_joint_states
  /joint_state_broadcaster/transition_event
  /joint_states
  /lcd
  /led
  /leds
  /parameter_events
  /robot_description
  /rosout
  /tf
  /tf_static
  
• Gave the command ros2 topic pub --once /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.1, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 1.0}}" which drove the wheels foreward!
• Switched to my Linux partition. No ros-humble on my Ubuntu 20.04. Looking into RoboStack.
• No mamba or micromamba installed, so start there. Will create mamba create -n ros_humble_env with mamba install ros-humble-ros-base.
• Failed because of some python2 dependency in anaconda2, so removed it from my path (which will give a conflict with Behavior-Based Robotics), and installed ros-humble-ros-base in the ros_humble_env.
• Set the ROS_DOMAIN_ID=40 to both the RAE and ros_humble_env. Now I see an extensive list of topics in ros_humble_env:
  audio_in
  /audio_out
  /battery_status
  /cmd_vel
  /diff_controller/odom
  /diff_controller/transition_event
  /dynamic_joint_states
  /joint_state_broadcaster/transition_event
  /joint_states
  /lcd
  /led
  /leds
  /parameter_events
  /robot_description
  /rosout
  /tf
  /tf_static
  
• Installed mamba install ros-humble-teleop-twist-keyboard and executed ros2 run teleop_twist_keyboard teleop_twist_keyboard with succes! Note, experimenting with the RAE on its back.
• As next step I installed mamba install ros-humble-teleop-twist-joy and executed ros2 launch teleop_twist_joy teleop-launch.py joy_config:='xbox'. I see only for the xbox-version that the input is coupled to /dev/input/js0 (with dmesg | tail). Still, no movement.
• When run without arguments with the Logitech Attack3, the device is recognized, because of the info:
  [teleop_node-2] [INFO] [1702918068.079061537] [TeleopTwistJoy]: Teleop enable button 8.
  [teleop_node-2] [INFO] [1702918068.079100994] [TeleopTwistJoy]: Turbo on button 10.
  [joy_node-1] [INFO] [1702918068.264146970] [joy_node]: Opened joystick: Logitech Attack 3. deadzone: 0.300000
  
• With ros2 launch teleop_twist_joy teleop-launch.py joy_config:='atk3' the RAE can be driven back and fort, when button 0 is activated. It doesn't turn unfortunatelly.
• Also the PS3 clone works (no joy_config needed, that is the default). Here the button to be activated is button 8, which is the select button. Still no turn.

December 15, 2023

• Started up RAE. Display indicates that it is connected to RoboHub. After a few minutes it seems to reboot.
• Looked at RAE-ROS, where I ended during the last session. There are a few updates from 2 weeks OK, but didn't do a pull (yet).
• Looked in RobotHub. My labtop is up, but keembay is seen 8 minutes ago (and my renamed RAE 2 months ago).
• Connected a USB-C and rebooted the robot. Now Keembay is up according to robohub.
• I can use the RAE device, which indicates Model 'RAE - 127.0.0.1'. Seems to indicate that the connection is build via the USB-cable.
• The web-terminal is not available on RAE.
• Restarted the Default app. Directly stops. This is v1.1.1, upgraded to v1.2.1.
• The Robohub Release note gives error 404, the Documentation link works. Yet, the ROS-integration page is TBD. Actually, the only page with information is about Luxonis OS.
• Was able to login via ssh with the USB-C cable. Could see that the RAE received an ip from LAB42, an could ssh. This same ip-address is also visible in the robot-overview dashboard:
  
• Couldn't find any app other than the RAE-Default App. Selected v1.2.1 to be installed. Start downloading this version and now it is running!
• Also the front-end of this app is working (using Chromium), displaying both the front and back camera:
  
• Could control the wheels with my arrow keys.
• Tried the Microsoft Joystick with USB-converter. No light shows up with the controller, so not seems to work. Checked with dmesg | tail, the Gamepad is recognized:
  [ 6267.779431] usb 3-4.3: Product: USB Gamepad
  [ 6267.786336] input: USB Gamepad as /devices/pci0000:00/0000:00:14.0/usb3/3-4/3-4.3/3-4.3:1.0/0003:0810:0003.000B/input/input35
  [ 6267.848545] hid-generic 0003:0810:0003.000B: input,hidraw5: USB HID v1.10 Joystick [USB Gamepad ] on usb-0000:00:14.0-4.3/input0
  
• Couldn't find another gamepad, other than the MangDang. Found a micro-USB with the tablet (from TomTom). Yet, this gamepad has diffulty to claim the input:
  7547.031723] usb 3-4.2: Manufacturer: Sony Computer Entertainment
  [ 7552.296090] sony 0003:054C:05C4.000D: failed to retrieve feature report 0x81 with the DualShock 4 MAC address
  [ 7552.296338] sony 0003:054C:05C4.000D: hidraw6: USB HID v81.11 Gamepad [Sony Computer Entertainment Wireless Controller] on usb-0000:00:14.0-4.2/input0
  [ 7552.296355] sony 0003:054C:05C4.000D: failed to claim input
  
• Time for lunch.

November 24, 2023

• Would be interesting to test Nerfstudio SLAM on the RAE.

October 19, 2023

• Tried to control the robot from my laptop. Installed ros-foxy-teleop-twist-keyboard, set ROS_DOMAIN_ID=0 bot at my laptop and the RAE, but it still not works.
• Running teleop_twist_keyboard from RAE doesn't work, because it is not installed (and apt-get doesn't work).
• Try again with mender, but that seems to be only for firmware.
• Note that /usr/bin in the RAE is quite poluted, including a Lena_Mask_512x512.raw
• According to the RAE HW, the host-computer is a Quad-core ARM A53 @ 1.5GHz, running Yocto Linux.
• Read this documentation, but seems that all installations have to be done when building the Yocto image.

October 18, 2023

• Tried to do step 4 (finishing up) from RAE deployment.
• The RAE robot indicates that it is connected to RobotHub. I can also see the robot in Wifiman (matched the MAC with the setting in iotroam). Wifiman indicates Intel Coporate as Manufacturer.
• Yet, the webbrowser with a Control interface rejected the connection.
• Sure that it is the RAE, because I could ssh to the same IP.
• If I check for port 80 (http) and port 443 (https), only port 80 gives an answer but only for the local machine:
  tcp 0 0 127.0.0.1:29900 0.0.0.0:* LISTEN 980/nanomq
  tcp 0 0 127.0.0.1:29901 0.0.0.0:* LISTEN 980/nanomq
• nginx -V is not installed, httpd works (but unclear where httpd.conf is located)
• The directory /home/robothub has containers.conf from com.luxonis.robothub.
• At the end of the advanced manual is indicated that the connection should be made to port 9010!
• Yet, I only get a _manage page, which shows the installed robothub perception apps.
• Yet, in robothub I now also have a web-terminal page. Only this gives 'waiting for device to connect'.
• Next to the Default App, I also installed the Demo App. Yet, when looking in logs, the launch crashes on "No module named 'robothub_core' "
• Installing another App fails because the Demo App has still the device assigned. Reassign Device doesn't work. Uninstalling this App.
• The Default App, which worked yesterday, now crashed on a ROS2 node in /lib/robothub/app.py (line 161).
• This is not a actual directory on the robot, so part of a container. Try again from Linux.
• Tried also under Linux. Still, the Default App crashes on /lib/robothub/app.py, even after a reboot of RAE.
• Also here I get that the WebInterface is not available on the RAE.
• Next step is getting started. The direct connection is the part which works.
• Looked at Running NN on rae. At least the /opt/openvino/tools/compile_tool is there. Yet, to do it I need a directory with a pretrained XML and BIN. As suggested, I downloaded a testset from coco128.zip and unpacked it in /data/assets/coco128. Yet, it seems that I missing a previous step.
• Better continue with rae-ros.
• Started with robothub-ctl stop. That also stops my ssh-session.
• Better to stop from robothub from my laptop.
• Checked the status at RAE. Both docker ps as docker images showed that no docker images are installed, nor running.
• First did git clone https://github.com/luxonis/rae-ros.git, but before building my own docker-image, first downloaded the latest image in this ~/git/rae-ros directory with docker pull luxonis/rae-ros-robot:humble
• After that, tried docker save luxonis/rae-ros-robot:humble | ssh -C root@192.168.197.55 docker load. This is a 5.81 GB image, so the usage of /data goes directly from 13% to 40%
• Did docker run -it --restart=unless-stopped -v /dev/:/dev/ -v /sys/:/sys/ --privileged --net=host luxonis/rae-ros-robot:humble, but this gives an error /ws/src/rae-ros/entrypoint.sh: line 5: echo: write error: Device or resource busy
• Attached in another shell to this container. At least ros2 topic list works.
• Next thing to try is ros2 launch rae_bringup robot.launch.py. The LEDs of the RAE become purple, and the small display shows a green bar:
  
• Opened another terminal and started a separate session with docker exec -it zsh. ros2 topic list now gives:
  /audio_in
  /audio_out
  /battery_status
  /cmd_vel
  /diagnostics
  /diff_controller/odom
  /diff_controller/transition_event
  /dynamic_joint_states
  /joint_state_broadcaster/transition_event
  /joint_states
  /lcd
  /led
  /leds
  /odometry/filtered
  /parameter_events
  /rae/imu/data
  /rae/left_back/camera_info
  /rae/left_back/image_raw
  /rae/left_back/image_raw/compressed
  /rae/left_back/image_raw/compressedDepth
  /rae/left_back/image_raw/theora
  /rae/right/camera_info
  /rae/right/image_raw
  /rae/right/image_raw/compressed
  /rae/right/image_raw/compressedDepth
  /rae/right/image_raw/theora
  /rae/stereo_back/camera_info
  /rae/stereo_back/image_raw
  /rae/stereo_back/image_raw/compressed
  /rae/stereo_back/image_raw/compressedDepth
  /rae/stereo_back/image_raw/theora
  /rae/stereo_front/camera_info
  /rae/stereo_front/image_raw
  /rae/stereo_front/image_raw/compressed
  /rae/stereo_front/image_raw/compressedDepth
  /rae/stereo_front/image_raw/theora
  /robot_description
  /rosout
  /set_pose
  /tf
  /tf_static
  
• The command ros2 pkg list gives:
  ackermann_msgs, ackermann_steering_controller, action_msgs, actionlib_msgs, admittance_controller, ament_cmake, ament_cmake_auto, ament_cmake_copyright, ament_cmake_core, ament_cmake_cppcheck, ament_cmake_cpplint, ament_cmake_export_definitions, ament_cmake_export_dependencies, ament_cmake_export_include_directories, ament_cmake_export_interfaces, ament_cmake_export_libraries, ament_cmake_export_link_flags, ament_cmake_export_targets, ament_cmake_flake8, ament_cmake_gen_version_h, ament_cmake_gmock, ament_cmake_gtest, ament_cmake_include_directories, ament_cmake_libraries, ament_cmake_lint_cmake, ament_cmake_pep257, ament_cmake_pytest, ament_cmake_python, ament_cmake_ros, ament_cmake_target_dependencies, ament_cmake_test, ament_cmake_uncrustify, ament_cmake_version, ament_cmake_xmllint, ament_copyright, ament_cppcheck, ament_cpplint, ament_flake8, ament_index_cpp, ament_index_python, ament_lint, ament_lint_auto, ament_lint_cmake, ament_lint_common, ament_package, ament_pep257, ament_uncrustify, ament_xmllint, angles, audio_msgs, backward_ros, behaviortree_cpp_v3, bicycle_steering_controller, bond, bondcpp, builtin_interfaces, camera_calibration, camera_calibration_parsers, camera_info_manager, class_loader, cliff_detector, common_interfaces, composition_interfaces, compressed_depth_image_transport, compressed_image_transport, console_bridge_vendor, control_msgs, control_toolbox, controller_interface, controller_manager, controller_manager_msgs, costmap_queue, cv_bridge, depth_image_proc, depth_sensor_pose, depthai-ros, depthai_bridge, depthai_descriptions, depthai_examples, depthai_filters, depthai_ros_driver, depthai_ros_msgs, diagnostic_msgs, diagnostic_updater, diff_drive_controller, domain_coordinator, dwb_core, dwb_critics, dwb_msgs, dwb_plugins, effort_controllers, eigen3_cmake_module, fastrtps_cmake_module, filters, force_torque_sensor_broadcaster, forward_command_controller, foxglove_bridge, foxglove_msgs, generate_parameter_library, generate_parameter_library_py, geographic_msgs, geometry_msgs, gst_bridge, gst_pipeline, hardware_interface, image_geometry, image_pipeline, image_proc, image_publisher, image_rotate, image_transport, image_transport_plugins, image_view, imu_filter_madgwick, imu_sensor_broadcaster, interactive_markers, ira_laser_tools, joint_state_broadcaster, joint_state_publisher, joint_trajectory_controller, kdl_parser, kinematics_interface, laser_geometry, laserscan_kinect, launch, launch_ros, launch_testing, launch_testing_ament_cmake, launch_testing_ros, launch_xml, launch_yaml, libcurl_vendor, libstatistics_collector, libyaml_vendor, lifecycle_msgs, map_msgs, message_filters, nav2_amcl, nav2_behavior_tree, nav2_behaviors, nav2_bringup, nav2_bt_navigator, nav2_collision_monitor, nav2_common, nav2_constrained_smoother, nav2_controller, nav2_core, nav2_costmap_2d, nav2_dwb_controller, nav2_lifecycle_manager, nav2_map_server, nav2_mppi_controller, nav2_msgs, nav2_navfn_planner, nav2_planner, nav2_regulated_pure_pursuit_controller, nav2_rotation_shim_controller, nav2_rviz_plugins, nav2_simple_commander, nav2_smac_planner, nav2_smoother, nav2_theta_star_planner, nav2_util, nav2_velocity_smoother, nav2_voxel_grid, nav2_waypoint_follower, nav_2d_msgs, nav_2d_utils, nav_layer_from_points, nav_msgs, navigation2, octomap, octomap_msgs, ompl, orocos_kdl_vendor, osrf_pycommon, parameter_traits, pcl_conversions, pcl_msgs, pcl_ros, pluginlib, pointcloud_to_laserscan, position_controllers, python_cmake_module, python_qt_binding, qt_gui, qt_gui_cpp, rae_bringup, rae_camera, rae_description, rae_hw, rae_msgs, range_sensor_broadcaster, rcl, rcl_action, rcl_interfaces, rcl_lifecycle, rcl_logging_interface, rcl_logging_spdlog, rcl_yaml_param_parser, rclcpp, rclcpp_action, rclcpp_components, rclcpp_lifecycle, rclpy, rcpputils, rcutils, realtime_tools, resource_retriever, rmw, rmw_dds_common, rmw_fastrtps_cpp, rmw_fastrtps_shared_cpp, rmw_implementation, rmw_implementation_cmake, robot_localization, robot_state_publisher, ros2_control_test_assets, ros2_controllers, ros2action, ros2cli, ros2cli_common_extensions, ros2component, ros2doctor, ros2interface, ros2launch, ros2lifecycle, ros2multicast, ros2node, ros2param, ros2pkg, ros2run, ros2service, ros2topic, ros_core, ros_environment, ros_workspace, rosapi, rosapi_msgs, rosbridge_library, rosbridge_msgs, rosbridge_server, rosgraph_msgs, rosidl_adapter, rosidl_cli, rosidl_cmake, rosidl_default_generators, rosidl_default_runtime, rosidl_generator_c, rosidl_generator_cpp, rosidl_generator_py, rosidl_parser, rosidl_runtime_c, rosidl_runtime_cpp, rosidl_runtime_py, rosidl_typesupport_c, rosidl_typesupport_cpp, rosidl_typesupport_fastrtps_c, rosidl_typesupport_fastrtps_cpp, rosidl_typesupport_interface, rosidl_typesupport_introspection_c, rosidl_typesupport_introspection_cpp, rpyutils, rsl, rtabmap_conversions, rtabmap_msgs, rtabmap_odom, rtabmap_slam, rtabmap_sync, rtabmap_util, rviz_assimp_vendor, rviz_common, rviz_default_plugins, rviz_imu_plugin, rviz_ogre_vendor, rviz_rendering, sensor_msgs, sensor_msgs_py, shape_msgs, slam_toolbox, smclib, spdlog_vendor, sros2, sros2_cmake, statistics_msgs, std_msgs, std_srvs, steering_controllers_library, stereo_image_proc, stereo_msgs, tango_icons_vendor, tcb_span, tf2, tf2_eigen, tf2_geometry_msgs, tf2_kdl, tf2_msgs, tf2_py, tf2_ros, tf2_ros_py, tf2_sensor_msgs, theora_image_transport, tinyxml2_vendor, tinyxml_vendor, tl_expected, tracetools, tracetools_image_pipeline, trajectory_msgs, tricycle_controller, tricycle_steering_controller, uncrustify_vendor, unique_identifier_msgs, urdf, urdf_parser_plugin, velocity_controllers, vision_msgs, visualization_msgs, xacro, yaml_cpp_vendor,
• The command ros2 pkg executables | grep rae gives:
  rae_bringup audio_spectrum.py
  rae_bringup battery_status.py
  rae_bringup scan_repub.py
  rae_bringup sound_test.py
  rae_camera camera
  rae_camera perception_ipc_rtabmap
  rae_camera stereo_depth
  rae_camera stereo_pair
  rae_camera yolo
  rae_hw test_encoders
  rae_hw test_max_speed
  rae_hw test_motors
  rae_hw test_speed
  
• Tried ros2 run rae_bringup battery_status.py, but I see no output.
• The command ros2 run rae_hw test_motorsi gave:
  Starting test procedure. Duration: 5s.
  Motor speeds - L: 16 R: 16 rad/s.
  Enc ratios - L: 187 R: 187 counts/rev.
  Max speeds - L: 32 R: 32 rads/s.
  terminate called after throwing an instance of 'std::system_error'
  what(): error requesting GPIO lines: Device or resource busy
  [ros2run]: Aborted
  
• Running ros2 topic pub /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.1, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 1.0}}" drives the wheels as expected forward.
• Finished the docker with an exit, followed with a shutdown now

October 16, 2023

• The RAE has finally arrived. Still have to unpack. Should look at getting started.
• Next steps would be RAE ROS, which is a ROS2 Humble docker image.
• Luxonis recomments to use the RobotHub Cloud first, although they also have an advanced manual.
• The RAE is RCV3 based, so algorithms can run on the ARM. Several SLAM algorithms are implemented as SoM container.
• Started the RAE. The screen is on the back, together with the High Resoluton camera.
• Reset my RobotHub password. As robot I had my XPS-13-9300. Connecting to vislab fails. Restarting RAE as suggested also didn't work.
• The XPS indicates that there is an update available, but once I upgrade I get a server error. Did the upgrade of the Agent manually, with the command provided in Connect robot. Still no connection, until I did robothub-ctl cloud connect which gave Joined team Intelligent Robotics Lab!.
• The OAK-D is also visible, but live-stream only works with Chrome (which freezes the computer).
• Now at least robothub-ctl status indicates DepthAI 2.21.2, and that I am connected to the cloud.
• I was connected via Lab42, switching to vislab. Yet, doesn't seem to work.
• In Troubleshooting, they point to step 2 of the advanced setup manual.
• Used here the connection via USB-C. Found the MAC address. Added the device to iotroam. Yet, still the RAE doesn't react on the QR-code. Also doesn't want to reboot.
• Forced it to reboot. Now it reacts on the QR-code and connects to LAB42.
• Yet, directly after connecting it rebooted again. Still connected. In RobotHub this robot is called keembay, with as Perception-App RAE - Default
• Renamed keembay to RAE-#. Yet, the RAE indicates that the Install failed. Details shows an Error: App is not build. Selected a new version 1.1.1, which is now a download in progress. After the download the App is running. Only don't see the control as specified.
• Switched to Chromium, which shows the Live View:
  

October 12, 2023

Previous Labbooks

• AMR Labbook2023.html
• AMR Labbook2022.html
• AMR Labbook2020.html
• AMR Labbook2018.html
• AMR Labbook2017.html
• AMR Labbook2016.html
• AMR Labbook2015.html
• AMR Labbook2014.html
• AMR Labbook2013.html
• AMR Labbook2012.html
• Probabilistic Robotics 2012
• AMR Labbook2010.html
• Labbook2009.html
• Labbook2008.html
• Labbook2007.html