May 30, 2023

• The students have finished Poster. Their report will follow later this week.

May 15, 2023

• Kimera-Semantics is semantic-SLAM algorithm which is part of a whole architecture for multi-robot exploration.
• The NVIDIA Isaac ROS Developer Preview 3 has localization package which converts both laser-scans and RGB-D to flatscans, which are localized on a grid (with covariance). Seems to run on a Orin Nano, no performance given for a Jetson Nano.
• How to register Objects of Interest is described in section 3.4 of Hector Open Source Modules for Autonomous Mapping and Navigation with Rescue Robots
• Love to experiment with Hector image projection package.
• Part of Hector grapher, is code to register a landmark (in C++), which is unfortunatelly not further described in the HectorGrapher paper (2021).
• It seemed that Emily got ar_track_alvar package working, although it quite old. The master=branch is achient (11y), but it has both a ros2 and noetic-dev branch.

May 2, 2023

• It would be interesting to try the code of 3D Scene reconstruction with the Manhattan-world assumption for this challenge. It seem to outperform DeepLabV3+, which the students currently are exploring. It is based on a conda environment.

April 24, 2023

• Continue with the installation of benchbot on ws10 (Ubuntu 22.04).
• Moved the benchbot-directory from ~/git/benchbot to /opt/benchbot
• Changed the group ownership with sudo chgrp -R docker benchbot/
• All requirements are fullfilled, the installation continues with building the docker containers.
• The docker containers are: Benchbot Simulator 'sim_omni', BenchBot Robot Controller, BenchBot API, BenchBot ROS messages.
• Encountered again the docker regular user error. Rebooted, and the process continues to build the containers, but maybe I should also install buildx as on April 21 on ws8.
• For buildx I can follow the steps of docker ubuntu installation.
• Simply only used the command sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin, followed by sudo apt-get upgrade (38 packages to be installed). The docker-buildx-plugin is now version 0.10.4-1.
• The sim_omni docker container is build, but fails on unable to open display ":1". Reboot.
• Seems that this is related that I was still running as super-user. Changes to a regular account. Checked benchbot_run -u, which shows that the code has to be updated.
• Fails on pip_check. As super-user python3 -m pip --version gives pip 23.1.1, while as normal user I get the error 'pip' is a package and cannot be directly loaded.
• Both run the same versions of python3 (v 3.10.6), but the pip3 install --upgrade pip installed it in ~/.local/lib/python3.10/site-packages, which I don't have as regular user.
• Copied the pip-package from super-user site-packages to my local site-packages. That solves this check. In addition, the Sim Python screen appears, although I see some problems with isaac-sim in running the simulation_app.py from isaac-sim. Still, it is waiting on the ray tracing shaders and continues with the next containers.
• The last error is the missing benchbot_hosts in /etc/hosts. Added the following benchbot specific hosts:
  172.20.0.100 benchbot_ros
  172.20.0.101 benchbot_robot
  172.20.0.102 benchbot_supervisor
  172.20.0.200 benchbot_debug
  
• The command ./benchbot_run -u gives out-of-date, which can be solve with suggested git config --global -add safe.directory. The miniroom shows up.
• Note that the directories addons, api and eval are no from me as normal user, while the others are from the super user. Changed also for those three directories the group to docker, so that the other users can modify the content.
• Note that Michelle now gets an error message on the docker-ce version on ws7, so check if there is a conflict.
• Looked into Michelle's problem, but she is getting the docker-ce error inside a container, which had indeed no docker-ce installed. Solved that problem (although her setup still gives a [[gpu]] error).

April 21, 2023

• Nvidia has released NVIDIA Isaac ROS Developer Preview 3, which includes NITROS (NVIDIA Isaac Transport for ROS)
• Another interesting for this CV2 project is Isaac ROS Map Localization, which can automatically estimates its initial position. This package includes utilities like PointCloud to FlatScan and ROS FlatScan to LaserScan.
• Next step would be Isaac ROS Occupancy Grid Localizer Nav2 Isaac Sim Tutorial.
• All this is intended for ROS 2 Humble.
• Trying to install benchbot on ws7, but it seems that I am a loop there (uninstalling nvidia-driver-525, and installing it again).
• Problem is the PPA-check:
  NVIDIA driver from a standard PPA: PPA is invalid
  
  The installed NVIDIA driver was detected to have come from somewhere other
  than the NVIDIA / Ubuntu graphics-drivers PPAs. The driver was sourced from:
  'http://archive.lambdalabs.com/ubuntu/pool/main/n/nvidia-graphics-drivers-525/nvidia-driver-525_525.105.17-0lambda0.20.04.1_amd64.deb'
  
  Our Docker container only supports official drivers, & GPUs only work in
  containers if the driver version exactly matches the version of the host.
  
• Commented out this test, but the next step fails on missing the 530 drivers
• Tried this uninstallation instructions from Lambda Stack.
• That didn't help.
• Instead, went to /etc/apt and removed lambda from the resources (and the preference.d). That worked.
• The installation now holds on starting the dockers. I added myself to the group docker, but that only works after reboot.
• The docker checks if a docker can be started as regular user. In the commnadline run hello-world works. The complain is the option -t. The script seems to use b=$(docker run --rm -t '$DOCKER_TAG_CORE' /bin/bash -c "$fn" | \ sed '"'"'s/[[:space:]]*$//'"'"'). In bin/.helpers you can see DOCKER_TAG_CORE="benchbot/core:base".
• When I run docker run --rm -t "benchbot/core:base" /bin/bash I get the error:
  Unable to find image 'benchbot/core:base' locally
  docker: Error response from daemon: pull access denied for benchbot/core, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
  
• The command that goes wrong seems to be docker buildx build -t "$DOCKER_TAG_CORE" -f "$PATH_DOCKERFILE_CORE" \ --build-arg TZ=$(cat /etc/timezone) \ --build-arg NVIDIA_DRIVER_VERSION="${nvidia_driver_version}" \ --build-arg CUDA_DRIVERS_VERSION="${cuda_drivers_version}" \ --build-arg CUDA_VERSION="${cuda_version}" $PATH_ROOT && \ build_ret=0 || build_ret=1
• Checked the docker buildx installation. The plugin should be in ~/.docker/cli-plugins
• It seems that the docker installation wasn't complete, and some cli-functionalilty is missing. Followed the steps of docker ubuntu installation.

April 20, 2023

• I read benchbot_addons management page.
• Installation of the add-ons manager should be possible with pip3 install .
• Printing the local addons folder can be done with:
  from benchbot_addons import manager as bam
  
  print(bam.local_addon_path())
  
• Looked into the official addons, such as examples_base. The hello_benchbot is a python-script that lets a robot drive forward and backbward (following a line). This implemented with an action sequence containing three commands (move_distance 1m, move_angle 90deg, move_angle 90deg)
• Read the nvidia-blog on benchbot.
• Issac Sim itself also has tutorials, such as hello robot. Control is on joint-level, while the benchbot could be controlled on move_distance-level.
• Omniverse has a troubleshoot page with gpu-problems. Suggestion is to explictly export DISPLAY=:0.0 when encountering a GLFW initialization error.
• Did pip3 install . as normal user in /opt/benchbot/addons. Now benchbot_run -u works.
• The python-script above gives as local addons folder:/opt/benchbot/addons/benchbot_addons/.local/my_addons
• Also tried to run print(bam.addon_path()), which gives addon_path() missing 2 required positional arguments: 'repo_user' and 'repo_name'.
• When I run print(bam.addon_path('benchbot-addons','examples_base')) I get:
  ~/.local/lib/python3.10/site-packages/benchbot_addons/benchbot-addons/examples_base
• This directory actually doesn't exist. The directory ~/.local/lib/python3.10/site-packages/benchbot_addons only contains the manager.py
• Checked the same directory for the superuser. There the benchbot_addons not even exists, although benchbot-api.egg-link does.
• Checked bam.exists('examples', [('name', 'hello_benchbot')]), which returned false.
• Checked out this github repository. Copied the examples directory to /opt/benchbot/addons/benchbot_addons/.local/my_addons/examples. Running python3 .hello_benchbot/hello_benchbot gives from benchbot_api import BenchBot: No module named 'benchbot_api'. Doing python3 -m pip install benchbot_api as normal user gives No matching distribution found. As superuser it gives Requirement already satisfied: benchbot_api in /opt/benchbot/api.
• Went as normal user to /opt/benchbot/api and did pip3 install .. Now python3 .hello_benchbot/hello_benchbot gives aiting to establish connection to a running supervisor ... .
• Yet, running benchbot_run –list-examples --robot carter_omni --environment miniroom:1 --task semantic_slam:passive:ground_truth gives xhost: unable to open display ":1". Also export DISPLAY=:0.0 doesn't help. Trying a reboot.
• After reboot benchbot_run -u still works.
• Yet, benchbot_run --list-robots indicates that no robot is installed.
• The command benchbot_install --list-addons shows all the official addons that can be installed.
• Followed the using the add-ons manager instructions and opened a python3 prompt, and gave the commands from benchbot_addons import manager as bam; bam.install_addons('benchbot-addons/robots_sim_omni')
• Now benchbot_run --list-robots gives now carter_omni, as indicated in repository.
• Continued with python3 command bam.install_addons('benchbot-addons/envs_bear_challenge_sim_omni') that gives environments appartment, company and office (all 1:5)
• Continued with python3 command bam.install_addons('benchbot-addons/envs_bear_develop_sim_omni') that adds environments house and miniroom (all 1:5)
• Continued with python3 command bam.install_addons('benchbot-addons/tasks_data_collect') that gives tasks data_collect:active:dead_reckoning, data_collect:active:ground_truth, data_collect:passive:ground_truth
• Continued with python3 command bam.install_addons('benchbot-addons/tasks_ssu') that gives tasks scd:active:dead_reckoning till semantic_slam:passive:ground_truth.
• This is enough to start the simulation as normal user with command benchbot_run --robot carter_omni --env miniroom:1 --task semantic_slam:active:ground_truth.
• Also installed bam.install_addons('benchbot-addons/examples_base'), went to ~/.local/lib/python3.10/site-packages/benchbot_addons/benchbot-addons/examples_base/examples, run python3 .hello_benchbot/hello_benchbot.
• The result is impressive:
  

April 19, 2023

• Simulation didn't start, but some installations needed a reboot.
• After the reboot the simulator started.
• The command benchbot_run –list-examples, as suggested on the challenge page didn't work (unknown robot). Also environment and task where empty.
• The command benchbot_run –list-examples --robot carter_omni --environment miniroom:1 --task semantic_slam:passive:ground_truth worked, but seems to start a full simulation (while the old was still running).
• benchbot_run is installed at /usr/local/bin/benchbot_run, so should be also be callable as normal user. Should look for option by calling --help.
• Checked with benchbot_run -u: BenchBot is up-to-date.
• Checked the version with benchbot_run -v:
  Version '__unnamed__', from branch 'master'
  (f7573674da2905b12d62345721595de7f26a777e)
• Should try to run benchbot_run --robot carter_omni --env miniroom:1 --task semantic_slam:active:ground_truth as normal user.
• Stopped the environment still running with benchbot_run -k
• Tried to run as normal user, but the /usr/local/bin/benchbot_* scripts are links to the git/benchbot directory of the super user. Moved that directory to /opt/benchbot, and updated the symbolic links /usr/local/bin/benchbot_*.
• That seems to work, although it fails that it gets no rights on Display :2.
• Rebooted. Now I have another problem:
  ModuleNotFoundError: No module named 'benchbot_addons'
  Robot 'carter' is not a supported robot. Please check '--list-robots'.
• Yet, this benchbot_addons exists, including a directory /opt/benchbot/addons/benchbot_addons/benchbot-addons/examples_base/ with three base examples :
  - hello_active
  - hello_benchbot
  - hello_passive
  
• Problem seems to be that there was also a link: ls /usr/local/bin/addons. Also did a recursive update of the group ownership of all the files in /opt/benchbot
• Trying if it works after a reboot.
• Still the same. /opt/benchbot/bin/addson is actually script (should be called benchbot_addson when installed in /usr/local/bin!).
• Note that benchbot_run uses a environment variable PATH_ADDONS_INTERNAL
• Those variables are defined in /opt/benchbot/bin/.helpers.
• That seems to work, except this error:
  Creating shared network 'benchbot_network':
  7f87b6d586660b5c94da3ed00f9fa1341fab7ee3394c7c986abea6545664dfcd
  Authorization required, but no authorization protocol specified
  xhost: unable to open display ":1"
• Rebooted, but add-ons problem is back:./benchbot_run -u:
  Checking installed BenchBot add-ons are up-to-date ... Traceback (most recent call last):
  File "", line 1, in
  ModuleNotFoundError: No module named 'benchbot_addons'
  
• Note that the .helpers script uses a PATH_CACHE.
• Making this member of the docker group didn't help.
• I should read benchbot_addons management page tomorrow.

April 17, 2023

• Trying to reproduce the progress of yesterday on nb-dual (with eGPU).
• Install script starts to build the docker images.
• Isaac Sim starts up, but again only with RTX loading. Some red error messages can be found in the log. One is the same as for the students, with the message that a Titan X is not a supported device:
  2023-04-17 13:34:03 [411ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-RTX GPU: NVIDIA GeForce GTX TITAN X
  
  |---------------------------------------------------------------------------------------------|
  | Driver Version: 530.30.02 | Graphics API: Vulkan
  |=============================================================================================|
  | GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
  | | | | | | Device-ID | UUID |
  |---------------------------------------------------------------------------------------------|
  | 0 | NVIDIA GeForce GTX TITAN X | | | 12534 MB | 10de | 0 |
  | | | | | | 17c2 | 910fb8f5.. |
  |=============================================================================================|
  | OS: Linux b7dcb254e72d, Version: 5.15.0-69-generic
  | XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
  | Processor: Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz | Cores: Unknown | Logical: 8
  |---------------------------------------------------------------------------------------------|
  | Total Memory (MB): 15582 | Free Memory: 13187
  | Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
  |---------------------------------------------------------------------------------------------|
  2023-04-17 13:34:03 [412ms] [Error] [gpu.foundation.plugin] No device could be created. Some known system issues:
  - The driver is not installed properly and requires a clean re-install.
  - Your GPUs do not support RayTracing: DXR or Vulkan ray_tracing, or hardware is excluded due to performance.
  - The driver cannot enumerate any GPU: driver, display, TCC mode or a docker issue. For Vulkan, test it with Vulkaninfo tool from Vulkan SDK, instead of nvidia-smi.
  - For Ubuntu, it requires server-xorg-core 1.20.7+ and a display to work without --no-window.
  - For Linux dockers, the setup is not complete. Install the latest driver, xServer and NVIDIA container runtime.
  
• Run the install script at ws7. Here I get some warnings instead of errors:
  [Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.isaac.kit.app_framework'] were loaded before SimulationApp was started and might not be loaded correctly.
  [Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
  Starting kit application with the following args: ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--allow-root']
  Passing the following args to the base kit application: []
  [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
  [Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230417_152925.log
  2023-04-17 15:29:25 [11ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.sensors.nv.lidar' or '/isaac-sim/exts/omni.sensors.nv.lidar/config'
  2023-04-17 15:29:25 [11ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.sensors.nv.radar' or '/isaac-sim/exts/omni.sensors.nv.radar/config'
  [0.129s] [ext: omni.stats-0.0.0] startup
  [0.168s] [ext: omni.rtx.shadercache-1.0.0] startup
  [0.176s] [ext: omni.assets.plugins-0.0.0] startup
  [0.179s] [ext: omni.gpu_foundation-0.0.0] startup
  [0.200s] [ext: carb.windowing.plugins-1.0.0] startup
  [0.221s] [ext: omni.kit.renderer.init-0.0.0] startup
  2023-04-17 15:29:25 [235ms] [Warning] [carb.graphics-vulkan.plugin] No command queue family supports flags: 0x100, queue type: 3. No queues of this type will be created
  
  |---------------------------------------------------------------------------------------------|
  | Driver Version: 530.30.02 | Graphics API: Vulkan
  |=============================================================================================|
  | GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
  | | | | | | Device-ID | UUID |
  |---------------------------------------------------------------------------------------------|
  | 0 | NVIDIA GeForce RTX 2080 Ti | Yes: 0 | | 11510 MB | 10de | 0 |
  | | | | | | 1e07 | 70659f70.. |
  |=============================================================================================|
  | OS: Linux 9e52a285d8ed, Version: 5.19.0-38-generic
  | XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
  | Processor: Intel(R) Core(TM) i9-9900 CPU @ 3.10GHz | Cores: Unknown | Logical: 16
  |---------------------------------------------------------------------------------------------|
  | Total Memory (MB): 64135 | Free Memory: 62190
  | Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
  |---------------------------------------------------------------------------------------------|
  
• Yet, the App has a long waiting time, followed by a crash. After that it starts to install the benchbot_api and extensions.
• After that, the installation is finished. Started the simulation with benchbot_run --robot carter_omni --env miniroom:1 --task semantic_slam:active:ground_truth, as specified in documentation.
• The result is impressive:
  
• Note that I have run this super-user, not as regular user. When I did the installation as regular-user with a super-user terminal, I got an error that I couldn't control the display.

April 16, 2023

• Trying to install benchbot on XPS9300; an Ubuntu 22.04 system with a RTX 1080.
• Following the instructions of benchbot page.
• The installation script now starts, although a wrong cuda key is downloaded (ubuntu20.04 key instead of ubuntu22.04 key).
• Installation script stills crashes, but that seems because there seems installation-lock (although no installation is running).
• Did a manual sudo apt update; sudo apt upgrade, which gets the installation at least one-step further.
• Script asks for a reboot, before continuing with the install (some kernel-methods were updated)..
• The script does some checks (Ubuntu version OK, GPU OK, Nvidia driver valid from standard PPA).
• The next step is to install the cuda-drivers (version 12.1).
• Had the problem that there wasn't enough spece for a 32GB image. Moving the docker /var/lib/docker to one of the DATA-partitions didn't work (by the script), because also the sudo systemctl stop docker.socket was needed, and after that the existing docker images seemed to be corropted (and didn't want to be removed). Boldly doing docker system prune -all solved it. Also 62Gb of space was freed, which meant that the benchbot docker-image could be installed on the default location.
• Another small was with imagetk, which was installed but loaded from my old dreamvu environement. Corrected my ~/.bashrc and the install script starts with building the new benchbot docker-image.
• Everything seems to go fine, until a nvml error at end:
  => [ 7/10] RUN rosdep init && rosdep update && mkdir -p /benchbot/ros_w 13.2s
  => [ 8/10] RUN git clone https://github.com/qcr/benchbot_msgs /benchbot/ 9.8s
  => [ 9/10] RUN git clone https://github.com/qcr/benchbot_robot_controll 20.6s
  => [10/10] RUN apt update && apt install -y python3 python3-pip && p 9.5s
  => exporting to image 5.8s
  => => exporting layers 5.8s
  => => writing image sha256:e32eb7d18dd8d67814dce041d708606bc904bc63bda06 0.0s
  => => naming to docker.io/benchbot/simulator:sim_omni 0.0s
  
  BAKING SIMULATOR DATA INTO 'sim_omni' DOCKER IMAGE:
  docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
  nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown.
  ERRO[0000] error waiting for container: context canceled
  
• Will try an reboot, maybe that solves the driver problem.
• That works. The GUI starts, although no scene is loaded:
  
• Problem seems to be related with the HydraEngine:
  
• Look if I could reproduce this error tomorrow on one of the WS-machines, if still needed.

April 13, 2023

• 5 MscAI-students will work on the Robotic Vision Scene Understanding Challenge
• As initial reading, I provided An Overview on Visual SLAM: From Tradition to Semantic.
• One of the recommendations for the project was to start simply.
• The Visual SLAM overview paper pointed to for the traditional approach, including a toolbox.
• The toolbox was actually made for the tracking competition of ISMAR, as described . The description is not very clear which algorithm is inside the toolbox, it only mentions that ATAM is extension of PTAM, and the toolbox is Windows based!
• Note that the tracking competition was also organized in 2017, where they point to Vuforia as SDK (mature), although this Engine is for Augmented Reality (including Unity plugin).
• The overview paper An Overview to Visual Odometry and Visual SLAM: Applications to Mobile Robotics is much better, and also shows some figures and illustrations they have seen at their Projective Geometry lecture.
• The BenchBot looks like the Nvidia Carter. The Carter had a Velodyne Puck, while the BenchBot seem to provide 2D laser scans(together with RBG and RGBD images). The omniverse based version is the latest, Nvidia ISAAC version is depreciated (but used in the competition?!).
• As started, the point to the DF-VO visual odometry algorithm, which is a paper with code
• In simulation I don't expect much texture, so the PL-SLAM algorithm (extension of ORB-SLAM) looks promising. It is also a paper without code. An implementation (mono) by Jiagang Chen can be found at github. Another promising implementation (stereo) by Ruben Gomez-Ojeda can be found at github. That one is also published in journal.
• Yet, in simulation using a plane based approach could be an option.
• Concerning RGBD-algorithms, the indirect method RGBD SLAM v2 is claimed to be very accesable, and is ROS-compatible (although ROS kinetic).
• The BenchBot seems to have no IMU, so Visual-Inertial SLAM is out.
• The founder of the competition maintains the Semantic SLAM website, which mainly describes the QuadricSLAM from 2018 (no code or dataset).
• The code of QuadricSLAM can be found at github.
• But Niko Nünderhauf has several interesting research projects. Semantic SLAM is part of Scene Understanding.
• Looking how difficult it will be to install benchbot on a Ubuntu 18.04 machine like ws4-ws6.
• First test would be benchbot_run –list-examples, as indicated here
• Next would be to apply Votenet, as this tutorial.
• Started with the install on ws5, but the first check is if the system is Ubuntu 20.04 or above. Moved to ws7 (Ubuntu 22.04)