Started Labbook 2022.

December 28, 2021

• Decided to call my new Duckiebot dagobert. Unfortunatelly, the flashing fails on the verification step.
• The command dts init_sd_card --help gives no information on the steps, although this post gives some clues.
• Looked into ~/.dt-shell/commands-multi/daffy/init_sd_card/command.py. The SUPPORTED_STEPS are licence, download, flash, verify, setup.
• Running the command with --steps setup, which works. Trying if booting works with this card.

December 27, 2021

• Assembled Duckiebot db21.
• Note that the Duckiebot hut as Stemma Qt connectors, next to I2C connectors. Also the display is the same as the Nanosaur.
• As bonus, the Duckiebot DB21 has now also a time of flight sensor on its nose.

December 22, 2021

• The PhD thesis from Sascha Jannik Steyer contains dynamic state estimation based on radar.

December 20, 2021

• The Biomorphic Intelligence Lab has 4 vacancies, where the third one is on event camera's (and the fourth one on neuromorphic chips).

December 17, 2021

• Read Boudewijn Bodéwes' thesis. Notice that in Fig. 12 tree canopies are clearly visible as in the disparity error map, indicating that there is fluctations in the disparity (leaves) but not in the segmentation.

December 9, 2021

• Stratis has a NeurIPS paper on Roto-translated Local Coordinate Frames.
• They demonstrate that on the inD dataset, which is a recording with a drone of an German intersection.

November 24, 2021

• Looking through the EZTH 2017 projects for a traffic light reference implementation. Found none.
• Also looking through the 2019 and 2020 projects. The Intersection Navigation project has no traffic light code, but stears the robot left,straight,right based on ros parameters, and has a nice bird-eye view of the intersection with the cv2.warpPerspective function.
• The Object Detection project uses a TPU to recognize objects, and has recognizes 7 objects (Traffic light is no 3). It doesn't distinguish the color-state of the traffic light.
• From Udacity, there is still the CarND Object Detection lab. This lab makes use of tensor flow models. The link to the zoo doesn't work, because there is now a tensor flow v2 zoo and Tensor flow v1 zoo. The Jupyter notebook was build for tf1. The Kitti-trained model looks interesting (and the Edge TPU models!).
• Luckily, there is also a traffic-light classifier assignment from Intro to Self Driving Car. Here is an example solution from one of the ISDC students.

November 22, 2021

• This article on Domain Adaption could be interesting for Henk.
• Also the SeasonDepth dataset could be interesting. The dataset itself is available on github.

November 18, 2021

• Nice picture of camera, lidar and radar combination by Waymo in the fog of San Francisco.

November 12, 2021

• Martin pointed me to two interesting papers:
  • DeepSVG
  • PolyMapper

November 9, 2021

• Followed a class from ConstructSim to detect with Yolo objects with a NanoSaur, based on a Nvidia Jetson.
• The building instructions of NanoSaur include an expansion board and two Leds for the eyes.
• The Darknet port for ros-foxy can be found on github.

.

November 4, 2021

• Amazon now also has its own DeepRacer.
• Team 1 from Epoch is now on Rank #8 / 24 in this competition, mentored by Jesse Maas, one of our MscAI students.

November 2, 2021

• Started up the Jetson racer again. The latest modification was of May 18, 2021, when I was experimenting with the DreamVU camera.

November 1, 2021

• This Sensors journal article (with the code in the text!) recorded road obstacles with the ZED-camera.

September 20, 2021

• Impressive talk on unsupervised depth estimation, inspired by the original monoDepth paper (CVPR 2017), including code, models and talk at website.
• This was augmented with Ego motion estimation in SynDeMo paper (ICCV 2019), unfortunatelly without code or models.

September 7, 2021

• Weijie Wei current work is inspired by RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation.
• Qi Bi pointed to Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes.

August 24, 2021

• Maël is planning to use a pyramid pooling layer between the finale feature maps and the fully connected layers, as done by Zhao et al. Note that RetinaNet also is a ResNet with a FPN on top, as indicated in the original RetinaNet paper and by Zhu et al, who uses it for vehicle behavior recognition from a bird-eye view.

August 16, 2021

• Also Emma Quist has used CARLA to train an autoencoder on Bird's eye-view segmented images for Bsc thesis
• Segmenting the off-road path is according to David Cerny well described by Sharma 2019 and Holder 2016
• David Cerny's works was inspired by World Models, where a small controller combines current Vision with a RNN Memory to decide good actions. The current Vision is compressed to smaller abstract feature vector by a Variational Autoencoder (VAE).
• David Cerny has tested his algorithm on Town07. Another paper which uses Town07 is Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation.

July 27, 2021

• The presentations of 3D Deep Learning for Autonomous Driving workshop are now available on youtube.

July 7, 2021

• Waymo now released Simulation City

June 10, 2021

• In my rescue labbook, I was looking at 3D scene visualisation, which works great for the Kitti dataset.
• Found lidar transfer to convert datasets.

June 3, 2021

• A post on Roberto's GNC algorithm in MathWorks News.

June 2, 2021

• At ICRA there will be the next workshop on the NuScenes and DuckieTown challenge.

May 25, 2021

• Read the 2nd blogpost on Learning in Depth from Toyota Research.
• Also scanned the Radar-to-Lidar article from Rong Xiong.

May 17, 2021

• Found this paper to improve photorealism (in GTA V).

May 11, 2021

• For large-scale point clouds, RandLA-Net seems to be state-of-the-art.

March 28, 2021

• Waymo have published a new Motion dataset, which could be used for behavior prediction.
• The dataset comes also with Jupyter notebook based tutorial.
• More details in the paper Offboard 3D Object Detection from Point Cloud Sequences.
• Waymo has now four challenges which will close on May 31.

March 24, 2021

• Hoàng-Ân Lê points to Object Scene Flow for Autonomous Vehicles (2015) in his thesis.
• And Dynamic 3D Scene Analysis from a Moving Vehicle (2007).
• And Segmentation and Recognition Using Structure from Motion Point Clouds (2008).

March 18, 2021

• Encountered an article which uses Wide-FOV cameras for localization while driving in Urban environments.

March 17, 2021

• Looked at Institut Pascal Long-Term dataset, which contains recordings of the same electric shuttle over the same parking lot over a full year. The extract_rosbag.py extracts the images and imu data, although the rosbag also contains lidar and gps data.
• Installed ros-melodic-ros-comm on WSL Ubuntu 18.04 of my home-computer, which allow me to do a rosbag info 2020-01-22-10-19-51.bag.
• This rosbag not only contained the lidar and gps data, but also 648 velodyne messages!
• Actually, there is a long-term visual localization workshop, which is both in 2019 and 2020. In 2020 there were only two submissions with public results!
• The workshop has multiple datasets, for the visual localization for autonomous vehicles challenge it uses these three:
  • Extended CMU Seasons.
    The original location of the dataset is only available on the Wayback machine. A subset is still available at this mirror.
  • RobotCar Seasons v2
  • SILDa Weather and Time of Day Dataset
    Also this dataset is no longer available, not even on Wayback machine. This post gives some impression of this dataset. The dataset is also no longer available on github, although it was also used in Image Matching workshop. Found the dataset at a personal github. The download.sh seems to work, I could download the camera-intrinsics.tar.xz. Left the other 60Gb for a moment on the Imperial College server.
• In 2019 the same datasets where used (with a previous version of RobotCar Seasons).
• In 2019 there were two winners and one runner-up of the Visual Localization Challenge:
  • Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, Marcin Dymczyk, From Coarse to Fine: Robust Hierarchical Localization at Large Scale
  • Tianxin Shi, Shuhan Shen, Xiang Gao, Lingjie Zhu, Yurun Tian, Qingtian Zhu, Visual Localization Using Sparse Semantic 3D Map
  • Hugo Germain, Guillaume Bourmaud, Vincent Lepetit, Sparse-to-Dense Hypercolumn Matching for Long-Term Visual Localization
• In 2020 there was a clear winner on all Challenges and a running up for the Autonomous Vehicle Challenge:
  • Paul-Edouard Sarlin: Hierarchical Localization with hloc and SuperGlue
  • Martin Humenberger, Yohann Cabon, Nicolas Guerin, Julien Morat, Jérôme Revaud, Philippe Rerole, Noe ́ Pion, Cesar de Souza, Gabriela Csurka, Late Fusion of Global Image Descriptors for Visual Localization

March 4, 2021

• Cyrill Stachniss published his preprint of 4D Panoptic LiDAR Segmentation.
• The example implementation licence plate detection is interesting, once Transfer Toolkit is installed as done for @Home applications

March 1, 2021

• TU Delft has now HERALD Lab: Human-aware Robust Artificial Intelligence for Automated Driving. They are looking for 5 PhD students.

February 28, 2021

• Maybe I should try to install the login manager slim on my RB5, as suggested on askubuntu

February 24, 2021

• Continue with the install of open3d on an ARM-processor.
• The Eigen/Core problem is back, but by uncommenting the previous removed (due to the dependency on the latest cmake) open3d_link_3rdparty_libraries the building continues.
• Added typedef CBLAS_ORDER CBLAS_LAYOUT; /* for backward compatibility */ to cpp/open3d/core/linalg/BlasWrapper.h. Checked with apt-cache policy libblas-dev, which indicates v3.7.1 (while v3.8.0 is available since Nov 2017).
• Cloning lapack-release, which contains libblas v3.8.0.
• Have not installed v3.8.0 (yet), because the CBLAS_LAYOUT hack works. Open3D is now fully build. Continue with the make install command, which installs the package in ~/open3d_install, as specified with the cmake command. The Open3DConfig.cmake is installed in ~/open3d_install/lib/cmake/Open3D.
• Continue with the Open3D python package.
• Quanser also has a self-driving car studio which is Jetson TX2 based.
• Everything is build. Continue with depthai. Both the sudo wget -q0- http://doc.luxonis.com/_static/install_dependencies.sh | bash and python3 install_requirements.py work, although the optional package open3d is skipped. Installed that package directly with pip3 install /home/jetson/git/Open3D/build/lib/python_package/pip_package/open3d-0.12.0+31e5d4d1-cp36-linux-aarch64.whl.
• Still python3 depthai_demo.py crashes on SIGILL in gotoblas_dynamic_init().
• Installed v3.8.0 of libblas, still crashes.
• Upgrading to Ubuntu 20.04.
• Reading Highly Automated Vehicles and Self-Driving Cars, which gives a nice recent overview of what is needed for self-driving cars. Could be used as reference the ADAS levels.
• Upgrading to Ubuntu 20.04 was the solution. python3 install_requirements.py gave some warnings. Downloading python3-opencv_4.2.0 for amd64 solved that (although numpy and pyyaml were still missing).
• Receive a tracker-extract internal error, not clear if this related with python depthai_demo.py.
• Also the disparity channel can be accessed with python3 depthai_demo.py -s metaout previewout,12 disparity_color,12:
  

February 23, 2021

• Continue with making cmake 3.18 on the JetRacer.
• During compilation I read the article in Journal of Field Robotics on Fusion of neural networks, for LIDAR‐based evidential road mapping, which indicates that the NuScenes dataset has LIDAR scans tuned to detect nearby obstacles, for road detection on larger distances this dataset is less usefull.
• The dataset with VLP32C LIDAR (unique) are publicly available.
• The algorithm was also compared with the KITTI dataset, although also there critical notes were made on unlabelled laserscans (because no corresponding camera-segmentation label was available).
• The installation was done succesfully, although I made cmake v3.20.
• Needed a fresh terminal to access the new version of cmake. Also made a fresh build directory for Open3D.
• Continue with make -j$(nproc).
• Everything seemed to go OK, but target all fails after completing ext_filament.
• Had to search deep for an error, but the PinHoleCameraIntrinsic fails finding include Eigen/Core. Could try to define EIGEN_INCLUDE_PATH as /usr/local/include/eigen3, of specify -DENABLE_PRECOMPILED_HEADERS=OFF. For the moment, a new make continues with building (without any modification). Tried sudo apt install libeigen3-dev, but I have already the latest version. The include files can be found at /usr/include/eigen3. eigen3 is mentioned in CMakeCache.txt and Open3DTargets.cmake.
• Now the build fails on another part of the project: src/lib_json/CMakeLists.txt:125 patch does not apply. Patch is an optional MSVC-patch, so commented that line out from build/CMakeFiles/ext_jsoncpp.dir/build.cmake.
• Next fails the visualisation gui, on missing GLFW/glfw3.h. Installed sudo apt-get install libglfw3-dev and tried again.
• Now the visualisation/gui fails on Eigen/Core. Included -I/usr/include/eigen3 in cpp/open3d/visualisation/gui/CMakeFiles/GUI.dir/flags.make.
• Next is an dependency on sudo apt-get install libfmt-dev. That fails that the include tries to include format.cc, so tried to solve that sudo apt install libspdlog-dev. That is not the solution. Made a link from /us/include/fmt to /usr/include/spdlog/fmt/bundled/format.cc. Also fmt/ranges.h was needed, which is part of fmt v7.1.3. Cloned the library from github and did a sudo make install.
• The json patch problem was back, so commented that line out in ~/git/Open3D/3rdparty/jsoncpp.

February 22, 2021

• Matlab contains now the GNC algorithm for pose graph optimization, which handles outliers in SLAM much efficiently.

February 20, 2021

• Try again to install DepthAI on Nvidia Jetson. They claim that the same install instructions as Ubuntu work fine on the Jetson.
• Part of the install instructions of open3d on an ARM-processor is the installation of libblas-dev (the instruction that crashes). Yet, after installing the prerequisites the crash is still there.
• Building Open3D with the Nvidia Jetson options -DBUILD_CUDA_MODULE=ON and -DBUILD_GUI=ON specified for cmake.
• The required version of cmake is 3.18, which is not available for Ubuntu 18.04. The suggestion was to include cmake 3.18 from the repository, but no ARM-implementation was available.
• Protected the 3.18 specific calls with an if(${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.18.0"). Still the different packages have problems with open3d_link_3rdparty_libraries. Hope it works, because I got still errors: $: unknown language.
• Making the C++ library seems to work. After more than an hour the make failed on package ext_faiss, which requires a higher version of cmake. Lowered the requirement, which didn't help because of missin CMAKE_HOMEPAGE_URL_COMPILER_ENV_VAR.
• Cloned v3.18.6 from gitlab and did a ./bootstrap.

February 9, 2021

• Interesting new article in Journal of Field Robotics on Fusion of neural networks, for LIDAR‐based evidential road mapping.

February 5, 2021

• Continue with the Windows installation of the DepthAI camera.
• The command python3 is not known, but just python works fine.
• Got a warning that open3d was not installed, because my platform wasn't armv7l. Still, the OAK-1 seems to work fine.
• It not only works from a powershell, but also from a regular commandline. When starting the demo, mobinet-ssd\FP16 was downloaded, to recognize objects.
• Used my Thunderbold USB-C cable directly. Still 30 fps.
• Also tried the OAK-D with python depthai_demo.py -s metaout previewout,12 disparity_color,12 and python depthai_demo.py -s metaout previewout,10 depth,10. No complains on open3d.

February 4, 2021

• In their video, SLAMcore demonstrated the combination of a Jetson and a DepthSense, including PanOptic segmentation.

February 3, 2021

• Received the OpenCV AI Kits. First trying installation instructions on Windows on my nb-dual.
• Had to reboot, so I switched to the Ubuntu installation. Works directly, both for the OAK-1 and OAK-D.
• Read the post on DepthAI on Nvidia Jetson.
• Continued with the AI JetRacer. Installed depthai-0.4.1.1 with python3 -m pip -U install depthai.
• Building the depthai wheel takes a long time. Not sure if I should have installed it by building the source from the github.
• Running python3 install_requirements.py also tries to build a depthai wheel (and pyyaml).
• After several tries, the install succeeds, except the optional dependency on open3d==0.10.0.0.
• When needed, it is possible to build open3d from source.
• Yet, python3 depthai_demo.py crashes with an illegal instruction (SIGILL in gotoblas_dynamic_init().
• Simple cheese says no device found. Cheese works on my nb-dual, but gives the OAK-D not as possible device.

January 31, 2021

• PAL now comes with 3D vision system for a reasonable price of $ 999,-.

January 21, 2021

• Read the ECCV paper Geographically Local Representation Learning with a Spatial Prior for Visual Localization. Would be interesting to apply this idea to the Erfoud dataset. In the paper they make a comparison between global and local features available. The global features are baseline features, which could be matched with satelite images, but are less distinctive. Problem is that the Erfoud dataset contains no local or global features. The code of the paper is also available on github.
• Also read the IEEE Robotics and Automation article CNN Based Road User Detection Using the 3D Radar Cubei (pre-print. The code is also published on github. The dataset is recorded with Continental radar sensor.
• Continental now also has Camera with Lidar combination, which is suited for pedestrian detection. That sensor was already avaible in 2015.
• Continental participated in 2016 in the RoboCar challenge. No further information on pedestrian detection, only the product description of Emergency Brake Assist.
• Our group is also involved in Efficient Deep Learning project.
• Compared Ricoh Theta V vs Z1. Image quality of the Z1 looks superior.

January 19, 2021

• Ozzy Ülger and Qi Bi are our two PhD-students working in the Atlas lab.

Previous Labbooks

• Labbook 2020.
• Labbook 2019.