Skip to content

Explore a project that develops a SLAM-based navigation system using vision-language data inputs. This project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.

License

Notifications You must be signed in to change notification settings

YasiruDEX/SLAM-Ros2-bot-Working

Repository files navigation

SLAM-based Navigation System with Vision-Language Inputs

Explore the repository for a cutting-edge project aimed at developing a SLAM-based navigation system using vision-language data inputs. The project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.

Key Features:

  • URDF Model: Custom differential drive robot with Kinect V2 sensor.
  • ROS2 Control: Differential drive and position controllers.
  • ROS SLAM Toolbox: Utilizes Xbox Kinect sensor data for effective mapping.
  • Teleoperation: Control via Xbox game controller with seamless multiplexer support.
  • Frontend: Next.js interface for visual and natural language input.

Machine Learning Architecture:

  • Multi-model approach integrating NLP and computer vision.
  • NLP with vocal data using OpenAI Whisper and BLIP for image feeds.
  • Mistral 7B large language model quantized to int8 for robot action prediction.
  • Utilizes Huggingface Transformers and CTransformers for local deployment.

Localization and Navigation:

  • AMCL for localization with prerecorded environment maps.
  • Nav2 stack for precise navigation to goal destinations.
  • ROS 2 control for armature actions.

Tools & Platforms:

Acknowledging the support of ROS 2 Iron Irwini, Ubuntu 22.04 Jammy Jellyfish, Gazebo, HuggingFace Transformers, Salesforce, Whisper, Nav2, and Mistral AI for making this project possible.

Check out the GitHub repository for more details and code implementation.

Stay tuned for LUNA, the broader real-world implementation of this SLAM project!

🔗 Video explanation: Explore Here

#Robotics #AI #ROS2 #MachineLearning #SLAM #VLN #Innovation #TechProjects

About

Explore a project that develops a SLAM-based navigation system using vision-language data inputs. This project integrates natural language vocal instructions and image feeds to guide a differential drive robot equipped with a Kinect V2 sensor through dynamic environments.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published