Natural Language Driven Robotic Manipulation with Computer Vision & LLM Control

Overview

• An intelligent manipulation framework on the Dobot Magician Lite that executes conversational commands like "take the farthest blue block and stack it on the yellow piece".
• Modular pipeline: LLM command parsing, HSV based object detection, calibrated coordinate transformation, and kinematically validated motion control.

Design & Build

• Language: A Llama 3.3 70B subsystem, served on Groq through LangChain, parses instructions, classifies actions (stacking, positioning, rotation), generates validated execution code, and keeps conversational context so "that block" resolves across turns.
• Vision: HSV color segmentation with adaptive thresholding extracts per color object IDs, pixel centroids, and contour areas, serialized to JSON in real time.
• Calibration: Least squares affine transformation over multiple reference points maps camera pixels to robot coordinates with sub millimeter reference accuracy.
• Motion: Safe approach trajectories, height aware descent, vacuum gripper control, and stack aware placement with persistent state that survives system restarts.

Challenges

• Ambient lighting variations breaking HSV color detection.
• Periodic recalibration of the camera to robot transformation.
• Stack aware z height management for multi block towers.
• Validating LLM generated commands before physical execution.

Result

• Successful execution of multi step natural language operations: retrieval, stacking, directional placement, and color based selection.
• ±2 mm positioning precision in the xy plane, 15 to 30 second cycle per pick and place.
• State memory persists across sequential commands and restarts.

Skills & Tools