Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Gimbal Interface #3386

Draft
wants to merge 37 commits into
base: master
Choose a base branch
from

Conversation

robertlong13
Copy link
Collaborator

@robertlong13 robertlong13 commented Aug 4, 2024

Still a bit to do, but I have some stuff working. It's going to be a massive PR, so I want to get some of this out there to start review early.

  • I have a test setup using RealFlight that I will need to document so anyone can play around with this.

  • I have a lua script I need to publish that sends VIDEO_STREAM_INFORMATION messages so Planner can detect the URIs for the video.

  • ArduPilot's CAMERA_FOV_STATUS message has a bug that causes the wrong units to be used for alt. Needs fixing.

  • Most of the buttons and right-click options do not work yet

  • Need a settings page so people can adjust the stuff like keybindings

  • Things you can do

    • WASD to slew. Hold shift to go faster, ctrl to go slower.
    • Hold E to zoom in, hold Q to zoom out.
    • Clicking sets the POI to the lat/lon of the point you clicked, though more options are intended. (THIS ONLY WORKS IF YOU HAVE PATCHED CAMERA_FOV_STATUS to send the correct units)
    • Alt-F to take a picture
    • Alt-R to toggle recording
    • L to toggle between lock and follow
    • N to go neutral
    • H to point at home

(Terrible gif quality, had to really squeeze it down to fit in a github post, so there are a ton of artifacts; these obviously aren't in the real thing)
gimbal_early_demo

@rmackay9 rmackay9 requested a review from meee1 August 5, 2024 06:59
@rmackay9
Copy link
Contributor

rmackay9 commented Aug 5, 2024

This is really great to see! Thanks for also finding the issue with FOV_STATUS. We added support ages ago but no GCS has used it yet so I guess we missed the bug.

@Davidsastresas
Copy link

Very neat! :)

@IuliuNovac
Copy link

How did you manage to get the simulation with a video feed up and running, is that gazebo?

@robertlong13
Copy link
Collaborator Author

robertlong13 commented Aug 18, 2024

How did you manage to get the simulation with a video feed up and running, is that gazebo?

I use RealFlight. I'm sure the same thing could be done in Gazebo, but I don't know my way around it nearly as well. You need this model and this lua script. You also need to be running on AP master (until the FOV message bug fix gets backported to 4.5)

I use a gstreamer pipeline to grab a cropped area of a monitor. I toyed with using OBS to stream as RTSP, but that introduced more than a second of latency.

dx9screencapsrc x=10 y=60 width=640 height=360 monitor=1 ! videoconvert ! video/x-raw,format=BGRA ! appsink name=outsink sync=false

Adjust the monitor, x, y, width, and height for your setup.

If you configure your autopilot to provide VIDEO_STREAM_INFORMATION messages, with gst://[pipeline] as the URI, this new UI can grab the pipeline from there automatically (it doesn't auto connect to it yet though; that's a planned feature).

The airport in this video was Pendleton Airport (KPDT), but I don't know if I can share my RealFlight model of that airport publicly. Instead, I'll share Base INFOCA, which has the side benefit of being within the real terrain model RealFlight pulled from Spain, so all the terrain actually lines up: Base INFOCA_AP.zip. The downside is that I went WAY overboard manually adding one metric crapload of trees to it. Disable trees with View > Scenery > Trees to disable them if your computer is choking on them. Launch SITL with -O 37.412620,-3.075734,1052.0,0

@robertlong13 robertlong13 force-pushed the pr/gimbal_interface branch 3 times, most recently from 9fb28b3 to 56a2a6b Compare September 15, 2024 03:40
@robertlong13 robertlong13 force-pushed the pr/gimbal_interface branch 3 times, most recently from eb3d908 to 15628d5 Compare September 29, 2024 06:45
@robertlong13
Copy link
Collaborator Author

This PR is getting close to ready. All buttons and (almost) all keybindings are implemented, and I have a nice settings page where you can customize every keybinding.

Still needed:

  • Click to track (need a testing setup for this one)
  • Auto connect to first video stream if found
  • Consider making a way to embed this on the flightdata page with minimap, and have the ability to swap between map with mini video and video with mini map.

@meee1
Copy link
Contributor

meee1 commented Oct 1, 2024

need to tweak for android/apple - currently failing. system.drawing cant be used easerly....

@@ -7,6 +7,8 @@ namespace MissionPlanner.Controls
{
public partial class Video : Form
{
private readonly GStreamer stream = new GStreamer();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to confirm if this can be instance based. i have a feeling it cant. as it will lead to native crashs

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can open multiple GStreamer sources (multiple gimbal windows and a separate stream on the HUD) with no apparent issues if that's what you mean.

@robertlong13 robertlong13 force-pushed the pr/gimbal_interface branch 2 times, most recently from 6cd30be to 6adf829 Compare October 13, 2024 08:02
@robertlong13
Copy link
Collaborator Author

Tracking added

@robertlong13
Copy link
Collaborator Author

Added new ways to display the new control

image

Full Sized. In this mode, there is a mini map by default, but it can be hidden with a button in the right-click menu.
image

Mini
image

Mini and full-sized can be swapped at any point with the "swap with map" when right clicking on the video.

And pop-out works exactly as before (except now it remains always on top, which I think is always better, you can always intentionally minimize it if it's in the way).

- Added cameras to detected autopilot components, not just cameras
- Added XML documentation to all public members
- Removed video stream RTSP handling from the class, it will later be
  re-added as a separate class.
- Encapsulated camera commands in methods of CameraProtocol. More
  methods will be added later.
- Added properties for capabilities (including mode-dependency)
- Appropriate camera messages, based on camera capability, are requested
  from the camera at the same rate as the EXTENDED_STATUS stream. We may
  eventually add a separate rate option for camera messages.
- Removed unused code
- Renamed some public methods with capital letter
- Suppressed some naming warnings
- Removed all commented-out code
- Removed all scratch-pad comments
- Removed unused usings
- Fixed various other minor suggestions from Visual Studio
Do everything through keyboard and context menu
Use this to store the lowest found gimbal instead of using it to store
capabilities flags of all gimbals.
Previously, this function did nothing. It was also trying to do the
opposite of what it claimed to do.
This library was originally copy-pasted from ArduPilot and ported to C#,
but many mistakes were made in the process. For example, many of the
methods in the C++ implementation take a reference variable as input,
and the C# code was taking a regular input and outputting nothing.

This commit overhauls the whole library, making it more C#-like and
documents all methods.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants