About usage in VoiceAttack

[WilliamRiker2701]

Hi, i'm writing a modified version of a plugin for VoiceAttack called MSFSAgent, which makes it possible through VoiceAttack software to trigger SimConnect events through voice commands. My objective is to create a virtual monitoring pilot that helps with cockpit work using VoiceAttack. I already accomplished this thanks to MSFS Agent. The problem of that VA plugin is that it is limited to SimConnect events, so with third party aircrafts there's problems. i'm wondering if your tool could be used to create a plugin for VA that makes it possible to modify local variables through voice commands. what do you think about it?.
Cheers!

[mpaperno]

Hi William,

Yes, absolutely!

That (sounds like) exactly what I do with a plugin for a data display/macro launcher software (and one of the reasons I started this project in the first place): MSFS Touch Portal Plugin. It's even in C# already.

This also works both ways since Touch Portal can display data -- meaning the plugin also gets any data it needs from SimConnect or WASimModule, including Local variables or results of RPN calculator code.

When I originally inherited that plugin project it also used hard-coded events (and SimVars) and used SimConnect for everything. I've since expanded it so a user can activate pretty much anything "remotely" (pun intended!) possible (setting Sim or Local vars, H events, calculator code, etc). And it will use WASimClient if WASimModule is detected installed on the sim, or fall back to SimConnect otherwise (with limited functionality of course).

For reference, all the actual Simulator interaction, and where WASimClient is used alongside SimConnect as needed, happens in SimConnectService.cs

LMK if you have any other questions. You can also ping me on that plugin's Discord channel/room if you'd like TouchPortal#msfs2020

HTH!
-Max

[WilliamRiker2701]

Hi again Max!

I've been looking at your Touch Portal code during the past few weeks and trying to fusion its logic with MSFSAgent to get WASM to work with the MSFSAgent plugin for Voice Attack. I got rookie skills when it comes to programming, so it has been a learning curve. I'm determined to accomplish this thing though. A bit of guidance into what i'm looking for would be great help, just to be sure i'm making the right calls here regarding what this SimConnectService.cs code contains.
I'll dissect this thing into a few functionalities.
First, i need a code that connects to SimConnect and WASM to look for the state of a variable (SimVars or EventID's in the case of SimConnect and lvars and stuff for the case of WASM). It seems to me that the function Connect() makes the first part, but i'm not sure which function makes it able to look for the state of the variable and return that value. I was thinking on RequestDataOnSimObject() but i'm not sure if that one does the WASM part... Any suggestions on exactly where to look for that?
The second part is to be able change the value of that variable. So with this i've been looking at TransmitClientEvent(), or SetDataOnSimObject(), or SetVariable() but that one looks like only acts on WASM, so i'm not sure again.

I would greatly appreciate your input and suggestions on this.

Thanks!

-William

[mpaperno]

Hi William,

OK, let me try to dissect the code a bit. I'm afraid there's a bit of "legacy" code in SimConnectService (SCS) which isn't really used as public anymore since I've been trying to consolidate/move most of the actual sim interaction into that class (instead of just being a proxy/abstraction for SimConnect which is what it started out life as... not that useful really). For example in most cases it is now up to SCS to choose how to communicate with the sim -- SimConnect or WASimClient.

Requesting variables:

On the variable/data request side, there's one layer in the plugin code that's not very obvious.
All the variable requests (including for calculator string results) are kept in a custom collection class.

Basically it's the data repository of SimVarItems, but with "ItemAdded" and "ItemRemoved" events which SCS connects to. (Despite the (legacy) name, SimVarItem can represent any type of variable/data request, not just actual "A" type vars.)

The main PluginService can modify this collection of SimVarItems based on config files or user actions or whatever, and the changes are automatically propagated to SCS. Which is the part you want to know about....

In the c'tor SCS connects to the collection's events: https://github.com/mpaperno/MSFSTouchPortalPlugin/blob/next/MSFSTouchPortalPlugin/Services/SimConnectService.cs#L86

When a new SimVarItem is created this leads to RegisterDataRequest which is what I think you were looking for in that department. You can see that from there it takes different actions based on the type of variable/request.

If the request is for anything but an actual SimVar ("A" type) or the user requested a custom refresh rate in ms, and WASimModule was detected by WASimClient, then the request is routed via WASimClient. Otherwise via SimConnect if it's an "A" type request, otherwise it fails. (I use SimConnect for "A" vars even if WASimModule is found because, why not? Less for the WASM to do.)

So in other words all roads for variable requests lead to RegisterDataRequest() and that decides how to handle it.

Setting variables:
(Not to be confused with "Key" Events/SimConnect Event IDs/)

Right now this is not abstracted in SCS, so the PluginService decides which method to call, either SetDataOnSimObject() which uses SimConnect.SetDataOnSimObject() and only works for previously-registered SimVars ("AddToDataDefinition()" and all that), or setVariable() which is essentially a pass-through for WASimClient.setVariable().

The WASimClient way of course is the only way for var types besides "A", but it's also nice for A types since it can set any variable by name or ID w/out first having to register it. Keep in mind of course that not all A vars are settable (attempts are silently ignored by both SimConnect and WASimModule/Gauge API).

I have code to abstract this part also (so the PluginService doesnn't have to worry about how the variable get set, but haven't had time to finish yet.

Events:

There's some possible confusion with event vs. variable since "H" type variables are actually called "HTML Event" type and are used to trigger events in some models. But they're not related to Key Events, or what's referred to as "Event IDs" in the MSFS SDK docs. H type vars/events can also be set/triggered with calculator code.

The Key Events also have a variable type, "K", and can be triggered via calculator code. Up until SU10 that was the only way to trigger Key Events with multiple parameters.

With SimConnect EventIDs are transmitted with SimConnect.TransmitClientEvent() and now the new Ex1 version (but first they have to be registered with SimConnect.MapClientEventToSimEvent() and then the ID is used in TransmitClientEvent() instead of the actual event name).

With WASimClient this is again simpler because it's one step, WASimClient.sendKeyEvent() which can be either by name or by an ID which can also be looked up with WASimClient/Module and then cached (which is what SCS does).

So really in either case we turn an event name into an ID used to refer to the event, either an ID SCS makes up and assigns (for SimConnect) or the ID of the actual Key Event as returned from WASimCLient.lookup(). And if lookup() fails then it falls back to trying SimConnect (because it turns out SimConnect understands a few names not documented elsewhere... surprise).

This part is abstracted in SCS with the 3 overloaded TransmitClientEvent() public methods (the TransmitClientEvent(EventMappingRecord eventRecord, ...) one I linked to is probably most interesting since that is where most of the choices between WASM and SimConnect happen). The EventMappingRecord you'll see passed around in there is just a simple struct to store the event name, dynamic ID and maybe other meta data).

OK, not sure if it's any simpler to read my ramblings than my code... but I hope that helps! :-P

Also if you haven't yet, I suggest you open the MSFSTouchPortalPlugin project in VS so you can at least click around to the different parts and find references easily. Even I have a hard time following it by reading a "static" version like right now... :)

Cheers,
-Max

[WilliamRiker2701]

Thank you for this long explanation!, it definitely sheds light on a lot of stuff i didn't understand right. I already took the liberty of opening MSFSTouchPortalPlugin directly in VS, it is definitely the best way to do what i'm trying to do hehe.
I'll try to make the best out of this information, and if i have some specific questions later i'll get back to you.

Thank you again,
Cheers,

-William

[WilliamRiker2701]

Okay so i've been looking at RegisterToWasm() (since RegistedDataRequest() as far as i understand takes exclusively "L" type variables through here)

I'll use an example to make my questions: (L:A32NX_ADIRS_ADIRU_1_STATE)
This would be the local variable that holds the value for the IRS knob number 1 position. The value for this variable can be 0, 1, or 2 depending on the actual position of the knob.

The RegisterToWasm() method takes the simVar structure and takes a few code variables out of it that are crucial to return the variable value (Which i'm guessing it's hr).
simVarName is one, varId is another one. So in my example, 0, 1 or 2 corresponds to hr variable, and "A32NX_ADIRS_ADIRU_1_STATE" to simVarName isn't it? what about varId?

What about Key Events? i see RegistedDataRequest() in the following line throws them over RegisterToWasm()
https://github.com/mpaperno/MSFSTouchPortalPlugin/blob/next/MSFSTouchPortalPlugin/Services/SimConnectService.cs#L318
But then over RegisterToWasm() it just presents a direction for 'L' and 'A' type variables. Then Key Events are processed through RegisterToSimConnect()?

Thanks, cheers!

[mpaperno]

Okay so i've been looking at RegisterToWasm() (since RegistedDataRequest() as far as i understand takes exclusively "L" type variables through here)

No, RegisterToWasm() gets everything except "A" type vars, and even those in some cases when a custom timing interval is requested (because SimConnect can only do whole seconds). Besides L and A type vars that also includes calculator code requests (which return results, I call them "Q" vars), as well as a list of others... B, C, E, M, R, T, Z (MSFS RPN reference). I use "T" for Token type vars even though that's not on the official list. Most of those types aren't as useful as the first 3 though, and in fact on the back end in WASimModule all those other types (except T) are actually converted to calculator code requests automatically (I thought this was documented somewhere but I can't find it).

The RegisterToWasm() method takes the simVar structure and takes a few code variables out of it that are crucial to return the variable value (Which i'm guessing it's hr). simVarName is one, varId is another one. So in my example, 0, 1 or 2 corresponds to hr variable, and "A32NX_ADIRS_ADIRU_1_STATE" to simVarName isn't it? what about varId?

Data requests are asynchronous, just like with SimConnect's SimVars. You submit the request and you listen for updates via the OnDataReceived event. That's why there's all those properties like update frequency, delta values, etc.

The result value of saveDataRequest() is a status ("HRESULT") of the actual data submission request. Not the value of the requested thing (you may, however, get a value delivered via "OnDataReceived" event before this function even returns).

If you just want to get the value of a variable once, synchronously, use WASimClient::getVariable() which is a heck of a lot simpler but not the same thing.

What about Key Events? i see RegistedDataRequest() in the following line throws them over RegisterToWasm() https://github.com/mpaperno/MSFSTouchPortalPlugin/blob/next/MSFSTouchPortalPlugin/Services/SimConnectService.cs#L318 But then over RegisterToWasm() it just presents a direction for 'L' and 'A' type variables. Then Key Events are processed through RegisterToSimConnect()?

Key Events have nothing to do with data requests or variables at all. They can be activated via calculator code (setting "K" type "variable"), but they do not return any result. You no longer need WASM to activate any standard Event ID since they now added the SimConnect_TransmitClientEventEx1() functions which take multiple parameter values. Or you could use WASimClient::sendKeyEvent() methods I mentioned.

To activate "HTML Events" which are very popular with add-ons (and maybe some Asobo craft as well) you still need to run calculator code to set "H" type vars (just like with "K" vars), which means WASM.

Some add-ons also activate events by watching L vars for changes (or maybe others?), which again means WASM.

For both of the above types you could use WASimClient::setVariable(), or executeCalculatorCode() directly as mentioned in setVariable() docs.

Best,
-Max

[mpaperno]

Are you familiar with HubHop?

This is probably the most popular integration the Touch Portal plugin has for triggering events, at least on 3rd party aircraft.

For example the plugin users have this action:

The 2 selectors on the left are drill-downs to the Event Name (sort of like on the HH site but more compact). Some events in the HH database also accept values (they call them "potentiometer" types).

Ultimately all those events are RPN calculator code. The plugin has a copy of the HH data and can look up the code to execute based on the selected properties in that action in the screenshot. It then just runs the code via WASimClient (or, if WASimModule isn't detected it adds the "MobiFlight." prefix to the event name and sens it via SimConnect, in case the user does have the MobiFlight WASM installed, which there is no way to detect).

When the plugin user submits that action it eventually ends up here which is actually pretty simple (most of the preamble stuff is getting and validating data from the submitted action like the event identifier and possible value).

The whole HubHop data handling part is encapsulated in Configuration/HubHop. Data is stored in sqlite3 and there's functions included to update the data via https. HubHopPresetsCollection is the main entry point.

Cheers,
-Max

[WilliamRiker2701]

Thank you Max, this conversation made me understand a lot better this subject, and also helped me realize using the simple WASimCommander functions was enough for my purposes.

I just made it work with L variables exactly the way i needed.

I'll keep polishing the code now and experimenting with a few other functions.

Thanks again,
cheers!

-William

[WilliamRiker2701]

Hey Max, i'm struggling a little bit with these methods that use pointers and references in c++.

WASimCommander/src/WASimClient_CLI/WASimClient_CLI.h

Line 145 in 3e671f9

HR getVariable(VariableRequest ^var, [Out] double %pfResult) {

I'm not sure how to use them adequately in C#. I tried to follow the C# example in the testing folder exactly like this:

https://github.com/mpaperno/WASimCommander/blob/main/src/Testing/CS_BasicConsole/Program.cs#L94

But then when i'm running the plugin through the application and it encounters the GetVariable() function, it alerts about an exception and the plugin gets shut off. I think it possibly has to do with the arguments i'm giving it.

"context" is a string with the variable name (for example PUSH_OVHD_OXYGEN_CREW) and "getVar" hasn't been initialized before in the code. The log part doesn't ever execute (before this i was trying also with HR.OK), the plugin shuts off at the previous line. Am i doing it right?

Other question. Using SetLocalVariable() works, but sometimes it doesn't work. Right know i got configured to be triggered with a key from the keyboard, and sometimes i got to press it a lot of times for the switch in the plane to actually move even though every time the function returns HR.OK. ¿Could it be because of a timeout reason?

Tried SetVariable() too and i got the same result as with GetVariable(). I was doing the following:

The log looks like this:

[mpaperno]

Hi William,

But then when i'm running the plugin through the application and it encounters the GetVariable() function, it alerts about an exception and the plugin gets shut off. I think it possibly has to do with the arguments i'm giving it.

I'm not sure why it would crash there, looks OK from the screenshot. Could you try with a "new VariableRequest()" explicitly?

client.getVariable(new VariableRequest(context), out double getVar)

Other question. Using SetLocalVariable() works, but sometimes it doesn't work. Right know i got configured to be triggered with a key from the keyboard, and sometimes i got to press it a lot of times for the switch in the plane to actually move even though every time the function returns HR.OK. ¿Could it be because of a timeout reason?

Have you debugged the actual key press detection part? I'm guessing so if you see the HR.OK result. I'm really not sure what is going on with that... for setVariable() the client just fires off the command to the WASM and doesn't wait around to verify what the server returns. You should enable logging from the server to your local console (like in the C# example) which will let you see any errors on the server side. Also open the MSFS Dev Console while you're testing, because some errors are only going to be reported there (the MSFS Gauge API functions often happily return an "ok" status when actually what you tried to do didn't work).

But why it would be intermittent like that, I have no idea... maybe a threading issue on your side? setVariable() would execute in your current thread, as would the SimConnect call to actually transmit the data (from inside WASmClient).

Tried SetVariable() too and i got the same result as with GetVariable(). I was doing the following:

Got me here too... In the example it runs that setLocalVariable() in a tight loop here and I never saw it not set the variable. Does the example work for you?

The log looks like this:

I'm not sure which crash results in this log (set or get var, or both), but in any case the message is not very useful... at least to me. Something is clearly null which shouldn't be, but at which line or point of your code I cannot tell.

LMK what you find... or if you have some (more) complete code/reproducible example I could look at maybe I'll see something in that. You can shoot me a mail at [email protected] if you prefer not to post it.

Cheers,
-Max

[WilliamRiker2701]

Hey Max, i've been watching at log files and console.

Do these messages ring some bell to you? This happens every single time i try to set some variable. These 4 messages show even when the action successfully moves the switch in the cockpit. The "SendId" value changes arbitrarily every time i try to set the event even though i try to set the exact same variable every time.

Another thing, could you give me an example on when using executeCalculatorCode() to set a local variable, how should look the text that goes into the method? because i think it's not just the bare name of the variable is it?

Thanks!

[mpaperno]

Why is the client connecting and disconnecting every time?

The SendID of the SimConnect exceptions tells us nothing useful. The module should (by default) be tracking the last 25 or so Send IDs and then reporting the actual SimConnect message which caused the exception message. From that log line it looks like that feature is disabled entirely for some reason (is your WASM install from my one of my distros?). There is a setting for this (requestTrackingMaxRecords) in the WASimModule's server_conf.ini which should be in the same Community subfolder as the WASimModule.wasm.

Anyway I can't tell what's going on in your code from one little snippet of a cut-off log screenshot, sorry.

And TBH that's the most awkward way to keep track of the server-side logging. You do see in the C# example where it pulls in all the messages from the server and client right into your console, yes? And you can enable debug-level logging to get more details.

Check HubHop for thousands of calculator code examples.

-Max

[WilliamRiker2701]

Hi Max

Thanks for the suggestions. I have good news this time.
The client is connecting and disconnecting every time because that's the way Voice Attack works, at least that's the way MSFSAgent plugin works. For every action like pressing a key or saying something, it connects to the simulator does what it needs to do, and then disconnects. As this logic worked great with previous versions of the plugin, i didn't thought it was needed to change it.
I had set the logging for client and server at trace level but nothing useful came out of it. Everything seemed to work perfectly. The only "weird" thing was that message in the sim console.
But then i noticed with WASimUI (since that the plugin connection in Voice Attack also showed in the live logging of WASimUI), that failed variable triggering was happening on the ocassions the client didn't disconnect after previous activity. I made sure it disconnects in the code after every action and now it seems to work fine.
Also i figured out the calculator code so i'm exclusively using that function now, the "get variable" part with calc code i haven't tried yet but i don't think i'll have issues.

Anyways, i managed to accomplish what i was looking for, so thank you, your help was crucial.

Cheers!

-William

[mpaperno]

Hi William, good to hear!

I'd recommend not re-connecting for every command, that is just extra overhead for each new connection (it would be even worse with SimConnect actually). The connection isn't an open socket or anything like that, it's just an established session, so the next command doesn't have to re-establish the session again. There are no resources being used while a connection stays "open," unlike, say, keeping a file or a network socket open the whole time. I'm not sure why someone would design it that was in the first place, but maybe out of some sense that it would be more efficient that way -- it isn't.

Also if you need to subscribe to variable changes you'd have to stay connected. Don't use getVariable() in a loop or timer or something and disconnect between uses -- do a proper subscription request if you need regular updates.

that failed variable triggering was happening on the ocassions the client didn't disconnect after previous activity

Can you elaborate on that? Failed variable triggering? "UNRECOGNIZED_ID"?

Honestly there should be no unexplained errors... there are thousands of people using the Touch Portal plugin and I've not heard one report of anything funky with the WASimCommander setup... I'd know if it's spamming the sim logs for people. Not saying there can't be some issue, but in general for what you're trying to do I don't really understand why you'd see any warning or errors.

Did you ever figure out why the SimConnect request tracking isn't enabled?

Cheers,
-Max

[WilliamRiker2701]

I'll definitely consider your suggestions about the connection. I know everything you say about it makes sense, but for some reason i thought the downside of having it connect and disconnect at every action (meaning the longer time spent for every action) wasn't too bad compared to having to rethought the way the code acts in Voice Attack. It doesn't need to be able to input actions incredibly fast one after the other. But now that i think about it, it should be fairly simple to change inside the code.

About the thing you ask, i never knew what the "UNRECOGNIZED_ID" message in the sim console was, it just kept happening even when the variable modification was successful. I meant when i mentioned earlier that with SetLocalVariable() sometimes worked and sometimes didn't, remember? This problem happened specially when trying to pass a series of actions too fast. Well the same thing happened with the calculator code function, and the logging didn't show any error message so that kept me thinking it was maybe about the connection/disconnection part.

Then i was modifying variables through Voice Attack (using calc code) and at the same time watching WASimUI's logging, and i noticed that with only Info level logging, at every successful action (pressed the key, and the switch actually moved in the cockpit) this showed:

Nothing weird about it, but then i noticed sometimes the log only showed the "client connected" line and not the "disconnected" part, meaning, sometimes after an action it didn't disconnect . Everytime this happened, the following action i triggered wasn't successful (pressed the key, nothing happened in the cockpit). So i guess that for some reason i currently don't understand, when trying an action again and wasn't previously disconnected, there was some kind of conflict that prevented the action to pass (i mean, as for the logging the action passed with no errors, but in the sim it didn't show an effect).

Then i noticed the code didn't had the disconnection part active, so i made sure that after every action it goes through the 3 disconnection functions that shows in the example code, and after making this change, the actions passed successfully every single time no matter how fast i triggered them.

About the variables request i haven't tried that part a lot for the moment, but probably soon i'll have to make a section about it in the code.

Cheers!

[mpaperno]

You should use a unique client ID when creating WASimClient. If you're running the example CS script and your app at the same time (w/out restarting MSFS), the server could get confused which client it's talking to. In theory anyway.

I can fire off several WASimClient commands within a ms and everything works peachy (set, get, whatever). For example Touch Portal has "slider" input widgets, eg. to control a throttle, and those can fire events very rapidly. So I'm not sure what you're seeing with your code.

I suppose there's a physical limit to how quickly one can shout commands at a computer though.... :) OTOH if the plugin allows "stacking" events, eg. one voice command fires a series of events, those should get executed ASAP of course. Like "prepare for takeoff" could initiate a whole series of commands for instance.

-Max