Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update momo shell with better help features and ability to add packages #153

Merged
merged 1 commit into from
Jan 7, 2015
Merged

Update momo shell with better help features and ability to add packages #153

merged 1 commit into from
Jan 7, 2015

Conversation

timburke
Copy link
Contributor

@timburke timburke commented Jan 4, 2015

This PR adds a new syslog implementation that does not require sprintf'ing things, allows the logging of typed data and in the future can facilitate hardware testing by allowing the logging of typed data into a special 'test' log that can be inspected to see if hardware tests were successful for or not. The implementation of syslog and usage is detailed here.

The reason why I wanted to change it was so that we can get better data on running momo systems including the actual RAM variables for debugging and development purposes. This required some fairly extensive additional work to automatically match python types with firmware types automatically and is used in syslog as well as read_ram to read controller ram variables and in the future can be used for symbolic debugging in the simulators as well.

The main disadvantage of the new systemlog vs the old one is that you now need a log definition file in order to translate the log messages into english text and match types with the logged parameters. If you don't have a log definition handy for the module, the syslog will do the best it can and always show you as much as it can with the information that it has. There is enough metadata stored with each logged parameter to display it as either an integer, ascii string or binary blob without a log definition but with a log definition file you can give each parameter a name and associate rich types with each binary blob that can have complicated processing and display routines.

Changes from Old System Log

  • any amount of data can be logged with a message
  • the message string is not embedded in the log entry but instead a unique 32-bit identifier is stored that is used to look up the message and learn how to parse and display the typed data (as detailed in 92-SystemLog.md linked above)
  • modules that want to use the syslog need to define the format of the log entries that they use in a log_definition.ldf file according to the above linked documentation. This has been done for mainboard and shared/pic24 code.
  • the systemlog can be inspected and displayed using the momo tool

Changes to the MoMo Tool

  • Documentation for the MoMo tool is now here
  • typedargs has been updated to allow for dynamically adding types from external packages. This is needed so that firmware modules can define python types for the variables that they wish to log or otherwise expose to the user.
  • MoMo has an optional config file that allows the running of initialization commands for certain contexts to repetitively inject types and log definitions for commonly used modules. This is described in the above documentation for the momo tool.

Additional Changes

  • Scons now tracks pic24 include statements to add dependencies on header files so that modifying a header triggers a rebuild.
  • MoMo tool help system is fixed (it could throw exceptions sometimes before)
  • MoMo tool compatibility with windows is fixed to have working tab completion
  • Unit tests are now run for pymomo using the nose testing framework (provision.sh and build_all.sh have been updated to run these tests and import the required packages)

Additional work that can be done on these changes (for a future PR) is detailed in Issue #152.

@timburke
Copy link
Contributor Author

timburke commented Jan 4, 2015

Hold off on this one for a sec. I need to resolve a build dependency issue.

- You can now add a package and have it show up as a context in momo
- Functions are now listed with short descriptions
- Various bugs dealing with handling contexts are fixed

Update pymomo unit tests for compatibility with API.

Update syslog and add unit tests

Add LDF compiler and add header dependency tracking to xc16 build system

Update all logging statements to use the new logging format

Finish initial support for system log reading and parsing.

- supports generating headers from ldf files during build
- supports reading the system log over RPC and displaying it
- supports parsing the system log messages using ldf files

Finished version 1 of system log code

- supports displaying parameters with type information

Initial documention

Update types to correspond with typedargs types

Verified that automatic wrapping of long log data works

Update momo tool to be able to clear the system log

- also remove old modtool log command since this functionality
is now part of momo.

Allow syslog to display lists of firmware types

- log definitions can now have lists of variable length
- can now display complex types like scheduled_task and task structures
- types now can have a default formatting function

Update shared pic24 code for compat with field_service_unit

- When no flash is available for persistent logging, just keep
everything in the ring buffer.

Update documentation to deal with lists of parameters

Update doc for firmware types

Update system_log documentation

Finish initial version of syslog code

Update tests and add ringbuffer test code

Update pymomo with initial syslog reading implementation

Update wording on docs

Update unit tests for syslog python component

Update automation scripts to run python unit tests

Add unit testing for injecting types from external python packages

Update momo tool to 0.9

- fix help string printing
- refactor shell into HierarchicalShell class in typedargs
- allow type injection into momo tool
- allow context initialization using rcfile
- create RCFile
- move firmware_types to shared/pic24 where it belongs

Add momo tool documentation

Update momo documentation

Check and fix windows compatibility

Remove debugging statement
@timburke
Copy link
Contributor Author

timburke commented Jan 5, 2015

Alright this should be good to go now. I had to update the log definition file compiler to pull in any special python types that the log might reference and added an option to run momo without using its config file so that builds can be repeatable without any dependence on the machine's environment (i.e. the contents of the momo_config.txt file).

@amcgee
Copy link
Member

amcgee commented Jan 7, 2015

I have a few thoughts about how we could make this even more awesome, but it's pretty awesome as-is!

Merging, will open an issue for future enhancements.

amcgee added a commit that referenced this pull request Jan 7, 2015
Update momo shell with better help features and ability to add packages
@amcgee amcgee merged commit 728c07d into WellDone:dev Jan 7, 2015
@timburke
Copy link
Contributor Author

timburke commented Jan 8, 2015

Looking forward to it. I love the awesomeness.

On 1/7/15 3:58 PM, Austin McGee wrote:

I have a few thoughts about how we could make this even more awesome,
but it's pretty awesome as-is!

Merging, will open an issue for future enhancements.


Reply to this email directly or view it on GitHub
#153 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants