Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fetching streams and events within a time range #90

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

slack0
Copy link
Contributor

@slack0 slack0 commented Jul 3, 2019

This PR adds the long-desired feature of fetching stream data, events and raw_events by bounding them within specific start and end points, if required.

Currently, AnalysisGroup's fetch_stream(), fetch_events() and fetch_raw_events() download all the data associated with the input slug. This feature is fine with the slug corresponds to a specific device or a datablock.

There is a lot of good functionality implemented into cloud_api - persistent connection, concurrent fetching of resources, HTTP/REST error handling, fetching of API resources by object etc.

To use all of this functionality implemented into AnalysisGroup and IOTileCloudChannel for other applications - specifically arch_factory's machine models, fetch methods in AnalysisGroup need to take time range into account. This is to prevent DB loading on the cloud.

Since AnalysisGroup and its methods are used in several reports - this change could be disruptive - so enabling this by retaining the default behavior of the fetch_* methods.

What's in this PR:

  • Updated fetch_stream() method to allow start and end specification. They are set to None by default. The input to these arguments could be a valid datetime-compatible string or a datetime object

  • date_utils.py get_utc_ts - that accepts start, end arguments and creates a valid, ISO-format UTC timestamp from the arguments. If timezone information is not present in the arguments, then UTC timezone is forced

  • Updated implementation of fetch_datapoints() in IOTileCloudChannel to account for start and end to df and data API

TODO:

  • Fix test failures and add robust tests
  • Update fetch_events and fetch_raw_events methods

slack0 added 6 commits June 27, 2019 06:51
…e range

- This commit adds specifying `start` and `end` timestamps to pass to
`fetch_datapoints` method

- Either or both arguments can be set to None
- Simplified/DRY'ed `fetch_datapoints` in `IOTileCloudChannel` to pass UTC
timestamps in ISO format to df or data API

- Added utility to handle datetime-compatible string or datetime object in calls
to `AnalysisGroup.fetch_datapoints`

- Updated relative paths
@slack0 slack0 requested review from timburke and dkarchmer July 3, 2019 01:11
@slack0
Copy link
Contributor Author

slack0 commented Jul 3, 2019

@dkarchmer This is the PR for adding time range based querying in AnalysisGroup.

I am basically modifying the API calls to take in the start and end arguments - which they already support. However, I am having test failures in iotile_analytics_core. In particular, see the following call that fails. But when I use the same API on iotile.cloud (with a different stream slug), it works.

"GET /api/v1/df/?filter=s--0000-0077--0000-0000-0000-00d2--5001&format=csv&mask=1&start=2019-07-02T15%3A49%3A00Z HTTP/1.1" 200

Wanted to check with you if the test failure: https://travis-ci.org/iotile/iotile_analytics/jobs/553507199 is related / specific to the mock_cloud used in the tests.

Copy link
Contributor

@dkarchmer dkarchmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this has been on the back burner for long enough. The changes looks safe enough and backwards compatible to approve

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants