get_metric_statistics returns error in library but not on cli #18

brpandey · 2019-07-19T21:12:23Z

Not sure where this goes - ex_aws or ex_aws_cloudwatch

The aws CLI tools works fine when I run get-metric_statistics -- I always get data

aws --profile XXXdev_admin cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2019-07-15T21:54:55.096486Z --end-time 2019-07-16T21:54:55.096486Z --period 3600 --namespace AWS/EC2 --statistics Average --dimensions Name=InstanceId,Value=i-XXX61

{
    "Label": "CPUUtilization",
    "Datapoints": [
        {
            "Timestamp": "2019-07-16T01:54:00Z",
            "Average": 0.11666666666666667,
            "Unit": "Percent"
        }, 
... 
}

However when I run using ExAws/ExAws.Cloudwatch, I get the SweetXML(v0.6.6) error below

iex(15)> x
%ExAws.Operation.Query{
  action: :get_metric_statistics,
  params: %{
    "Action" => "GetMetricStatistics",
    "Dimensions.member.1.Name" => "InstanceId",
    "Dimensions.member.1.Value" => "i-XXX61",
    "EndTime" => "2019-07-16T21:54:55.096486Z",
    "MetricName" => "CPUUtilization",
    "Namespace" => "AWS/EC2",
    "Period" => 3600,
    "StartTime" => "2019-07-15T21:54:55.096486Z",
    "Statistics.member.1" => "Average",
    "Version" => "2010-08-01"
  },
  parser: &ExAws.Cloudwatch.Parsers.parse/2,
  path: "/",
  service: :monitoring
}
iex(16)> x |> ExAws.request()
** (MatchError) no match of right hand side value: :error
    (sweet_xml) lib/sweet_xml.ex:667: SweetXml.to_cast/3
    (sweet_xml) lib/sweet_xml.ex:441: SweetXml.xpath/2
    (sweet_xml) lib/sweet_xml.ex:531: anonymous fn/3 in SweetXml.xmap/3
    (elixir) lib/map.ex:791: Map.get_and_update/3
    (sweet_xml) lib/sweet_xml.ex:531: SweetXml.xmap/3
    (sweet_xml) lib/sweet_xml.ex:530: SweetXml.xmap/3
    (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
    (sweet_xml) lib/sweet_xml.ex:526: anonymous fn/4 in SweetXml.xmap/3

Its not clear what is actually happening and what the aws response was?

The text was updated successfully, but these errors were encountered:

brpandey · 2019-07-19T21:16:43Z

I will say that

    ExAws.Cloudwatch.list_metrics(namespace: "AWS/EC2") |> ExAws.request()

works everytime

brpandey · 2019-07-22T00:51:48Z

I'd like to see the raw response back from aws. Is this accessible via ExAws? Any thoughts @benwilson512

Appreciated

cammellos · 2019-07-22T04:16:42Z

@brpandey sorry for the late reply, I am currently on holiday :). I can take a look at this in a couple of days, and let you know! Thanks for reporting the issue.

brpandey · 2019-07-22T15:14:43Z

@cammellos Enjoy your vacation! In the meantime, I will see if I can figure this out

brpandey · 2019-07-22T16:30:37Z

Update --

Looking at the ExAws docs you can easily override the default http client behaviour with your own. So I've added these entries

config :ex_aws,
  http_client: ExAws.Request.HTTPClient2

and (basically wrapping the default hackney call with an IO.inspect)

defmodule ExAws.Request.HTTPClient2 do
  @behaviour ExAws.Request.HttpClient

  def request(method, url, body \\ "", headers \\ [], http_opts \\ []) do
    resp = ExAws.Request.Hackney.request(method, url, body, headers, http_opts) |> IO.inspect(label: "httpclient2")

    resp
  end
end

Which when triggered and for my specific usecase produces:

iex(25)> ExAws.Cloudwatch.get_metric_statistics("AWS/EC2", "CPUUtilization", start_time, end_time, 3600, options) |> ExAws.request()
httpclient2: {:ok,
 %{
   body: "<GetMetricStatisticsResponse xmlns=\"http://monitoring.amazonaws.com/doc/2010-08-01/\">\n  <GetMetricStatisticsResult>\n    <Datapoints>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T11:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T06:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T16:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T19:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.11666666666666667</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T01:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T04:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T14:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T09:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T12:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-15T23:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T17:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-15T21:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T20:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T07:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T02:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T10:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.13333333333333333</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T05:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.016666666666666666</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T15:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T18:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T13:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T00:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T03:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-15T22:54:00Z</Timestamp>\n      </member>\n      <member>\n        <Average>0.0</Average>\n        <Unit>Percent</Unit>\n        <Timestamp>2019-07-16T08:54:00Z</Timestamp>\n      </member>\n    </Datapoints>\n    <Label>CPUUtilization</Label>\n  </GetMetricStatisticsResult>\n  <ResponseMetadata>\n    <RequestId>aba280d4-ac9c-11e9-b6ee-ff0676301ff5</RequestId>\n  </ResponseMetadata>\n</GetMetricStatisticsResponse>\n",
   headers: [
     {"x-amzn-RequestId", "aba280d4-ac9c-11e9-b6ee-ff0676301ff5"},
     {"Content-Type", "text/xml"},
     {"Content-Length", "3835"},
     {"vary", "accept-encoding"},
     {"Date", "Mon, 22 Jul 2019 16:20:50 GMT"}
   ],
   status_code: 200
 }}
** (MatchError) no match of right hand side value: :error
    (sweet_xml) lib/sweet_xml.ex:667: SweetXml.to_cast/3
    (sweet_xml) lib/sweet_xml.ex:441: SweetXml.xpath/2
    (sweet_xml) lib/sweet_xml.ex:531: anonymous fn/3 in SweetXml.xmap/3
    (elixir) lib/map.ex:791: Map.get_and_update/3
    (sweet_xml) lib/sweet_xml.ex:531: SweetXml.xmap/3
    (sweet_xml) lib/sweet_xml.ex:530: SweetXml.xmap/3
    (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
    (sweet_xml) lib/sweet_xml.ex:526: anonymous fn/4 in SweetXml.xmap/3

Now to see what SweetXml is trying to do...

brpandey · 2019-07-22T17:08:04Z

The error happens in this block of code where xml is the body string value from above

parsed_body = SweetXml.xpath(xml,
             ~x"//GetMetricStatisticsResponse",
             metric_statistics:    [
        	~x"./GetMetricStatisticsResult/Datapoints/member"l,
	        average: ~x"./Average/text()"f,
	        # TODO probably fix this
        	extended_statistics: ~x"./ExtendedStatistics/text()"m,
	        maximum: ~x"./Maximum/text()"f,
        	minimum: ~x"./Minimum/text()"f,
        	sample_count: ~x"./SampleCount/text()"f,
        	sum: ~x"./Sum/text()"f,
        	timestamp: ~x"./Timestamp/text()"s,
        	unit: ~x"./Unit/text()"s
             ],
             label: ~x"./GetMetricStatisticsResult/Label/text()"s,
             request_id: ~x"./ResponseMetadata/RequestId/text()"s
           )

brpandey · 2019-07-22T18:25:11Z

Xpath values in the x_sigill which are floats (f) that are not found in the xml body will error, thus instead of putting f an optional f should be used and it will parse correctly, e.g. of

https://hexdocs.pm/sweet_xml/SweetXml.html#sigil_x/2

like such:


parsed_body = SweetXml.xpath(xml,
             ~x"//GetMetricStatisticsResponse",
             metric_statistics:    [
        	~x"./GetMetricStatisticsResult/Datapoints/member"l,
	        average: ~x"./Average/text()"of,
	        # TODO probably fix this
        	extended_statistics: ~x"./ExtendedStatistics/text()"m,
	        maximum: ~x"./Maximum/text()"of,
        	minimum: ~x"./Minimum/text()"of,
        	sample_count: ~x"./SampleCount/text()"of,
        	sum: ~x"./Sum/text()"of,
        	timestamp: ~x"./Timestamp/text()"s,
        	unit: ~x"./Unit/text()"s
             ],
             label: ~x"./GetMetricStatisticsResult/Label/text()"s,
             request_id: ~x"./ResponseMetadata/RequestId/text()"s
           )

schnittchen · 2021-03-24T19:37:44Z

I ran into the same issue and it looks like the PR #19 still works.

7evam · 2025-01-13T02:46:05Z

I ran into the same issue getting this error

23:43:52.182 [error] GenServer #PID<0.206.0> terminating
** (MatchError) no match of right hand side value: :error
    (sweet_xml 0.7.5) lib/sweet_xml.ex:873: SweetXml.to_cast/3
    (sweet_xml 0.7.5) lib/sweet_xml.ex:630: SweetXml.xpath/3
    (sweet_xml 0.7.5) lib/sweet_xml.ex:737: anonymous fn/3 in SweetXml.xmap/3
    (elixir 1.18.1) lib/map.ex:957: Map.get_and_update/3
    (sweet_xml 0.7.5) lib/sweet_xml.ex:737: SweetXml.xmap/3
    (sweet_xml 0.7.5) lib/sweet_xml.ex:736: SweetXml.xmap/3
    (elixir 1.18.1) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
    (sweet_xml 0.7.5) lib/sweet_xml.ex:732: anonymous fn/4 in SweetXml.xmap/3
Last message: :fetch_metrics
State: %{}

I added some logging to sweet_xml and found that it was expecting all statistics in opts. I was only passing through Maximum. I updated my code to the below and the error went away.

    end_time = DateTime.utc_now()
    start_time = DateTime.add(end_time, -5 * 60, :second)

    opts = [
      {:statistics, ["SampleCount", "Maximum", "Sum", "Minimum", "Average"]},
      {:dimensions, [{"InstanceId", @instance_id}]}
    ]

    ExAws.Cloudwatch.get_metric_statistics(
           "AWS/EC2",
           "CPUUtilization",
           start_time,
           end_time,
           300,
           opts
         )

This seems like the same bug that should have been fixed in the previous PRs. I'm using what seems to be the latest version, 2.0.4. Is this fix going to be updated or is there a newer version?

brpandey changed the title ~~get~~ get_metric_statistics returns error in library but not on cli Jul 19, 2019

brpandey mentioned this issue Jul 22, 2019

Fixed parser error on get metric statistics #19

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

get_metric_statistics returns error in library but not on cli #18

get_metric_statistics returns error in library but not on cli #18

brpandey commented Jul 19, 2019 •

edited

Loading

brpandey commented Jul 19, 2019

brpandey commented Jul 22, 2019 •

edited

Loading

cammellos commented Jul 22, 2019

brpandey commented Jul 22, 2019

brpandey commented Jul 22, 2019

brpandey commented Jul 22, 2019

brpandey commented Jul 22, 2019 •

edited

Loading

schnittchen commented Mar 24, 2021

7evam commented Jan 13, 2025

get_metric_statistics returns error in library but not on cli #18

get_metric_statistics returns error in library but not on cli #18

Comments

brpandey commented Jul 19, 2019 • edited Loading

brpandey commented Jul 19, 2019

brpandey commented Jul 22, 2019 • edited Loading

cammellos commented Jul 22, 2019

brpandey commented Jul 22, 2019

brpandey commented Jul 22, 2019

brpandey commented Jul 22, 2019

brpandey commented Jul 22, 2019 • edited Loading

schnittchen commented Mar 24, 2021

7evam commented Jan 13, 2025

brpandey commented Jul 19, 2019 •

edited

Loading

brpandey commented Jul 22, 2019 •

edited

Loading

brpandey commented Jul 22, 2019 •

edited

Loading