heyoutline / cloud_watch Goto Github PK

The logs are showing up in the console with local time, not UTC.

👋

I'm getting

CaseClauseError: no case clause matching: {:error, {"InvalidSequenceTokenException", "The given sequenceToken is invalid. The next expected sequenceToken is: 49619543027323424921201786257049013329532568378470856290", "49619543027323424921201786257049013329532568378470856290"}}
  File "lib/cloud_watch/aws_proxy.ex", line 124, in CloudWatch.AwsProxy.request/2
  File "lib/cloud_watch.ex", line 186, in CloudWatch.do_flush/4
  File "lib/cloud_watch.ex", line 47, in CloudWatch.handle_info/2
  File "gen_event.erl", line 627, in :gen_event.server_update/4
  File "gen_event.erl", line 609, in :gen_event.server_notify/4
  File "gen_event.erl", line 398, in :gen_event.handle_msg/6
  File "proc_lib.erl", line 226, in :proc_lib.init_p_do_apply/3

since the match in CloudWatch.AwsProxy.request/2 expects {:error, {type, info}} and {:error, {type, into, sequenceToken}} is returned from ex_aws.

I'm using ex_aws 2.2.4 that was recently released.

Fix: #25

I'd like to propose following updates:

• Create new branch release_0.3 for compatibility with older applications
• Drop support for Elixir 1.5 (builds are failing already). Use release_0.3 if 1.5 is needed
• Replace Travis CI with GitHub actions. Define matrix of Erlang + Elixir versions to test against. See PR #16
• Remove dependency on Poison. This would bring relief to projects that are using exclusively Jason as their JSON encoder, see issues #13, #15. Also note that both aws-elixir and ExAws are switching from Poison to Jason. See PR #17
• Add support for newer versions of aws-elixir. PR #14 submitted upgrade to 0.6.0, but 0.7.0 is already out (with incompatible module names). We should support all AWS versions from 0.5.0 upward
• Permit wide range of versions of httpoison. We should not restrict users in their choices
• Fix application boundary warnings for Elixir 1.11

Hi, I'm ready to submit a pull request with metadata filtering and matching feature but I just saw @pmenhart fork and it has some interesting features that I would like to use in my code.

@pmenhart what do you think about pushing your code in this repo (or @lboekhorst the other way around)? I think it would be more useful for the community to have a single repo with all developed features.

Hello, I had this problem here, and I do not know what I can do to solve it.

It looks like I've exceeded the limit, but I do not have a boundary value set.

Process #PID<0.20400.23> terminating: {:EXIT, {{:case_clause, {:error, {"ThrottlingException", "Rate exceeded"}}}, [{CloudWatch, :flush, 2, [file: 'lib/cloud_watch.ex', line: 93]}, {:gen_event, :server_update, 4, [file: 'gen_event.erl', line: 577]}, {:gen_event, :server_notify, 4, [file: 'gen_event.erl', line: 559]}, {:gen_event, :handle_msg, 6, [file: 'gen_event.erl', line: 300]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}

Apparently, my application went offline because of this error. Can this happen or does the dependency work in isolation and an error returned from Amazon would not knock out an application?

I'm trying to run mix phx.digest in my app and it throws the following error.
== Compilation error in file lib/cloud_watch/input_log_event.ex == ** (ArgumentError) Poison.Encoder is not available (elixir) lib/protocol.ex:69: Protocol.assert_protocol!/2 lib/cloud_watch/input_log_event.ex:4: (module) (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6 could not compile dependency :cloud_watch, "mix compile" failed. You can recompile this dependency with "mix deps.compile cloud_watch", update it with "mix deps.update cloud_watch" or clean it with "mix deps.clean cloud_watch"

It seems that there is not a dependency for poison in the cloudwatch project, I'm currently using Jason, there is an option for selecting another JSON encoder?

Both Timber and CloudWatch installed in project so I was able to view the logs on Timber even though CloudWatch was crashing due to the log formatter.

Getting this error in logs of Timber:
CloudWatch installed in Logger terminating ** (Protocol.UndefinedError) protocol Jason.Encoder not implemented for {{2020, 11, 13}, {14, 32, 36, 913}} of type Tuple, Jason.Encoder protocol must always be explicitly implemented. This protocol is implemented for the following type(s): CloudWatch.InputLogEvent, Ecto.Association.NotLoaded, Ecto.Schema.Metadata, DateTime, Integer, List, Float, Map, Time, Atom, Decimal, NaiveDateTime, Date, Any, BitString, Jason.Fragment (jason 1.2.2) lib/jason.ex:150: Jason.encode!/2 (ex_aws 2.1.6) lib/ex_aws/request.ex:19: ExAws.Request.request/6 (ex_aws 2.1.6) lib/ex_aws/operation/json.ex:49: ExAws.Operation.ExAws.Operation.JSON.perform/2 (cloud_watch 0.3.2) lib/cloud_watch/aws_proxy.ex:61: CloudWatch.AwsProxy.request/2 (cloud_watch 0.3.2) lib/cloud_watch.ex:121: CloudWatch.do_flush/4 (cloud_watch 0.3.2) lib/cloud_watch.ex:41: CloudWatch.handle_info/2 (stdlib 3.11) gen_event.erl:577: :gen_event.server_update/4 (stdlib 3.11) gen_event.erl:559: :gen_event.server_notify/4 (stdlib 3.11) gen_event.erl:347: :gen_event.handle_msg/6 (stdlib 3.11) proc_lib.erl:249: :proc_lib.init_p_do_apply/3 Last message: :flush State: %{buffer: [%CloudWatch.InputLogEvent{message: "[debug] QUERY OK source=\"event\" db=1.7ms\nSELECT max(e0.\"createTimeStamp\") FROM \"event\" AS e0 WHERE (e0.\"gameID\" = $1) [<<196, 234, 188, 198, 236, 184, 69, 45, 155, 81, 180, 126, 71, 252, 32, 236>>] \n", timestamp: {{2020, 11, 13}, {14, 32, 36, 913}}}], buffer_length: 1, buffer_size: 225, client: %{}, flushed_at: nil, format: {Statstrack.CloudWatchFormatter, :format}, level: :debug, log_group_name: "statstrack-web", log_stream_name: "statstrack-web-statstrack-prod-prod", max_buffer_size: 10485, max_timeout: 60000, sequence_token: nil}

The previous hurdle I managed to clear was CloudWatch crashing on CloudWatch.InputLogEvent in a similar fashion to how it is crashing here with the Erlang date type. I added a custom log formatter so it would catch this type so Jason wouldn't crash.

defmodule MyApp.CloudWatchFormatter do
  require Protocol

  Protocol.derive(Jason.Encoder, CloudWatch.InputLogEvent, only: [:message, :timestamp])
  # If I could put in another derive here to catch the tuple that would be great but there's no way for derive to do that afaik

  @pattern Logger.Formatter.compile("[$level] $message $metadata\n")

  def format(level, message, timestamp, metadata) do
    Logger.Formatter.format(@pattern, level, message, timestamp, [])
  rescue
    error -> "could not format: #{inspect(error)} #{inspect({level, message, metadata})}\n"
  end
end

And then plugged that in to the cloud_watch config:

config :logger, CloudWatch,
  format: {MyApp.CloudWatchFormatter, :format},
  level: :debug,
  log_group_name: "myapp-web",
  log_stream_name: "myapp-logs",
  max_buffer_size: 10_485,
  max_timeout: 60_000,
  metadata: []

This can't address the Erlang timestamp which looks like: {{2020, 11, 13}, {14, 32, 36, 913}} and I can't accommodate the tuple in the formatter the way I can with a module like CloudWatch.InputLogEvent.

I tried forking this lib and doing the formatter myself making it safe for Jason but I couldn't get it working so in the end I just switched ex_aws json parser back to Poison and now it's working... or possibly just on to the next iteration of problem.

It would be great if someone smarter than I could make this change so Jason can handle it.

My production application uses this package to push our server logs to CloudWatch and everything has been great until this morning. Our server ended up crashing. We got a lot of error messages related to this package, but it's still unclear what lead the application to crash completely. From my understanding, the processes created by this package should be spawned by Logger and so a cascading failure here shouldn't affect the main app, but I could be wrong there.

We ended up getting about 200 of the following messages:

** (exit) {:EXIT, {{:case_clause, {:error, {:closed, ""}}}, [{CloudWatch.AwsProxy, :request, 2, [file: 'lib/cloud_watch/aws_proxy.ex', line: 61]}, {CloudWatch, :flush, 2, [file: 'lib/cloud_watch.ex', line: 93]}, {CloudWatch, :handle_info, 2, [file: 'lib/cloud_watch.ex', line: 46]}, {:gen_event, :server_update, 4, [file: 'gen_event.erl', line: 577]}, {:gen_event, :server_notify, 4, [file: 'gen_event.erl', line: 559]}, {:gen_event, :handle_msg, 6, [file: 'gen_event.erl', line: 347]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}

It seems that the request to flush the buffer failed and there was no clause in the case statement opened on line 93 to handle it. Looking over the AwsProxy, it seems that, when using the AWS package, it should return {:error, HTTPoison.Error.t} and that's what the case statement expects. However, we're using ExAWS and that seems to only return {:error, {type, message}} so it can't match.

We're using ex_aws @ 2.1.0 and cloud_watch @ 0.3.0

It looks like it will only flush() when the buffer is full? If there is a log entry sitting there that hasn't filled the buffer, it will sit forever? Is there any time-based flush as well?