refactor: switch cachex stats to telemetry polling

[ruslandoga]

ANL-1033

Removed log-based Cachex stats collection in favor of telemetry-based polling system.

[ruslandoga]

This evaluates to

[
  {Logflare.TeamUsers.Cache, :team_users},
  {Logflare.Partners.Cache, :partners},
  {Logflare.Users.Cache, :users},
  {Logflare.Backends.Cache, :backends},
  {Logflare.Sources.Cache, :sources},
  {Logflare.Billing.Cache, :billing},
  {Logflare.SourceSchemas.Cache, :source_schemas},
  {Logflare.Auth.Cache, :auth},
  {Logflare.Endpoints.Cache, :endpoints},
  {Logflare.Rules.Cache, :rules},
  {Logflare.SavedSearches.Cache, :saved_searches}
]

or should it be hardcoded?

[Ziinc]

can you inverse the logic so that we can do list_contexts_to_cache() and in list_caches it will call list_contexts_to_cache/0 and concat the .Cache

[Ziinc]

the module.split isn't very pretty 😆

[Ziinc]

besides that, its fine with keeping it dynamic.

[ruslandoga]

I went into a slightly different direction and added

  def list_caches_with_metrics do
    [
      {TeamUsers.Cache, :team_users},
      {Partners.Cache, :partners},
      {Users.Cache, :users},
      {Backends.Cache, :backends},
      {Sources.Cache, :sources},
      {Billing.Cache, :billing},
      {SourceSchemas.Cache, :source_schemas},
      {Auth.Cache, :auth},
      {Endpoints.Cache, :endpoints},
      {Rules.Cache, :rules},
      {KeyValues.Cache, :key_values},
      {SavedSearches.Cache, :saved_searches}
    ]
  end

I think it's a bit safer for atoms to exist in the source code.

[ruslandoga]

@Ziinc 👋

I think this is ready for review.

[Ziinc]

just minor refactoring for cleaner code

[Ziinc]

can you inverse the logic so that we can do list_contexts_to_cache() and in list_caches it will call list_contexts_to_cache/0 and concat the .Cache

[Ziinc]

the module.split isn't very pretty 😆

[Ziinc]

besides that, its fine with keeping it dynamic.

[ruslandoga]

It appears to be working with local collector, no errors in logflare-collector-1 logs and Logflare shows some events incoming:

Logflare dashboard screenshot

It might be worth adding to DEVELOPMENT.md that the collector in docker-compose.yml doesn't work out of the box for external contributors and its config needs to be modified to point to local logflare with a source/api keys/etc. set up :)

Off-topic OTEL-related errors

[error] [] Handler {OtelMetricExporter.TelemetryHandlers, :otel_metric_exporter,
 [:logflare, :repo, :query]} has failed and has been detached. Class=:error
Reason=:badarg
Stacktrace=[
  {:erlang, :round, [nil], [error_info: %{module: :erl_erts_errors}]},
  {OtelMetricExporter.MetricStore, :write_metric, 5,
   [file: ~c"lib/otel_metric_exporter/metric_store.ex", line: 116]},
  {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1688]},
  {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1688]},
  {:telemetry, :"-execute/3-fun-0-", 4,
   [
     file: ~c"/Users/x/Developer/logflare/deps/telemetry/src/telemetry.erl",
     line: 167
   ]},
  {:lists, :foreach_1, 2, [file: ~c"lists.erl", line: 2310]},
  {Ecto.Adapters.SQL, :log, 5, [file: ~c"lib/ecto/adapters/sql.ex", line: 1292]},
  {DBConnection, :log, 5, [file: ~c"lib/db_connection.ex", line: 1701]},
  {Postgrex, :query, 4, [file: ~c"lib/postgrex.ex", line: 298]},
  {Ecto.Adapters.SQL, :struct, 10,
   [file: ~c"lib/ecto/adapters/sql.ex", line: 1166]},
  {Ecto.Repo.Schema, :apply, 4, [file: ~c"lib/ecto/repo/schema.ex", line: 1000]},
  {Ecto.Repo.Schema, :"-do_insert/4-fun-4-", 15,
   [file: ~c"lib/ecto/repo/schema.ex", line: 500]},
  {Ecto.Repo.Schema, :"-wrap_in_transaction/6-fun-0-", 3,
   [file: ~c"lib/ecto/repo/schema.ex", line: 1285]},
  {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3,
   [file: ~c"lib/ecto/adapters/sql.ex", line: 1458]},
  {...},
  ...
]

Note that telemetry detaches all failed handlers permanently, no retries. So exporters probably should be able to handle expected errors without throws/exits/errors.