ianks / attr-gather Goto Github PK

Hit a million different APIs and combine the results in one simple hash (without pulling your hair out). A simple workflow system to gather aggregate attributes for something.

License: MIT License

Shell 1.63% Ruby 98.37%

ruby workflows dry-rb api aggregator rails

attr-gather's Introduction

`about:me`

My name is Ian, welcome to my profile. My philosophy in life is that keeping a beginner's mindset, staying grateful and helping others are the keys to success. I try to do all three of those things and stumble on the way.

I'm also bad at organizing profile pages, so here's a brain dump. Enjoy!

`about:facts`

📍 Live in Atlanta, GA.
📥 Currently job is Staff Production Engineer @Shopify (hey Shopifolk!).
🎹 Produced electronic music as fnesse and iONik.
🎾 Tennis guru, Serena + Federer all day.
🇬🇧 50% British, 75.2% sarcastic.
🧼 In recovery (13 clean + serene).
❤️ Is the answer.

`about:skills`

🤝 Helping others first
🤘 Rust
💎 Ruby
🤔 Kubernetes + Terraform
🧪 React + Typescript
💥 Segfaulting in C

`about:opensource`

`about:misc`

twitter e-mail linkedin www eth

attr-gather's People

Contributors

Stargazers

Watchers

Forkers

psparrow chubchenko standardgalactic

attr-gather's Issues

Filter not applied unless defined after an aggregator

The following two conditions result in the filter being ignored:

an aggregator is not explicitly defined
an aggregator is defined before the filter

Simple DSL for contract

Instead of:

      Contract = Class.new(Dry::Validation::Contract) do
        params do
          required(:order_id).filled(:int?)
          optional(:shopify_order).filled(:hash?)
        end

        rule(:shopify_order) do
          key.failure('id must match input order_id') if values[:order_id] != values[:shopify_order][:id]
        end
      end

      filter :contract, Contract.new

Make it:

     filter :contract do
        params do
          required(:order_id).filled(:int?)
          optional(:shopify_order).filled(:hash?)
        end

        rule(:shopify_order) do
          key.failure('id must match input order_id') if values[:order_id] != values[:shopify_order][:id]
        end
      end

Explicit schemas / contracts?

It would be nice for the workflow to verify the results of task / final result to ensure no bunk data sneaks through.

Imagine we have a task:

register :fetch_product do |sku:|
   { rooooom_image_url: 'https://image.com/i.png' }
end

It would be nice to know that invalid data is being provided somehow, maybe a contract:

class ProductContract < Dry::Validation::Contract
  params do
    optional(:room_image_url).filled(:string)
  end
end

But there are a few open questions in my mind:

Do we abort the workflow if a task encounters bunk data?
Do we filter the bunk data and move on?
How do we notify the developer that bunk data is coming through?
Do we even need to verify at all?

Any feedback here would be greatly appreciated.

Declare task dependencies like rake

What do you think of declaring task dependencies similar to how rake does it?

old:

task :fetch_user do |t|
  t.depends_on = [:fetch_post]
end

new:

task fetch_user: :fetch_post

Rename task to something else?

Currently, the thing that does the fetching is called a task, should this be renamed? Currently:

task :fetch_post do |t|
  t.depends_on = [:foo]
end

Other Options

gather

gather :wordpress_blog_post do |t|
  t.depends_on = [:foo]
end

fetch

fetch :wordpress_blog_post do |t|
  t.depends_on = [:foo]
end

Can you think of a better name?

Allow for simple `inline` container

Sometimes it's nice to be able to have a method be called on the class, so something like this (with less boilerplate):

    class EnhanceMarketplaceOrder
      include Attr::Gather::Workflow

      container self

      task :fetch_shopify_order do |t|
        t.depends_on = []
      end

      def self.resolve(name)
        method(name)
      end

      def self.fetch_shopify_order(order_id:, **input)
        shopify_http = ::Shopify::HTTP.pool
        base_url = Shopify::Configuration.base_rest_api_path
        url = "#{base_url}/orders/#{order_id}"
        shopify_order = shopify_http.with { |p| p.get(url) }.parse

        { **input, shopify_order: shopify_order[:order] }
      end
    end

Container for registering task implementations, yay or nay?

Currently, task implementations are registered in a Dry::Container. The idea behind this is to seperate the workflow definition from the actual implementation code. This should allow for easier testing and make the code more readable. Also, there is an option to register alternate data providers that way.

# ./blog_post_enhancer/implementation.rb
module BlogPostEnhancer
  class Implementation
    extend Dry::Container::Mixin

    register 'fetch_post' do |id:, **_attrs|
      res = HTTP.get("https://jsonplaceholder.typicode.com/posts/#{id}")
      post = JSON.parse(res.to_s, symbolize_names: true)

      { title: post[:title], user_id: post[:userId], body: post[:body] }
    end

    register 'fetch_user' do |user_id:, **_attrs|
      res = HTTP.get("https://jsonplaceholder.typicode.com/users/#{user_id}")
      user = JSON.parse(res.to_s, symbolize_names: true)

      { user: { name: user[:name], email: user[:email] } }
    end
  end
end

With the implementation separated, we then have a very easy to read workflow file:

# ./blog_post_enhancer/workflow.rb
module BlogPostEnhancer
  class Workflow
    extend Attr::Gather::Workflow

    container Implementation

    task :fetch_post do |t|
      t.depends_on = []
    end

    task :fetch_user do |t|
      t.depends_on = [:fetch_post]
    end
  end
end

What are you thoughts on this? Could it be easier? Should be provide shortcuts? Open to any ideas here :)

ianks / attr-gather Goto Github PK

attr-gather's Introduction

about:me

about:facts

about:skills

about:opensource

about:misc