Code Monkey home page Code Monkey logo

data-upstream-service-201902's Introduction

README

setup

For development

brew install postresql
brew services start postresql

Add a database for development
psql postgres
CREATE DATABASE "dataupstream_development";
CREATE ROLE dataupstream LOGIN PASSWORD 'dataupstream';
GRANT ALL PRIVILEGES ON DATABASE dataupstream_development TO dataupstream;

By default, the RAILS_ENV is set to 'development'
export RAILS_ENV=development
bin/rake db:migrate
bin/rails s

For production

Below are the generic steps to setup production:

  1. Create a secret key unique to your machine (or server)
    RAILS_ENV=production rake secret
  2. Create and configure your database settings (similar to the step for development database above). Currently, the settings are hard-coded in database.yml, you may configure it with some local env variables.
  3. You may create an environment file to be sourced locally (of course, this can be set once in the server). Below is mine for reference.
#!/bin/bash

export RAILS_ENV=production
export SECRET_KEY_BASE=<secret key from step 1>
export PORT=<some port>
export CORS_ORIGIN=<some address>
export POSTGRES_USER=<...>
export POSTGRES_PASSWORD=<...>
  1. Source this file
    source <your file>
  2. Migrate your production database
    rails db:migrate
  3. Set the eager_load in production.rb to false, else you will get uninitialized constant error.
    This is one small nuisance, refer to here for details: rails/spring#519
    config.eager_load = false
  4. Start the server
    rails s
  5. You may access the server from <your machine address:port>
    If it's local, then http://localhost:<port>

see the doc in .... [todo]

  • (j)Ruby version

See the .ruby-version file in repo.

  • System dependencies

  • software developers

testing

  • testing

./bin/bundle install

./bin/bundle exec rspec

interface with jupyter

pip install jupyter cd APP_DIR/python-notebooks jupyter

get the notebooks from: ...... [todo]

[todo]

  • Configuration

  • Database creation

  • Database initialization

  • How to run the test suite

  • Services (job queues, cache servers, search engines, etc.)

  • Deployment instructions

data-upstream-service-201902's People

Contributors

alvinvoo avatar

Watchers

 avatar  avatar

data-upstream-service-201902's Issues

Session module methods cannot be found after changes of namespace to Api::V1


name: Session module methods cannot be found
about: After changes of namespace to Api::VI


Describe the bug
After changes of namespace to Api::VI, user can be created and login but any action that requires user token to be verified will throw error:

"status": 500,
"error": "Internal Server Error",
"exception": "#<NoMethodError: undefined method `require_read_write_user_access_token' for #Api::V1::Users::StreamsController:0xe42ded4>",

To Reproduce
Steps to reproduce the behavior:

  1. Using Postman, with base url : localhost:3000/api/v1
  2. Create a user locally with POST /users.json
  3. Or login with existing user with POST /users/access_token.json
  4. Tries to GET stream /users/streams.json or GET profile /profile
  5. See error

Screenshots (click to enlarge)
image

Device detail
(please complete the following information):

  • OS:
    Xbuntu 18.04
  • Java:
    java version "1.8.0_201"
    Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
    Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)

Additional context
Suspect is due to changes in namespace but Sessions module was not updated to reflect the correct namespace, hence methods could not be found.

Modification on web executor

Modification on request body to external cloud URL:

  1. Add current_device's device_id
  2. Substitute secret with 'secret' value from saved http_headers
  3. Remove 'event'

Modification on RestClient

  1. RestClient should be able to GET, POST or UPDATE based on saved method
  2. RestClient should be able to support port

After changes is done, need to test with a running cloud service

image

create stream not working

Started POST "/users/streams.json" for 0:0:0:0:0:0:0:1 at 2019-02-02 19:26:08 +0800
Error occurred while parsing request parameters.
Contents:

name=test&key_rotation_enabled=false

ActionDispatch::Http::Parameters::ParseError (unexpected token at 'name=test&key_rotation_enabled=false'):

actionpack (5.2.2) lib/action_dispatch/http/parameters.rb:117:in parse_formatted_parameters' actionpack (5.2.2) lib/action_dispatch/http/request.rb:381:in block in POST'
org/jruby/RubyHash.java:1263:in fetch' rack (2.0.6) lib/rack/request.rb:59:in fetch_header'
actionpack (5.2.2) lib/action_dispatch/http/request.rb:380:in POST' actionpack (5.2.2) lib/action_dispatch/http/parameters.rb:55:in parameters'
actionpack (5.2.2) lib/action_controller/metal/params_wrapper.rb:290:in _wrapper_enabled?' actionpack (5.2.2) lib/action_controller/metal/params_wrapper.rb:244:in process_action'
activerecord (5.2.2) lib/active_record/railties/controller_runtime.rb:24:in process_action' actionpack (5.2.2) lib/abstract_controller/base.rb:134:in process'
actionpack (5.2.2) lib/action_controller/metal.rb:191:in dispatch' actionpack (5.2.2) lib/action_controller/metal.rb:252:in dispatch'
actionpack (5.2.2) lib/action_dispatch/routing/route_set.rb:52:in dispatch' actionpack (5.2.2) lib/action_dispatch/routing/route_set.rb:34:in serve'
actionpack (5.2.2) lib/action_dispatch/journey/router.rb:52:in block in serve' org/jruby/RubyArray.java:1792:in each'
actionpack (5.2.2) lib/action_dispatch/journey/router.rb:35:in serve' actionpack (5.2.2) lib/action_dispatch/routing/route_set.rb:840:in call'
warden (1.2.8) lib/warden/manager.rb:36:in block in call' org/jruby/RubyKernel.java:1181:in catch'
warden (1.2.8) lib/warden/manager.rb:34:in call' rack (2.0.6) lib/rack/etag.rb:25:in call'
rack (2.0.6) lib/rack/conditional_get.rb:38:in call' rack (2.0.6) lib/rack/head.rb:12:in call'
activerecord (5.2.2) lib/active_record/migration.rb:559:in call' actionpack (5.2.2) lib/action_dispatch/middleware/callbacks.rb:28:in block in call'
activesupport (5.2.2) lib/active_support/callbacks.rb:98:in run_callbacks' actionpack (5.2.2) lib/action_dispatch/middleware/callbacks.rb:26:in call'
actionpack (5.2.2) lib/action_dispatch/middleware/executor.rb:14:in call' actionpack (5.2.2) lib/action_dispatch/middleware/debug_exceptions.rb:61:in call'
actionpack (5.2.2) lib/action_dispatch/middleware/show_exceptions.rb:33:in call' railties (5.2.2) lib/rails/rack/logger.rb:38:in call_app'
railties (5.2.2) lib/rails/rack/logger.rb:26:in block in call' activesupport (5.2.2) lib/active_support/tagged_logging.rb:71:in block in tagged'
activesupport (5.2.2) lib/active_support/tagged_logging.rb:28:in tagged' activesupport (5.2.2) lib/active_support/tagged_logging.rb:71:in tagged'
railties (5.2.2) lib/rails/rack/logger.rb:26:in `call'

Unauthorized error for /log_data.json endpoint


name: Accessing /log_data.json gives 401
about: This is an existing issue even in https://alpha.data-upstream.ch


Describe the bug
The frontend is actually sending the device_token as X-Access-Token and got the error response, but when I tried with the user's X-Access-Token, I got the same error response as well

{
"errors": [
"X-Access-Token is invalid."
]
}

Screenshots (click to enlarge)
image

Fixing Specs

@yihang I already upgraded the app to Factory-Bot wich is new and replaces Factory-Girl. (it is only actually a renaming).

When I run the spec
./bin/bundle exec rspec spec/controllers/log_data_controller_spec.rb

the following include is missing.
include Api::V1::Concerns::Sessions

Alvin, Yihang, could you take a quick look?

Enchancement on webhook model

Add a last_invoked column
web_executor should update this column whenever RestClient api calls return

Add a return_status column
web_executor should update this to true or false whenever RestClient api calls return 200 or otherwise

Fix profile

I pushed the quick fix to frontend. Unfortunately, the profile now fails. I put up a ticket,

Started GET "/api/v1//profile" for 0:0:0:0:0:0:0:1 at 2019-02-10 16:24:04 +0800
Processing by Api::V1::UsersController#profile as HTML
Completed 500 Internal Server Error in 1140ms (ActiveRecord: 0.0ms)

NoMethodError (undefined method `require_read_write_user_access_token' for #Api::V1::UsersController:0x4ce6ae07):

Chrome blocking due to CORS policy violation


name: Frontend is blocked from receiving request due to CORS policy violation
about: Rails need to specify CORS policy rules


Describe the bug
CORS policy issue with either Rails or Chrome

I see that Alex has installed rack-cors and written the cors.rb following the steps here:

image

Either this doesn't work for Rails 5 or the issue exists in the Angular client side.

Screenshots (click to enlarge)

image

Misc fixes

  • Some files still missing the Api::V1 namespaces
  • export header X-Total-Count in CORS

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.