Purpose

A good issue template guides the contributors to share the right details at the start of a thread. Therefore, we reduce the amount of time spent by maintainers and other contributors on gathering information to analyze the issue.

Details

A nice standard for code issues can be seen in the MaterialUI repository. We can adapt it to use on this project.

The MaterialUI issue template follows a standard similar to this:

• Issue description
• Check if there's a similar issue
• Check if the issue is present on current release
• Current behavior
• Expected behavior
• Table with the environment used by the creator of the issue

Describe the bug 🐛

AdWords Transformation AdMetricSync is failing due to the error below.

Failed. Took "00:00:29.1411271".
System.OverflowException: Value was either too large or too small for an Int32.
   at System.Number.ParseInt32(ReadOnlySpan`1 s, NumberStyles style, NumberFormatInfo info)
   at Jobs.Transformation.AdWords.AdMetricSync.<>c__DisplayClass4_3.<ListAdMetrics>b__5(SourceAdMetric a) in /home/ubuntu/BuildAgent/work/65d81077067c10a5/src/Jobs.Transformation/Google/AdWordsJobs.cs:line 233

• I have searched the issues of this repository and believe that this is not a duplicate.

To Reproduce
Steps to reproduce the behavior 😯:

1. Go to src/ConsoleApp
2. Run dotnet run -- jobs -j Jobs.Transformation.AdWords.AdMetricSync
3. See error

Expected behavior 🤔

In my opinion, this error can be happening due to a problem during fetching data or fetching a wrong type(expected int but it is a string).

Your environment:

• OS: Ubuntu 18.04
• Browser chrome

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

Jellyfish is a data lake that gathers statistical data from different social media. Adding the Instagram statistical data to the data lake will aggregate more value to this project.

Examples 🌈

Motivation 🔦

Instagram is one of the most popular social media right now, with more than 1 billion users using it every month. Having the ability to fetch data from Instagram and storing it on the data lake will aggregate a lot more value to this project.

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

During improvement on Facebook Fetcher, we implemented a function that first fetch the list of videos id and then the metrics for each id. On this issue, we would like to extend this functionality for other entities as posts, ads, media, etc...

How it should work

• The system will request the id list of each entity from Facebook Graph API
• For each id, it will fetch the metrics

Examples 🌈

Request list of videos id

• https://graph.facebook.com/v3.3/{page_id}/videos?fields=id,created_time,updated_time&access_token={access_token}

Graph API reference: https://developers.facebook.com/docs/graph-api/

Motivation 🔦

In the past, we were having the following problem on videos Fetcher.

{
   "error" : {
         "code": 1 ,
         "message": "Please reduce the amount of data you're asking for, then retry your request"
    }
}

Splitting the Fetcher process into two-stage and fetching each item individually, reduced the size of the request and solved the videos problem.

For this reason, we want to apply the same solution for other entities.

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

We should enable an automated Continuous Integration (CI) server to run the tests and other tasks to improve the project's development when we push new code. This will provide more confidence to the contributors developing with the code, if the commit passes all the tests executed by CI, the code is good to go.

How it should work

• Developers check out code into their private workspaces
• When done, commit the changes to the repository
• The CI server monitors the repository and checks out changes when they occur
• The CI server builds the system and runs unit and integration tests
• The CI server informs the team of the successful build
• If the build or tests fail, the CI server alerts the team
• The team fixes the issue at the earliest opportunity
• Continue to continually integrate and test throughout the project

Examples 🌈

We should consider using the Github actions to enable a CI server

Motivation 🔦

Continuous integration (CI) is a software practice that requires frequently committing code to a shared repository. Committing code more often detects errors sooner and reduces the amount of code a developer needs to debug when finding the source of an error. Frequent code updates also make it easier to merge changes from different members of a software development team.

By having an automated CI server that checks for new code commits in a repository, the code will be continuously built and tested to make sure that the commit doesn't introduce errors. The tests can include code linters (which check style formatting), security checks, code coverage, functional tests, and other custom checks. This is great for developers, who can spend more time writing code and less time debugging errors or resolving merge conflicts.

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

Since the frontend will be removed from this project the frontend tests will always fail.

To Reproduce
Steps to reproduce the behavior 😯:

1. Go to GitHub Actions
2. Click on any new Event
3. See the error on npm steps

Expected behavior 🤔

All the CI tests pass.

** How to solve it? **
The suggestion to solve the problem is to remove all CI steps for frontend tests.

Your environment:

• OS: Ubutntu 18.04

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

Right now, all the credentials related files - from Facebook, YouTube, Adwords, etc - are stored without any distinction in a folder called credentials inside the console app folder. This makes hard to identify which credential file belongs to which service. To solve this, we should separate the credential files by folder, for instance, a folder called Facebook inside the credentials folder should have all the credentials files related with Facebook.

Examples 🌈

The file client_secret.json belongs to YouTube then it should be in a folder called credentials/YouTube

Motivation 🔦

We should make easier to identify what are the roles of each file in the system.

Purpose

We should implement a code of conduct to foster an open and welcoming environment to everybody contributing and using jellyfish.

Details

A good code of conduct that we could adopt is The contributor covenant project. This a code of conduct adopted by open source projects like React, Git, and Linux (you can see more companies here).

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

When the user doesn't have set up the database properly and tries to run any Job, he receives the following unhandled exception instead of an informative message about the error.

To Reproduce

Having the credentials setup but without any database set up properly:

1. Go to src/ConsoleApp
2. Execute dotnet run jobs -t Fetcher
3. See error

Expected behavior 🤔
Instead of an unhandled exception, receive an informative message about the cause of the error.

Screenshots

Unhandled Exception: Npgsql.PostgresException: 28P01: password authentication failed for user "fee"
   at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:line 1003
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:line 1032
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.AuthenticateMD5(String username, Byte[] salt, Boolean async) in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.Auth.cs:line 132
   at Npgsql.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async) in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.Auth.cs:line 38
   at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\npgsql\src\Npgsql\NpgsqlConnector.cs:line 425
   at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken) in C:\projects\npgsql\src\Npgsql\ConnectorPool.cs:line 246
   at Npgsql.NpgsqlConnection.<>c__DisplayClass32_0.<<Open>g__OpenLong|0>d.MoveNext() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 300
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnection.Open() in C:\projects\npgsql\src\Npgsql\NpgsqlConnection.cs:line 153
   at ConsoleApp.Commands.ExportData.QueryFacebook() in /home/user/jellyfish/src/ConsoleApp/Commands/ExportData.cs:line 26
   at ConsoleApp.Program.<>c.<Main>b__0_19() in /home/user/jellyfish/src/ConsoleApp/Program.cs:line 202
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at ConsoleApp.Program.Main(String[] args) in /home/user/jellyfish/src/ConsoleApp/Program.cs:line 209

Your environment

• OS: [Ubuntu]
• Version [19]

Additional context

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

The Mailgun class and other configuration files (like database configuration) make references to FEE information, like email and domain. I think this should be replaced for 'jellyfish' or another generic term.

To Reproduce 😯

In a code editor

1. Go to src/Common/MailgunEmailService.cs
2. See the information there

Search for FEE in all files.

Expected behavior 🤔

Replace this with generic information like 'jellyfish'.

Screenshots

Your environment

• OS: [Ubuntu]
• Version [19]

Additional context

Describe the bug 🐛

Analyzing the system we figure out that views on the Facebook page do not look the same as the views on our Databases. As we can see in the images below, the number of views on Facebook Page is 2.2k while on the Analytics Platform it is 0.

• I have searched the issues of this repository and believe that this is not a duplicate.

To Reproduce

Steps to reproduce the behavior 😯:

1. Go to the Analytics Platform database
2. Search for views related to a videos id (e.g. 384165212202012)
3. Go to the Facebook Page related to this video. (link to the video)
4. See the discrepancy

Expected behavior 🤔

In my opinion, this problem can happen due to a problem during Fetcher:

• failing during fetch data
• failing during saving on database

or it can be a problem related to Transformation:

• failing during transform data
• a bad relationship between videos and metrics

Screenshots

• Facebook Page
  
• Analytics Platform
  

Your environment:

• OS: Ubuntu 18.04
• Browser chrome

Additional context

Videos' daily metrics are coming from the posts. This problem also can be related to the relationship between a post and video id.

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

Facebook Tests are broken and to fix these tests we need new mockup data.

To Reproduce
Steps to reproduce the behavior 😯:

1. On terminal navigate to where you have a clone of this repository
2. Then enter the following folder with cd /src/Tests
3. Execute the tests with the command dotnet tests
4. See error

Expected behavior 🤔
Facebook tests should be passing.

Screenshots

Your environment:

• OS: [Linux]

Additional context

Describe the bug 🐛

On the system we have multiple Jobs running in parallel, because of that we need a Jobs dependency to avoid that a Job that will use the result of another run first.

Let's say that Job A depends on Jobs B and C. We'd like that:

• A runs after B and C
• A runs if any of B and C succeeds.

Currently, if any of the Jobs fails, B or C, then it will lock the dependency and A will not be executed.

A consequence of the current policy is that a broken Facebook Fetcher will lock updates to the YouTube data.

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

The README file is outdated, it needs to be refactored to help users and contributors to get started with the project.

Examples 🌈

We can improve our README by adding the following:

• A better history explaining the project/product
• A summary
• Link for .NET SDK installation
• Link for PostgresSQL installation and also explain that pgadmin4 can help the develop experience
• Better explanation about the credentials, what are they? how to use them? etc..
• Why is this project using docker? better explanation about it.
• Add more badges like license and others (More info in https://shields.io/)
• Add the image from the project running on the introduction to make the readme prettier

Motivation 🔦

The README is like the face of our project. A project with nice README and screenshots will get better the attention of users since it’s a direct way to explain why this project matters, and why people should use and contribute to the project.

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

If you try to run all the Fetchers with any missing credential, you don't execute any Fetcher at all. In order to run this, you have to have all the credentials supported by Andromeda, even the social media that you don't want to get data.

To Reproduce 😯

On terminal

1. Go to 'Andromeda.ConsoleApp'
2. Execute 'dotnet run fetch'
3. See error

Expected behavior 🤔

Execute the Fetchers that the user has the credentials set up. If the user doesn't have any credentials, show a message explaining why the program failed (lack of credentials).

Screenshots

Your environment

• OS: [Ubuntu]
• Version [18.04]

Additional context

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

Jellyfish source code comes from year-ap, an analytics platform developed by Brick Abode and powered by FEE. In this issue, we will split Jellyfish from year-ap to make possible be used on different projects.

Motivation 🔦

This platform Fetch information from multiple Data Source (e.g. Facebook, Youtube) then join this information into an analytics platform. Our plan is to split Jellyfish from the year-ap platform to be able to use it on different platforms.

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

This feature request intends to improve the log's message when Jobs starts. The idea is to print what credentials were found and also what credentials weren't found. After, a new log printing what Jobs will start to fetch.

Motivation 🔦

The motivation for this feature request is to the user understands better what it's being fetched or not from APIs in a better format.

Describe the bug 🐛

Facebook Transformation AdSync is failing due to the error below.

Failed. Took "00:01:13.5936414".
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> Npgsql.PostgresException: 23503: insert or update on table "SourceAds" violates foreign key constraint "FK_SourceAds_SourceVideos_VideoId"
   at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()

• I have searched the issues of this repository and believe that this is not a duplicate.

To Reproduce
Steps to reproduce the behavior 😯:

1. Go to src/ConsoleApp
2. Run dotnet run -- jobs -j Jobs.Transformation.Facebook.AdSync
3. See error

Expected behavior 🤔
In my opinion, this error is happening because there are Ads on Facebook.Ads that are referencing videos that don't exist on Facebook.Videos.

Your environment:

• OS: Ubuntu 18.04
• Browser: chrome

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

Subscribers Gained and Subscribers Lost are metrics related to the number of subscribers that a YouTube Channel Gained/Lost from a specific video.

Currently, we are Fetching this information from YouTube Analytics API, but we are not using it.

References

• YouTube Analytics API
• Metrics:
  • Subscribers Gained
  • Subscribers Lost

Examples 🌈

Motivation 🔦

Adding the metrics Subscribers Gained and Subscribers Lost will make the application database more complete.

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

A nice improvement for the project is download via CLI(command line) the database data to a csv file.

Features:

• Export as csv
• Filter by Facebook, Youtube or Adwords
• Define dates (e.g. since x to y)
• Define channel (Considering that we have multiples channels)

Examples 🌈

run the command on terminal dotnet run export Facebook

Motivation 🔦

With this will be possible to see and download your own data and use it as needed.

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

If the user tries to run Andromeda into the server and credentials are missing the error below will happen.

To Reproduce 😯

1. Go to andromeda
2. Run dotnet Andromeda.ConsoleApp.dll fetcher -s YouTube
3. See error

Expected behavior 🤔

Screenshots

• Error message

Unhandled Exception: System.AggregateException: One or more errors occurred. (Failed to launch browser with "https://accounts.google.com/o/oauth2/v2/auth?access_type=offline&response_type=code&client_id={app-id}.apps.googleusercontent.com&redirect_uri=http%3A%2F%2F127.0.0.1%3A41331%2Fauthorize%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyt-analytics-monetary.readonly" for authorization. See inner exception for details.) ---> System.NotSupportedException: Failed to launch browser with "https://accounts.google.com/o/oauth2/v2/auth?access_type=offline&response_type=code&client_id={app-id}.apps.googleusercontent.com&redirect_uri=http%3A%2F%2F127.0.0.1%3A41331%2Fauthorize%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyt-analytics-monetary.readonly" for authorization. See inner exception for details. ---> System.ComponentModel.Win32Exception: No such file or directory
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Google.Apis.Auth.OAuth2.LocalServerCodeReceiver.OpenBrowser(String url)
   at Google.Apis.Auth.OAuth2.LocalServerCodeReceiver.ReceiveCodeAsync(AuthorizationCodeRequestUrl url, CancellationToken taskCancellationToken)
   --- End of inner exception stack trace ---
   at Google.Apis.Auth.OAuth2.LocalServerCodeReceiver.ReceiveCodeAsync(AuthorizationCodeRequestUrl url, CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.AuthorizationCodeInstalledApp.AuthorizeAsync(String userId, CancellationToken taskCancellationToken)
   at Google.Apis.Auth.OAuth2.GoogleWebAuthorizationBroker.AuthorizeAsync(Initializer initializer, IEnumerable`1 scopes, String user, CancellationToken taskCancellationToken, IDataStore dataStore, ICodeReceiver codeReceiver)
   at Google.Apis.Auth.OAuth2.GoogleWebAuthorizationBroker.AuthorizeAsync(ClientSecrets clientSecrets, IEnumerable`1 scopes, String user, CancellationToken taskCancellationToken, IDataStore dataStore, ICodeReceiver codeReceiver)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Jobs.Fetcher.YouTube.YouTubeFetchers.GetUserCredential(String clientSecretFileName, String dataStoreFolder) in /app/andromeda/Jobs.Fetcher.YouTube/YouTubeFetchers.cs:line 73
   at Jobs.Fetcher.YouTube.YouTubeFetchers.GetServicesCredential(String clientSecretFileName, String dataStoreFolder) in /app/andromeda/Jobs.Fetcher.YouTube/YouTubeFetchers.cs:line 62
   at Jobs.Fetcher.YouTube.YouTubeFetchers.GetJobs(JobType type, JobScope scope, IEnumerable`1 names, JobConfiguration jobConfiguration) in /app/andromeda/Jobs.Fetcher.YouTube/YouTubeFetchers.cs:line 29
   at Andromeda.Commands.RunJobsCommand.CreateJobList(JobType jobType, JobScope jobScope, IEnumerable`1 jobNames, JobConfiguration configuration) in /app/andromeda/Andromeda.Commands/RunJobsCommand.cs:line 74
   at Andromeda.Commands.RunJobsCommand.RunJobs(JobType jobType, JobScope jobScope, IEnumerable`1 jobNames, JobConfiguration configuration, Boolean debug) in /app/andromeda/Andromeda.Commands/RunJobsCommand.cs:line 92
   at Andromeda.ConsoleApp.Program.<>c__DisplayClass0_1.<Main>b__7() in /app/andromeda/Andromeda.ConsoleApp/Program.cs:line 74
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Andromeda.ConsoleApp.Program.Main(String[] args) in /app/andromeda/Andromeda.ConsoleApp/Program.cs:line 181
./run.sh: line 57:    31 Aborted                 (core dumped) dotnet Andromeda.ConsoleApp.dll fetcher

Your environment

• AWS instance t2.small

Additional context

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

When the user doesn't have the Youtube/Facebook credentials and tries to run any Job, he receives the following unhandled exception instead of an informative message about the error.

To Reproduce
Without having the credentials setup:

1. Go to src/ConsoleApp
2. Execute dotnet run jobs -t Fetcher
3. See error

Expected behavior 🤔
Instead of an unhandled exception, receive an informative message about the cause of the error.

Screenshots

Unhandled Exception: System.IO.FileNotFoundException: Could not find file '/Users/user/github/jellyfish/src/ConsoleApp/credentials/client_secret.json'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
   at Jobs.Fetcher.YouTube.YouTubeFetchers.GetUserCredential(String clientSecretFileName, String dataStoreFolder) in /Users/user/github/jellyfish/src/Jobs.Fetcher.YouTube/YouTubeFetchers.cs:line 49
   at Jobs.Fetcher.YouTube.YouTubeFetchers.GetJobs(JobType type, JobScope scope, IEnumerable`1 names, JobConfiguration jobConfiguration) in /Users/user/github/jellyfish/src/Jobs.Fetcher.YouTube/YouTubeFetchers.cs:line 26
   at ConsoleApp.Commands.RunJobsCommand.CreateJobList(JobType jobType, JobScope jobScope, IEnumerable`1 jobNames, JobConfiguration configuration) in /Users/user/github/jellyfish/src/ConsoleApp/Commands/RunJobsCommand.cs:line 74
   at ConsoleApp.Commands.RunJobsCommand.RunJobs(JobType jobType, JobScope jobScope, IEnumerable`1 jobNames, JobConfiguration configuration, Boolean debug) in /Users/user/github/jellyfish/src/ConsoleApp/Commands/RunJobsCommand.cs:line 92
   at ConsoleApp.Program.<>c__DisplayClass0_1.<Main>b__8() in /Users/user/github/jellyfish/src/ConsoleApp/Program.cs:line 85
   at Microsoft.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at ConsoleApp.Program.Main(String[] args) in /Users/user/github/jellyfish/src/ConsoleApp/Program.cs:line 194

Your environment:

• OS: [Ubuntu]
• Version [19]

Purpose

A contribution guideline helps developers to start with the project, by describing step by step how to contribute to this project. A good example that we can follow is the MaterialUI how to contribute file.

Details

The document must introduce the code of conduct(#1) and the issues template (#2) on it, with other important information that a new developer needs to know/follow to contribute properly with Jellyfish).

• I have searched the issues of this repository and believe that this is not a duplicate.

Describe the bug 🐛

Verifications are always needed on the project, with this on the file MigrateCommand.cs when the program calls the functions and after them the project is just printing success so a verification for exceptions on this functions is requested.

To Reproduce 😯

1. Go to andromeda/Andromeda.ConsoleApp/Commands/MigrateCommand.cs
2. Scroll down to lines 31 and 39
3. See that functions MigrateDataLake() and MigrateFacebook() doesn't handle exceptions

Expected behavior 🤔

We expect that database functions are prepared to handle exceptions.

Your environment

• OS: Ubuntu 18.04

Describe the bug 🐛

Our code was developed to use the Uncrustify (c# linter) v0.67. However, we are not being able to install the CI (using the ci.sh file). The only version that we were able to install was 0.66 and it didn't with our code.

• I have searched the issues of this repository and believe that this is not a duplicate.

To Reproduce
Steps to reproduce the behavior 😯:

1. Go to the file ci.sh
2. add sudo apt install uncrustify
3. commit and push on your fork
4. See the error on GitHub actions

Expected behavior 🤔

Parse all the c# files

Your environment:

• OS: ubuntu 18.04