therosettafoundation / solas-match-backend Goto Github PK

https://github.com/fuCtor/QAMQP

See the SOLAS-Match issue list for details

It seems that we should change the "From" email address to [email protected]. This will probably give us a somewhat better reputation for Spam.

Also http://www.postfix.org/ADDRESS_VERIFICATION_README.html (only used by some sites e.g. http://www.clustermail.de/fehler.html#11 ) tries to see if the "From" email address exists using a probe message. If this fails it can give a fatal 550-Verification error which will be in /var/log/mail.log

Alan.

Remove any code that checks for admins seperately. It is now done in the stored proc

Have the message re-queued so we don't lose anything

Add a link to the task that the user is providing feedback on to the feedback emails.

There seems to be a problem with message Acks. I am getting an exception when running the task scorer and the deadline checker every 3 seconds for an extended period. The error message is "Precondition Failed: unknown delivery tag". See the link below for more details on this message:
http://rabbitmq.1065348.n5.nabble.com/reply-code-406-reply-text-PRECONDITION-FAILED-unknown-delivery-tag-td21310.html

Add a list of administrator email address to the conf file. The back end will send any system emails to these email addresses.

Update the UserTaskScoreCalculator to be more generic so that it can take different kinds of timed tasks. Add a new timed task type for checking the deadline of tasks and sending emails to concerned parties.

Change ArchivedTask and Task to work with the new protos. Add support for Projects and archivedProjects.

Add support for the new notification types listed on the SOLAS Match issue list.

Create an use a stored proc for getting tasks that have passed their deadline. Currently the DeadlineChecker is getting all tasks and checking their deadline. This should be done in the DB.

Switch to using smart pointers where appropriate. Qt's QSharedPointer has all the properties we are looking for.

Add a class to the common library that can send emails. Qt does not have native support for sending emails (no convenience classes for it). After looking into it the best route would be to use QTcpSocket and connect to a mail server (pointed to by the conf file.

We just realised that the "Create organisation"button has disappeared from all the non-admin profiles.

For example, @Paulina-Rosetta can see it from her admin account, but not from her volunteer standard account.

The one in the screenshot is a profile of an organisation who is just registered as a volunteer because they can't go any further.

Thanks!
Stefania

Make it count for less than the native language.

CalculateTaskScore.cpp runs way too slow.

I have previously stopped succeeding invocations from proceeding until previous invocations have completed. This resolved any serious issues temporarily. However it was only a workaround.

The simplest approach is to move the addition of the number of days a Task has been unclaimed outside the loop. This will mean that normally Task Scores will not change so the optimisation that checks to see if a database update of the score to the very big UserTaskScores table is needed will nearly always avoid an update (and that update is VERY expensive as currently configured and dominates the time).

Instead of the continually changing of the score, I will add the number of days to the score in the database procedure getUserTopTasks() in the sort clause (ORDER by score).

Alan.

See: #44

@Paulina-Rosetta said...
Re unpublishing, I don't think it's useful to automatically unpublish tasks once the deadline passes. In the vast majority of cases, there is some flexibility with the deadline.

We sometimes unpublish tasks when we don't want them to be publically visible for some reason - even though anyone with the link can still access them when unpublished.

As soon as we become aware that a task is no longer needed, we delete it from the system. Tasks that end up unpublished because of an expired deadline usually end up being updated and published by us manually to make them available again.

@stefania91 Did I miss anything?

Currently We have found two errors in text of email templates:
In Tracked task uploaded template

To download the the latest version of the file 
    <a href="{{TASK_REVIEW}}">here</a>.

should be

To download the the latest version of the file click
    <a href="{{TASK_REVIEW}}">here</a>.

In Banned login

In the future please refrain from violation any of our terms of use. We value your
    participation.

should be

In the future please refrain from violating any of our terms of use. We value your
    participation.

Add a stored procedure for getting all taskids with their associated tagids and store them in a multi map (of task id associated to a list of tag ids). Add a stored proc for getting all user ids and their associated tag ids and store them in a multi map. Implement these in DAO calls that return the multimap. Should also get the secondary languages in this way.

See: TheRosettaFoundation/SOLAS-Match#1268

In fact, this is only a problem for user created badges, I will add a new template for them.

Alan.

Create an init that connects to the host and a send that sends the message. If this fails add a lock around the send.

Some urls have changed and must be updated in the email system.

Currently all emails assume the objects exist in the DB. If this is not the case the daemon will crash. Changes must be made to avoid this.

Strict notifications only contain tasks that match either the user's native language or one of the user's secondary languages.

We should send emails encrypted (using TLS). This is better practice and will stop Google placing a red opened padlock against all emails sent by the background.

Alan

These libraries work in a very similar way. There is a bug in QCTemplate that causes a memory leak. This is not present in CTemplate.

When a new project is created a request should be sent to RabbitMQ for the task's deadlines to be calculated. The deadlines should be relative to the project deadline with different task getting different deadlines based on their task type and their depth in the Graph (shallow nodes get shorter deadlines).

This is to ensure that tasks later in the workflow will get enough time to be completed.

This has been deemed superfluous.

Instead of hard-coding an email address for sender email addresses use the conf option.

The library we use (QxtMail) to send SMTP email does not wrap properly. When a word is copied to a line buffer, it is not emptied and ends up being added again to the following line. This happens (sometimes) when there are explicit CRLF.

Fixed by commit b0596b8

Move the TEMPLATE_DIRECTORY definition to the Common definitions header file. It is defined in the EmailDefinitions file and in the Task Score Calculator.

A change in the way templates work between ctemplates and qctemplates introduces a bug where only one line of updates is written to the template.

Create a new plugin that will read from a file and send messages to Rabbit MQ at the times specified in the file. For example the UserTaskScore calculator should be run every hour and should be triggered from the Plugin Scheduler. The file it reads from should be XML based with fields for start time, interval (hrs, mins, secs), exchange, topic, message. The message should be the name of a protobuf generated class which encapsulates the task to be carried out. Once triggered the message is populated by the Plugin Scheduler and sent to the Rabbit MQ for processing.

Paulina: I’ve received a query/complaint about the ‘Registered Badge’ email from an org member.
... However, it would be helpful if they received a welcome email instead.

An error has occurred in the SOLAS-Match worker Daemon.
Failed to generate UserTaskStream email: Unable to find relevant data in the Database. Searched for User with ID 657 and their notification data.

This (or similar) seems to happen about 4 times a year...

[16/11/2016 01:10:00:353] Debug: MySQLHandler: Failed to connect to database: "Lost connection to MySQL server at 'reading initial communication packet', system error: 9 QMYSQL: Unable to connect"
[16/11/2016 01:10:00:354] Debug: MySQLHandler::call - Initialization failed

Alan.

A volunteer has asked if the date could be added to the task information on the automatic volunteer reference email. Would this be a lot of work?

I guess either the date of when the task was claimed or submitted would be ok to include. He asked for the submission date, but whatever is easier...

We should add the List-Unsubscribe email Header so that users have an easy option to unsubscribe, rather than possibly marking us as Spam.

Alan.

The user can elect to use their real name on their private profile. If selected, reference emails should display their real name instead of their display name.

Using a new lib it should be possible to switch to using a push model instead of a pull model which should reduce network load. Must find a C++ library that can expose functions as web services. Part of #4

Add a new plugin to handle DB writes. Create a listener for each type (e.g. Task, User, Project, etc.). Off load other events to the appropriate plugin (e.g. EmailPlugin for emails).

Switch model classes to generated protobuf classes.

TheRosettaFoundation/SOLAS-Match#1264
turns out to be an issue with the backend.

When the deadlines are being calculated, the backend reads in a Task, updates the Deadlines and writes out the Task again. However, due to a bug, the "published" bit is always read in as True so it is written out again as always True.

Alan.

Wherever QSharedPointers are used a type_def should be created (in the common Definitions.h file) for that type.

e.g. QSharedPointer -> QSPTask

As more Users and Tasks are added, CalculateTaskScore is beginning to take longer (close to an hour now on average). So it looks as if the factor of 60 improvement is not enough (see #37 ).

Alan.

Change all references to SOLAS Match to a conf option reference. Also make sure the subject lines are accurate.

The link to the tag listed under each task is broken. It uses the tag label instead of the ID.

When a Tasks table record is updated, a side effect is that strings in the record are re-written. However they are corrupted because they are treated as ASCII when they are wrapped.

Additionally backslashes and single quotes are not escaped causing potential database fatal errors (or worse case an SQL injection security issue).

(there is a somewhat related issue with handling double quotes causing backslashes to be multiplied which has been fixed by commit f9b79eeffb9b6ae3ba1060a528d10df36ca14ce2 in the front end.)

Alan.

@aquilax and others,

In looking at another issue, I noticed that we do not clean JavaScript from outgoing emails.
JavaScript could appear in an email because of injected script tags in User names, Project Names, Task Names, Badges, Feedback and some others (see full list below).
We do not clean this (as we do for display on the Trommons website to avoid XSS).

It seems most email readers (e.g. GMail) suppress this from display in any case, so it might not be a big issue.

I am currently not working on changing this. Any suggestions on whether we should worry about this?

Alan.

List...
BannedLoginGenerator.cpp(39): dict.SetValue("USERNAME", user->display_name());
BannedLoginGenerator.cpp(60): commentDict->SetValue("COMMENT", banData->comment());
ClaimedTaskSourceUpdatedGenerator.cpp(36): dict.SetValue("USERNAME", user->display_name());
ClaimedTaskSourceUpdatedGenerator.cpp(38): dict.SetValue("TASK_TITLE", task->title());
ClaimedTaskUploadedGenerator.cpp(49): dict.SetValue("USERNAME", user->display_name());
ClaimedTaskUploadedGenerator.cpp(51): dict.SetValue("TASK_TITLE", task->title());
NewImageUploadedEmailGenerator.cpp(48): dict.SetValue("PROJECT_TITLE", project->title());
OrgCreated_OrgEmail.cpp(33): dict.SetValue("USERNAME", admin->display_name());
OrgCreated_SiteAdmin.cpp(32): dict.SetValue("USERNAME", admin->display_name());
OrgDeadlinePassedEmailGenerator.cpp(39): dict.SetValue("USERNAME", user->display_name());
OrgDeadlinePassedEmailGenerator.cpp(43): dict.SetValue("TASK_TITLE", task->title());
OrgFeedbackGenerator.cpp(55): dict.SetValue("USERNAME", claimant->display_name());
OrgFeedbackGenerator.cpp(59): dict.SetValue("TASKTITLE", task->title());
OrgFeedbackGenerator.cpp(60): dict.SetValue("FEEDBACK", feedback.toStdString());
OrgMembershipAcceptedGenerator.cpp(59): dict.SetValue("USERNAME", user->display_name());
OrgMembershipRefusedEmailGenerator.cpp(37): dict.SetValue("USERNAME", user->display_name());
OrgTaskRevokedGenerator.cpp(32): dict.SetValue("USERNAME", user->display_name());
OrgTaskRevokedGenerator.cpp(33): dict.SetValue("TASK_TITLE", task->title());
OrgTaskRevokedGenerator.cpp(38): dict.SetValue("CLAIMANT_NAME", claimant->display_name());
PasswordResetEmailGenerator.cpp(45): dict.SetValue("USERNAME", user->display_name());
ProjectCreatedGenerator.cpp(44): dict.SetValue("PROJECT_TITLE", project->title());
ProjectImageApprovedEmailGenerator.cpp(54): dict.SetValue("PROJECT_TITLE", project->title());
ProjectImageApprovedEmailGenerator.cpp(58): dict.SetValue("USERNAME", user->display_name());
ProjectImageDisapprovedEmailGenerator.cpp(54): dict.SetValue("PROJECT_TITLE", project->title());
ProjectImageDisapprovedEmailGenerator.cpp(58): dict.SetValue("USERNAME", user->display_name());
ProjectImageRemovedEmailGenerator.cpp(48): dict.SetValue("PROJECT_TITLE", project->title());
TaskArchivedEmailGenerator.cpp(52): dict.SetValue("USERNAME", user->display_name());
TaskArchivedEmailGenerator.cpp(56): dict.SetValue("TASK_TITLE", task->title());
TaskClaimedEmailGenerator.cpp(40): dict.SetValue("USERNAME", user->display_name());
TaskClaimedEmailGenerator.cpp(44): dict.SetValue("TASK_TITLE", task->title());
TaskClaimedEmailGenerator.cpp(45): dict.SetValue("TRANSLATOR_NAME", translator->display_name());
TrackedTaskUploadedEmailGenerator.cpp(54): dict.SetValue("USERNAME", user->display_name());
TrackedTaskUploadedEmailGenerator.cpp(55): dict.SetValue("TRANSLATOR_NAME", translator->display_name());
TrackedTaskUploadedEmailGenerator.cpp(56): dict.SetValue("TASK_TITLE", task->title());
UserBadgeAwardedGenerator.cpp(76): dict.SetValue("USERNAME", user->display_name());
UserBadgeAwardedGenerator.cpp(77): dict.SetValue("BADGE_NAME", badge_title.toStdString());
UserClaimedTaskEarlyWarningDeadlinePassedEmailGenerator.cpp(38): dict.SetValue("USERNAME", user->display_name());
UserClaimedTaskEarlyWarningDeadlinePassedEmailGenerator.cpp(42): dict.SetValue("TASK_TITLE", task->title());
UserClaimedTaskLateWarningDeadlinePassedEmailGenerator.cpp(38): dict.SetValue("USERNAME", user->display_name());
UserClaimedTaskLateWarningDeadlinePassedEmailGenerator.cpp(42): dict.SetValue("TASK_TITLE", task->title());
UserFeedbackGenerator.cpp(57): dict.SetValue("USERNAME", user->display_name());
UserFeedbackGenerator.cpp(58): dict.SetValue("FEEDBACK", feedback.toStdString());
UserFeedbackGenerator.cpp(62): dict.SetValue("TASK_TITLE", task->title());
UserFeedbackGenerator.cpp(64): dict.SetValue("CLAIMANT_NAME", claimant->display_name());
UserReferenceEmailGenerator.cpp(39): dict.SetValue("DISPLAY_NAME", displayName);
UserReferenceEmailGenerator.cpp(54): taskSect->SetValue("TASK_TITLE", task->title());
UserReferenceEmailGenerator.cpp(103): taskSect->SetValue("TASK_TITLE", task->title());
UserTaskClaimEmailGenerator.cpp(37): dict.SetValue("USERNAME", user->display_name());
UserTaskClaimEmailGenerator.cpp(46): dict.SetValue("TASK_TITLE", task->title());
UserTaskDeadlineEmailGenerator.cpp(38): dict.SetValue("USERNAME", user->display_name());
UserTaskDeadlineEmailGenerator.cpp(42): dict.SetValue("TASK_TITLE", task->title());
UserTaskRevokedGenerator.cpp(30): dict.SetValue("USERNAME", user->display_name());
UserTaskRevokedGenerator.cpp(32): dict.SetValue("TASK_TITLE", task->title());
UserTaskStreamEmailGenerator.cpp(68): taskSect->SetValue("TASK_TITLE", task->title());
UserTaskStreamEmailGenerator.cpp(102): tagsList->SetValue("TAG_DETAILS", tagDetails.toStdString());
UserTaskStreamEmailGenerator.cpp(103): tagsList->SetValue("TAG_LABEL", tag->label());
UserTaskStreamEmailGenerator.cpp(120): taskSect->SetValue("PROJECT_TITLE", project->title());