cividesk / com.cividesk.email.sparkpost Goto Github PK

When upgrading from 4.7 to 5.x this extension throws 500 errors when using the current released version.

Recent PRs have fixed this. Any chance the info.xml file can be updated and a release be done?

Thanks for this extension.

We tried installing it for a client and found that the 'Send test and save' returns an error 'Cannot install webhook'.

From the documentation, it looks like the api path for webhooks has moved from /webhooks to /relay-webhooks — but if I update this in the code, I also find that the the API starts returning with errors about malformed payloads.

Hi Nicholas,

We ran a stress test with 30K emails and found that in CiviMail reports most of these are recorded as bounced. However under sparkpost reports these are sent and the emails are also received. Can you take a look at this. I have attached some screenshots, let me know if you need any other information.

Tested in: Civi 4.6.14

-Madura

CiviCRM 4.7.3, WordPress 4.4.2

Setup SparkPost API key. Added via Outbound Email (SparkPost). Saved settings and received these errors:
SparkPost errors
Could not install webhook (Sparkpost error(s): POST to webhook target URL failed with response code 302).
SparkPost errors
Could not install/refresh webhook.

Test mail came through, however scheduled mailings do not.

On my site (Joomla! 3.8.10, PHP 5.6.36, this extension 1.1, CiviCRM 5.3.2), this extension causes the following issues:

• clicking on the System Status link on any Civicrm back-end page doesn't work, it creates a blank page
• when loading Civicrm for the first time, site often creates Error 500
• messages in log: PHP Fatal error: Call to undefined function civicrmVersion() in
  /blabla/www/media/civicrm/ext/com.cividesk.email.sparkpost/sparkpost.php on
  line 145

All these problems go away immediately the extension is disabled.

Upgrading to v1.2 is impossible due to the problems reported in Issue 62.

Hi, Nicolas,

I installed com.cividesk.email.sparkpost and spent more time than I'd like to admit resolving the error messages regarding the domain status, webhook installation, etc. It turns out that the key that is automatically generated when you create an account has a very limited set of permissions assigned to it. I'd like to add a step to the installation instructions to let folks know they should check/set these, but I'm not sure which permissions are relevant. For my test case I took a brute force approach and granted the API key everything, but I didn't read enough of the code to know whether that's necessary/appropriate. Please advise, and I'll make a PR for updating the doc.

Thanks!

We just sent out our first large mailing, about 4000 mails through Sparkpost. On the CiviCRM side, we show a very small number of bounces < 10. On the Sparkpost account, it shows on the order of about 400 bounces, of which many (>260) are hard bounces, most are "recipient doesn't exist." Why is there a difference? These emails went out over a 24 hr period in various logical batches, starting around 3pm Sat, Jan 21.

Do we just need to wait for the two to match? Also, this is a pretty new live CiviCRM instance. Most of our testing over a 3-month period was done using a test domain name. We then redirected to the final domain name about a month ago. Note that we also set SPF and DKIM for the new domain and ensured we were all green on the Sparkpost side. However, we didn't touch the CiviDesk Sparkpost configuration, etc. I'm wondering if the webhook isn't right. But if that were hosed, why would I get any bounces at all. Except they all seem to be of type "syntax" so perhaps that is not even being sent back by Sparkpost I had noticed a reply to another issue that said your webhook is setup on installation. Does that mean we need to reinstall?

Thanks for any help/advice

Hi there,

In https://github.com/cividesk/com.cividesk.email.sparkpost/blob/master/sparkpost.php#L130, the comment says "Don't know why it does not autoload ...".

AFAIK (though I haven't tested this) this is because Mail_Sparkpost is not within CRM_. I think CRM_Mail_Sparkpost would work.

Not a biggie. Just wanted to report since there was a comment there.

Can this extension do batch mail delivery to Sparkpost? From a quick look at the code, it didn't seem like it, but I could have missed something.

Use case: I am sending a mailing to 500,000 recipients, and I'd like the whole sending and delivery process to be as fast as possible, maybe an hour or so.

The solution, if you're using SMTP, is to open up a pool of concurrent connections, sending to perhaps thousands of recipients with each connection before closing it. Or if you're using HTTPS, send as many messages as possible with each request, perhaps using a pool of parallel requests.

I'm including this for reference only. The default PHP version for some hosting providers may not meet the minimum requirement for the extensio and you may get the following error:

Fatal error: Can't use function return value in write context in /home/customer/websites/customer.org/htdocs/sites/default/files/civicrm/extensions/com.cividesk.email.sparkpost/CRM/Admin/Form/Setting/Sparkpost.php on line 128

The extension's README does indicate that PHP 5.4 or higher is a prerequisite, but particularly for CiviCRM 4.4 users, where PHP 5.3 was acceptable (there's a deprecation warning for CiviCRM 4.6) this might be missed. I resolved this on Siteground with the following line in the .htaccess file
AddHandler application/x-httpd-php56 .php .php5 .php4 .php3

The following error occurs when clicking "Save and Send Test Email."

Fatal error: Call to undefined method CRM_Core_TemporaryErrorScope::ignoreException() in /path/to/site/sites/default/ext/com.cividesk.email.sparkpost/CRM/Admin/Form/Setting/Sparkpost.php on line 183.

Is this extension still maintained? Could one reasonably implement it on 5.x CiviCRM or does it need development resources? Deciding between this, Mandrill, and CiviSMTP.

I installed your extension yesterday, which resolved most of the issues I was having with the other one I tried. CiviMail is working correctly :-)

My problem now is that I don't seem to be able to send out transactional email from CiviCRM, such as when people subscribe to an email list or create a membership. Is there a scheduled job that needs to be created or activated that would allow these emails to send via SparkPost?

How do you process bounces, delays, etc?

The "other" sparkpost extension for CiviCRM & Joomla recommends a webhook (callback from Sparkpost) for failures, bounces, etc.

Just switching to your extension.

Hi,
Sparkpost use a eu.sparkpost.com for European users.
Replacing all URLs https://sparkpost.com by https://.eu.sparkpost.com allow the plugin to works fine.

It will be nice to add an option to define the URL (sparkpost.com or eu.sparkpost.com

Regards.

Hi,
Thanks for this great extension and I use it on a few sites I manage. All my clients are getting this final notice email. I assume this will not affect our implementation?
Thanks

Upon making a test donation, I get the following error message:

An error occurred when CiviCRM attempted to send an email (via SMTP). If you received this error after submitting on online contribution or event registration - the transaction was completed, but we were unable to send the email receipt.

The mail library returned the following error message:
Sparkpost error: HTTP return code 200. Check https://support.sparkpost.com/customer/en/portal/articles/2140916-extended-error-codes for interpretation.

This is probably related to a problem in your Outbound Email Settings (Administer CiviCRM » System Settings » Outbound Email), OR the FROM email address specifically configured for your contribution page or event. Possible causes are:

Your Sendmail path is incorrect.
Your Sendmail argument is incorrect.
The FROM Email Address configured for this feature may not be a valid sender based on your email service provider rules.
Check this page for more information.

However, the mail is delivered.

CiviCRM version: 4.4.21.
CMS: Drupal 7

I haven't dug into this yet. Just logging it so I don't lose track.

Related to #4, I think I'd have gotten to the bottom of my configuration problem faster with more nuanced handling of the response. For 403/Forbidden, the extension reports:

Check that the API key is valid.

... which suggests to me that the API key doesn't exist on the server (i.e., analogous to a user name/password mismatch). What it actually means, though, is that your valid API key doesn't have permission to do the requested action.

If I'm not mistaken -- and I could be, because I didn't find good documentation around the return codes for this API -- 401/Unauthorized means the API key is bogus. Did you find better documentation for return codes than this?

Do you agree with my assessment? If so, I'll gladly make a PR to add some clarity to the feedback messages.

Bounce processing was not working, when we setup a webhook on Sparkpost it started working.

Should the instructions include setting up a webhook? Or should bounce processing work without it?

Hi,

When sending a test email from the sparkpost configuration page (...civicrm/admin/setting/sparkpost) get this warning.

Issue is on both Civi 4.6 (tested on 4.6.14) and 4.7 (tested on 4.7.4). However test mailing is successful. Not sure if its a major issue, but thought ill send it to you to have a look.

-Madura

I got this info in an email from Sparkpost -- do these upcoming changes affect this extension?

https://www.sparkpost.com/blog/upcoming-api-transmission-endpoint-changes

As of CiviCRM 4.7, settings groups are toast. Each setting must have a unique name. This extension's apiKey setting name is just asking for name collisions. How about sparkpost_apiKey instead?

Problem:
A CiviCRM instance is running on a private network with a Proxy that is set up (using CiviProxy) to handle necessary communication with the outside world. Callbacks that are automatically configured by the extension will not reach the CiviCRM instance.

Proposed solution:
Add an option to the sparkpost admin settings form that allows the callback URL to be specified so that when the form is submitted and the setting is create/refreshed it is set to an appropriate URL on the proxy.

Happy to submit a PR along these lines if this sounds like an acceptable approach to you.

The scheduled jobs log for the Sparkpost Fetch Bounces job always ends with this: "Full message:
Finished execution of SparkPost Fetch Bounces with result: Failure, Error message: API (SparkPost, fetchbounces) does not exist (join the API team and implement it!)" Despite this, the job does complete successfully and reports bounce data that look reasonable. Sounds like a missing feature. Is this normal or do I have a setting wrong somewhere?

Just installed CiviDesk Sparkpost extension in CiviCRM.

When pasting API Key and clicking Save & Send Test Email, the plugin gives error:
Notice: Undefined property: stdClass::$description in CRM_Sparkpost::call() (line 149 of /var/www/html/crm/sites/default/files/civicrm/ext/com.cividesk.email.sparkpost/CRM/Sparkpost.php).

SparkPost error
Could not install webhook (Sparkpost error: HTTP return code 400, Sparkpost error code UNABLE_TO_VERIFY_LEAF_SIGNATURE (POST to webhook target URL failed with error code UNABLE_TO_VERIFY_LEAF_SIGNATURE: ). Check https://support.sparkpost.com/customer/en/portal/articles/2140916-extended-error-codes for interpretation.).

SparkPost integration v1.1, CiviCRM 4.7.14., Drupal 7.52

All SparkPost requests following the last message allowed by the hourly/daily/monthly limit are marked as bounces of type Syntax, which has hold_threshold of 3. As a result, organizations which exceed their limits just three times run the risk of marking swaths of their audience as do-not-mail.

The extension should create a new bounce type to account for this case. Perhaps we'd call the bounce type "API refused" to account for other non-bounce failures that CiviCRM would improperly treat as a bounce. Addressing this problem will also require creating at least one new bounce pattern. (The bounce pattern is used to match error messages to bounce types.) Unfortunately, the way the core code is written, we do not have the option of giving a bounce type an unlimited threshold, so the extension will have to set a high threshold (say 9,999).

Ideally, the extension would not treat this scenario as a bounce event, but I think this is a heavier lift, and I haven't done the homework yet to see what it would take for an extension to register a new mailing event with core. Or perhaps this shouldn't be the extension's job -- perhaps core needs a new mailing event so that other SMTP-related extensions don't have to implement the same event.

My logs are filling up with messages like:

May 14 00:00:05 [info] Deprecated: Group='SparkPost Extension Settings'. Name should be provided.

I haven't been able to determine what name this refers to. Can you point me in the right direction?

The extension installed cleanly. I set up a test civimail to one record, to myself. A draft email was delivered, however when it was submitted for delivery the outgoing mail is shown as bounced in the report, and it did not reach my account.

So I have a few questions: this system has been using CiviSMTP and the outbound mail setting are still set for that. Do I need to delete those settings? These are on the System Settings --> Outbound Email Settings page (currently it has the SMTP server set to civismtp.uas.coop) and the CiviMail --> Mail Accounts settings page (currently with bounce processing for CiviSMTP).

Thanks!

I have the latest code installed from master, and when I try to send a test email, I get the following error:

SparkPost error
Could not install webhook (Sparkpost error: HTTP return code 400, Sparkpost error code 400 (POST to webhook target URL failed with non-2xx response code 404: ). Check https://support.sparkpost.com/customer/en/portal/articles/2140916-extended-error-codes for interpretation.).

I checked Sparkpost and no webhook has been installed. Any idea why it's failing to install? My API key is correct, as above the error message it validated that my sending domain was configured properly.

I also tried creating a webhook manually at https://mysite.com/?page=CiviCRM&q=civicrm/sparkpost/callback which works in Sparkpost, but am still getting the same error in CiviCRM.

Thanks for any assistance!

Error below is displayed on the confirmation page when a user with an on-hold bounce address submits an event registration on CiviCRM 5.4.0 using Sparkpost ext 1.3. Registration is recorded and an email does get sent out.
Sparkpost error Sparkpost error: HTTP return code 404, Sparkpost error code (Resource could not be found: ). Check https://support.sparkpost.com/customer/en/portal/articles/2140916-extended-error-codes for interpretation.

The log contains this:

[message] => invalid data format/type
[description] => Problems parsing request as json
[code] => 1300

The problem presented during a 30K send. I was unable to reproduce it in a sandbox using the same message and a much smaller list of recipients. Code 1300 is described as "invalid data format/type/length."

Current theory is that the request fails because the payload is too big. Or it could be that the content gets truncated because of its length, resulting in invalid JSON.

SparkPost recommends transmissions with 10K or fewer recipients. (I didn't get a rationale for why.) SparkPost also pointed me to a thread in which another developer encountered a problem because "transmission-level substitution data must be less than 100,000 bytes," or 100 kilobytes. We're not using substitution data, but we might want a sanity check on the size of the request before we make it.

I'll do a little more digging and see if I can validate/invalidate the working hypothesis.

I can't assign issues in this project, but you're welcome to assign this to me... unless you've seen something like this before, in which case I'd be happy to pair up.

Does this extension support transactional emails? If so, in what way are bounces, deliveries and click-throughs on transactional emails processed, and reported to the user through the browser? FWIW, the Mandrill Transactional Email extension created a pseudo-CiviMail entry to allow recording of responses into the CiviCRM Mailing Tables and to re-use the existing mailing reports.

Hello,

We updated the Sparkpost extension to version 1.1 and tested if the bounces are correct in CiviCRM. Unfortunately there are no bounces counted in Civicrm, but there are lots of bounces in Sparkpost. We have tried with sending only for 2 email addresses, one of them is a wrong email address, and tried sending for 12 addresses, many of them should have been bounced, but none of them are in Civicrm. We have a scheduled job to fetch bounces, we ran it many times. These emails were sent last week on Friday, and waited till Monday to make sure they get bounced in Sparkpost, but they are still not counted in Civi. Do you have any idea what might be wrong?

Thank you!

Best,
Rita Torok

I recently installed the Sparkpost extension and all the mail functionality seems to be working with all the test emails I have sent out via Mailings > New Mailing - mail processed on time, opens, click-thrus, and bounces counted correctly. We are just configuring CiviCRM for the first time and do not have a real Payment Processor in place. We are using the default Test Processor (type Elavon) to test out membership and donation transactions. The transactions go through and are recorded in the database. However, the acknowledgment emails are not working. We get the message:

The SparkPost service is unavailable due to a sending error, and the backup mailer is not enabled or not configured.

Is this due to the Test Processor, or are them some other mail configuration settings that need to be configured. Is this likely a SparkPost error or a CiviCRM configuration error?

We are running CiviCRM 4.7.10 (on WordPress 4.6.1) with CiviDesk SparkPost extension 1.1

Thanks for any help!

Feature request: an email should be removed from the sparkpost suppression list if the on-hold flag is removed from Civi otherwise an email has to be un-holded in two places.

• there's another request against a different extension here: proexchange/com.pesc.sparkpost#1

Just flagging this to see if it affects this extension (via an email from SparkPost):

I wanted to let you know that we are making a breaking change to an existing API and give you the opportunity to make necessary changes to any automated processes. Specifically, we are changing the format of the Suppression-List GET results.

• Users who programmatically process the results of the Suppression-List API GET to update their system will be impacted by this change.
• If you don’t use the Suppression-List API, you can disregard this message.

In addition, we have made some significant enhancements to the Suppression-List API.

Currently every time the Sparkpost API setting is saved in CiviCRM, the webhook will be overwrite to point to the site which the save action happened. It makes it difficult to use the same Sparkpost account on a multisite environment.

A solution could be to amend the base_url in the webhook name so the the extension will catch the correct one to overwrite when save action happens.

This commit 85a4a89 breaks sending test emails for me. I didn't test whether it also breaks sending other email but I'm guessing it does.

When pressing "Save and Send Test Email" from: Administer->System Settings->Outbound Email (Sparkpost)

I get the following error:

Sending test email. FROM: XXX@XXX TO: YYY@YYY.
Oops. Your SparkPost settings are incorrect. No test mail has been sent.
An error occurred when CiviCRM attempted to send an email (via SMTP). If you received this error after submitting on online contribution or event registration - the transaction was completed, but we were unable to send the email receipt.

The mail library returned the following error message:
Sparkpost error: HTTP return code 405, Sparkpost error code 1102 (invalid http method: PUT is not supported on /api/v1/transmissions). Check https://support.sparkpost.com/customer/en/portal/articles/2140916-extended-error-codes for interpretation.

This is probably related to a problem in your Outbound Email Settings (Administer CiviCRM » System Settings » Outbound Email), OR the FROM email address specifically configured for your contribution page or event. Possible causes are:

Your Sendmail path is incorrect.
Your Sendmail argument is incorrect.
The FROM Email Address configured for this feature may not be a valid sender based on your email service provider rules.

I recently switched a client (Drupal 7 + CiviCRM 4.7) from using a direct-from-server civimail setup to using sparkpost (with a free account) and this extension.

It's working, but it takes 2hrs to send out about 8000 pieces of mail instead of the 15 minutes it used to take. I've got the outgoing mail cron configured to run every minute with 500 pieces at a time, which was appropriate for the direct-from-server setup, but I wonder if that's why it's now so much slower? Is there a recommended setup? Or is there throttling at sparkpost with the free account?

From CiviCRM 4.7.31 there is a new setting "Allow Mail to be sent from logged in contact's email address" on the Outbound mail settings. A new crm-region has been created to allow overriding the default mailer settings:
https://github.com/civicrm/civicrm-core/blob/master/templates/CRM/Admin/Form/Setting/Smtp.tpl#L36

Hi, N,

Not sure if this is an extension issue or a core issue. I think the core form in question is quite old and not very well-maintained...

In any case, you may recall that putting the {action.forward} token in an email generates a link through which users may forward messages to their friends. From what I can see in sparkpost_log, the generated email puts the subscriber's email in the FROM header and the domain address in the REPLY-TO header.

As you know, SparkPost will not allow sending from off-domain addresses. To work with SparkPost, the message must come from an organization address. It's unclear to me whether the problem is that the core form is designed to send from the constituent address or that the extension is mishandling params. Any insight?

Hi,
Even if à message is still delayed by SparkPost, it is shown in Civimail as successful.

Is it should be useful to report the different SparkPost status.

Olivier.

Testing on civicrm 5.8.1 shows that the extension has stopped working. The admin menu link to Outbound Mail Settings goes to /civicrm/setting/smtp instead of /civicrm/setting/sparkpost. The ConfigAndLog/sparkpost_log file contains many errors of the form:

====` ERROR in CRM_Sparkpost::call() ====
stdClass Object
(
    [errors] => Array
        (
            [0] => stdClass Object
                (
                    [message] => Forbidden.
                )

        )

)

This is using the latest extension version 1.3 on wordpress 4.9.9.

SparkPost sent out a notice yesterday indicating changes in their suppression list API. I'm fairly certain this does not impact this extension or the integration with CiviCRM, but was curious if you've reviewed that notice to confirm. I can supply it if you did not receive it.

I have followed the instructions under SETUP STEPS, but as sone as I install and enable the extension (v1.3) in my WordPress site running CiviCRM 4.6.12, it no longer sends email through CiviMail, however it still send transactional email from WordPress.

When enabled, the scheduled job for SparkPost Action: Fetchbounces completes with a result of: Success (a:0:{})

CiviMail email send fine when the extension is NOT enabled.

Any suggestions?

Cheers

I've just come across a problem with the extension where by it gives this error message when sparkpost is configured with subaccounts:

"Domain example.com is not created and not verified in Sparkpost. Please follow instructions at https://app.sparkpost.com/account/sending-domains."

This is despite the domain being verified in sparkpost (sparkpost support also confirmed it was correctly set up)

When I created a new API key I specified that it should be linked to a specific sub account (I want to do this for reporting purposes) as we have other email systems using sparkpost, with dedicated IPs and want to keep them completely separate.

As I suspected that this might be causing the problem I tried creating a new API key which wasn't linked to a sub account, tested it in civi and this got rid of the error message above.

So it seems that the extension is not able to 'see' a sending domain which is shared across all subaccounts where the API key is generated and associated with a specific sub account.

It would be good to know if this last assumption is correct, and if so it might be considered as a future improvement for the extension.

Thanks
Peter

Your extension has been a great addition to my site. Thankyou!

However I can't reconcile bounce reports on Sparkpost with CiviMail. It appears that certain bounces aren't reflecting correctly. On the mailing below Sparkpost reflects 27 bounces (20 blocks, 3 hard, 2 admin, 2 soft) whereas CiviMail only reflects 2 bounces.

Can you help? I'm using CiviCRM 4.7.4 on Drupal 6, with PHP5.4

After enabling the plugin and putting in the API information, which has all access granted to it, this is the error that I receive.
SparkPost error
Could not list webhooks (Sparkpost error: Permission denied. Check that the API key is authorized for request https://api.sparkpost.com/api/v1/webhooks.).

Let me know what information you need from me to help troubleshoot and patch the issue.

I haven't double checked but I was checking settings in conjunction with civicrm/civicrm-core#13048 & I'm pretty sure you will have an enotice on yours as $_settings is not defined - I would recommend changing from extends CRM_Admin_Form_Setting to just CRM_Core_Form - you are not really using functionality from the settings form and it's not really supported as an integration point - I think it could well be completely deleted in the future

Note I recommend the same action for com.cividesk.normalize & com.cividesk.helptab

Fatal error: Unsupported operand types in /path/to/civicrm/CRM/Core/Session.php on line 475

Looks like bad params to CRM_Core_Session::setStatus();