ipstenu / varnish-http-purge Goto Github PK

Hi there, this plugin really is simple to use for the main use case it was designed for (e.g. varnish running on 127.0.0.1:80), but is kinda non-trivial to configure for other cases.

I had to dig around a bunch to find out a mention of a variable that can be set in wp-config.php.

To make that easier, you could create a panel that is hooked up in the Settings section so that ppl can specify the varnish server's IP and port via the web interface.

Since the first pass of the age check may be an unprimed cache, run it twice to get a valid result.

Some hosters (ours is Gandi.net) offer a varnish based caching solution where you can use Cookies as part of the cached object. For instance, Pantheon and gandi.net Simple Hosting offer this using specially prefixed cookies. This is needed for instance for language cookies like Polylang or WPML use (search for "cookie" in https://polylang.pro/doc/php-constants/ and https://polylang.pro/doc/detect-the-browser-preferred-language/)

Whithout this cookie allowed by varnish, pages could not be cached here.

But this also has the downside, that when you want to purge such a page then

Use case:

We use Polylang and have STYXKEY_pll_language cookie which can be unset (no language set yet), de for German or en for English. So a URL gets cached in possibly 3 different variants.

How caching works for this use case:

1. When caching the page by calling the following (browser does the same), we make sure this page gets cached inside varnish with:

curl -v -s 'https://www.mysite.de/de/shop/' -o /dev/null
curl -v -s 'https://www.mysite.de/de/shop/' -H 'Cookie: STYXKEY_pll_language=de' -o /dev/null
curl -v -s 'https://www.mysite.de/de/shop/' -H 'Cookie: STYXKEY_pll_language=en' -o /dev/null

Now when you want to PURGE a page, you need to call it this way:

 curl -X PURGE https://www.mysite.de/de/shop/ 
 curl -X PURGE https://www.mysite.de/de/shop/ -H 'Cookie: STYXKEY_pll_language=de'
 curl -X PURGE https://www.mysite.de/de/shop/ -H 'Cookie: STYXKEY_pll_language=en'

I asked our hoster Gandi.net if a normal PURGE could not purge all 3 variants, but they said no because that this is the way varnish works the way here, and if I want to PURGE all variants then I'd have to call it this way three times.

Problem: There's no way to do this currently given the available filters by the varnish-http-purge plugin. There's varnish_http_purge_headers which could be used to set the Cookie header, but then it would be always called once and with the same kind. But we would need to have 3 different cookie headers: once none, once with de and once with en.

I thought a night over this and think I would go with this hacky workaround:

1. Set a filter on vhp_purge_urls using our child theme.
2. Within the filter do the purging on ourselves by doing more or less the same using wp_remote_request as the plugin does:

3. Then always return an empty array back to vhp_purge_urls so that the plugin does just nothing.

What do you think of this approach? I am especially wondering if in step 3 it is enough to just return an empty array to stop varnish-http-purge from executing any further.

In the long run, of course, I would be interested if there was a filter that would be possible to use for this. The above would be a shortterm hack/workaround. Hence opening this issue for tracking too.

Would it be stupid to do a varnish_http_purge_events_full on the customize_save_after event ?

When a user finds a conflicting plugin that isn't documented, we should emphasize reporting to the "Known Conflicts" list.

This is a spin-off of #63.

This "bug" applies to simple as well as variable products. This happens whenever an item has a restricted stock / inventory (see https://docs.woocommerce.com/document/managing-products/#section-6 and look for "stock management"), so they you sell physical goods (coffee, T-Shirts) then whenever an item is sold, then the inventory is reduced by that number. This causes to post to be saved.

Not sure why this happens but for every item in a cart the purge functionality (VarnishPurger->purge_post) is called 6 times. So when someone has an order of say 5 products, then it would be called 6 x 5 = 30 times. Each time for the $urls the purge response is done. It's about 20 URLs each time. This makes 30 x 20 = 600 purge requests in each checkout.

On our server the varnish backend is nearby, that is, a response is about 50-100ms. Still this makes 50-100ms x 600 requests = 30000-60000ms, that is 30 to 60 seconds delay just because of the varnish http purge plugin :-(

This is a bit of a weird edge case. If you have a published post that gets reverted back to draft (this would have for pending too), it doesn't get purged from Varnish. You have to go and click on the "Empty All" button.

This issue seems to be related to the timing of the call to the purgePost method. At that point, WordPress updated the post in the database and it's now a draft. get_permalink won't give us a useful link to purge.

I was able to work around the issue using the pre_post_update hook. Here's a proof concept solution below:

/**
 * Purges a post from Varnish when their status changes from a publicly accessible link to one without one.
 *
 * This method must be called before WordPress updates the database because VarnishPurger uses functions
 * that use current database values.
 *
 * @param int   $post_id
 * @param array $post_data
 */
function varnish_purge_non_accessible_post($post_id, $post_data) {
    global $purger;

    if (!$purger instanceof VarnishPurger
        || empty($post_data['post_status'])
        || !in_array(get_post_status($post_id), array('publish', 'private', 'trash'))
        || !in_array($post_data['post_status'], array('pending', 'draft'))
    ) {
        return;
    }

    $purger->purgePost($post_id);
}
add_action('pre_post_update', 'varnish_purge_non_accessible_post', 10, 2);

I didn't do a PR for this because this function doesn't quite fit in the mould of the purgePost and purgeNoID methods. You need both the post_id and post_data to validate the post status before and after the update so that we don't step over what the plugin does already. That said, if this code is adequate, I'm happy to do a PR for it. 😄

Suggestion in the WordPress Forums

I think making it filterable would be better since my goal in this is for a plugin with NO settings at all.

Since acfe0b0 the URL gets the home URL prefixed: $url = $this->varnish_purge->the_home_url() . esc_url( $url ); . That doesn't seem right.

To make it more obvious to debuggers, we should visibly confirm plugin check against blacklist, and include a call to report findings to the blacklist.

This check:

// Multisite - Site admins can purge UNLESS it's a subfolder install and we're on site #1
is_multisite() && !current_user_can('manage_network')

Is checking if its a multisite installation and the current user does not have the manage_network capability - meaning pretty much all users meaning that contributors are able to make purges. I think the check should be something like:

current_user_can('manage_options') ?

<link rel='stylesheet' id='varnish_http_purge-css' href='/wp-content/plugins/varnish-http-purge/style.css?ver=4.5.2' type='text/css' media='all' />

This should load only if the user is logged in.

We are looking into Varnish + this plugin to optimize our caching strategy – especially the invalidation based on Regex.
In addition to the website it self there is a mobile app which heavily uses the WordPress REST API.
The users are able to filter, sort or search using the /wp-json/wp/v2/posts endpoint.
So if a post is created or updated, we need to purge not only the actual permalink and the id-specific endpoint, but also the archive with all possible query parameters and combinations:

Of course, this should happen for every post type that has a rest_base defined and is public.
I tested a regex to play around.
Similarly this applies the taxonomy archives as well (when a term is created or updated).

As far as I understand, this plugin does not clear the post archive (REST-API) but only the single post.
Currently, the regex purging is only used for a full purge.
But it would be a good solution because for example search terms are unpredictable.

One solution is to hook into vhp_purge_urls, and possibly add the required urls too.
But we need to use some regex to match everything.
Maybe we can use the vhp-regex query parameter and assign a value with the regex (this would require possibly breaking changes to the plugin and a customized VCL file).
So that if a PURGE request contains a vhp-regex query parameter (even better use a header for this?) this is used instead of the requests url.

Is this something more people are interested in, or did someone approach this task?

I see some comments that somebody thought about it before :)

This was hell to debug. I discovered it because on my local system, all the PURGE calls were going through as actual GET pageloads, and my macbook was getting overwhelmed loading dozens of pages all at once each time I saved a published post (took over a minute).

TL;DR: If the current web server redirects PURGE requests as 302 then they become GET and load the actual URLs on the site, which makes saving a post extremely slow.

TL;DR2: The HTTP API allows requests to be non-blocking and thus not hold up the pageload, why not use 'blocking' => 0 this for this plugin?

It was slow because all the PURGEs were being done as GET

This part is mostly for information purposes, in case someone else stumbles onto a similar problem, but it also explains my situation and why I think the 'blocking' => 0 solution described below is important.

• We had the plugin active on our local sites and want to keep it that way so that the live and dev environments are as similar as possible.
• We don't have Varnish locally for predictable reasons, so obviously the PURGE requests don't "work" like they should, but we want them to fail silently.
• The plugin sends the PURGE requests for the dozen relevant URLs each time.
• WP ends up loading them as GET, which loads the actual URLs and overwhelms my laptop as a little mini-DOS, making post saving take 1min+

It has taken me literally years to figure out that this is what's happening, which is embarassing but related to how deep the problem lies. I had to dig into the logs and add logging deep inside the HTTP API methods to figure it out. The user experience was just that saving was excruciatingly slow on my local site.

After investigation the following became clear:

• It's because each PURGE request was being redirected from http->https, and the redirected URL was being fetched as GET instead of PURGE
• The reason for this was that the redirect returned by Apache was a 302, and WP has an explicit (underdocumented) internal filter on the HTTP API that converts ANY 302 redirect to GET regardless of it's starting method. You can read about how this happens on a StackExchange post I created about what's going on: Why does WP HTTP API switch the method (POST/PURGE) to GET when redirecting (302)?

Important notes:

• This was on my MAMP setup, where Apache was running things and which seems to be related to the 302 coming up.
• Our server, running Nginx, didn't have a similar behavior because it returned 301 errors in this same situation. So even PURGE requests could redirect successfully.

SO: In terms of this HTTP API confusion with redirection anyone with similar problems has a couple of options:

• Rework your system so the redirects don't happen (in our case, filter the URLs to be HTTPS from the start)
• Rework your system to make sure such redirects are 301, thus not triggering the WP filter in WP_Http->browser_redirect_compatibility()

Bad solution: In purge_url, pass 'redirection' = 0 to wp_remote_request()

i.e.
	$response = wp_remote_request( $purgeme, array(
			'method'  => 'PURGE',
			'headers' => $headers,
			'redirection' => 0,
		) );

This solved my problem with local dev because it means that when my Apache in MAMP gets the purge request and redirects it to HTTPS, it just silently stops.

The problem with this is that if the redirection is a 301, it DOES work! We discovered that our actual live server was relying on this for proper functioning. To me, this implies that this plugin shouldn't disable redirection, because there may be many users relying on the effectiveness of redirected PURGE calls, even if in the grander scheme they aren't reliable (and really, everyone should work to make sure their purges are happening without redirection).

That said, I wanted to point this out because for some cases it could be a good workaround, and it seemed like a good option to me at first.

Better solution? Use 'blocking' => 0 instead

By default all WP HTTP API requests are "blocking" and the whole system waits for the resulting $response object to be generated and returned. Obviously sometimes this is vital, but in the case of these PURGE requests I think the 'blocking' => 0 mode would be a really good fit:

• In a live environment, we would pretty much never want the page saving to hang and wait for slow PURGE requests to process. It's bad UX for it to be slow, could cause race conditions and if the PURGE requests don't go well there still won't be any information about it for the user.
• The plugin doesn't seem to do any material actions on the $response object at the end of purge_url(), it runs the after_purge_url filter but that filter isn't called inside the plugin itself.
• For situations like mine where Varnish wasn't present, it avoids any messiness with the request causing chaos with the performance.

Have you considered 'blocking' => 0 before? Maybe there's a reason not to use it that I haven't thought of?

The main reason I can see is to preserve the current functioning of the filter:

	do_action( 'after_purge_url', $url, $purgeme, $response, $headers );

If the non-blocking mode was used, there wouldn't be a $response object to include in that filter (or, if left as-is, it would always be a blank default response object).

That said, it strikes me that the $response object, though useful here, is something we can probably live without as long as the documentation makes it clear that the $response is always empty because we (logically) use the non-blocking mode for the request.

We can still get the response with the http_api_debug action in WP_Http->request()

do_action( 'http_api_debug', $response, 'response', 'Requests', $r, $url );

So IDK if this is something you'd want to build into the plugin, but IMHO if making it easy to debug the purging is a priority, and otherwise 'blocking' => 0 makes sense to you, then you can easily document how to debug these purges and/or add it as a plugin feature using the action above:

• Add a code example in the help docs for debuging purges that outputs an error_log() during http_api_debug any time the method in $r (request arguments) is PURGE
• Add that code yourself to this plugin, then use a constant (`VHP_DEBUG_PURGES) to enable the logging.

This would work just as well as the existing hook, and now that I think of it, you could keep after_purge_url intact if you wanted, by creating your own hook on http_api_debug and setting up the variables so that after_purge_url is backwards-compatible.

So after all that, the question is: Why not use 'blocking' => 0 as the default mode for the plugin?

On my end, despite all this digging, it seems it's still pretty slow on local, even if I set it to 'blocking' => 0, because MAMP still loads every page and it takes 18 seconds reliably before I get back to the refreshed post editor. If I want to keep this plugin active on my dev environment I think i'll need to set up something that quickly and silently returns 200 for PURGE requests to mimic Varnish.

Hi there,

Seems like there's still some merge conflict goodies left here on Github:
https://github.com/Ipstenu/varnish-http-purge/blob/master/varnish-http-purge.php#L425-L431

To streamline the debugging process, we should optionally run grep commands against plugin and theme directories for session and cookie functions.

These could be implemented with a --grep flag or similar.

Examples to follow include:

grep -RE '(PHPSESSID | session_start | start_session)' ./wp-content/ |cut -f1 -d ':'|sort|cut -f3,4 -d '/'|uniq
grep -RE '($cookie|setCookie)' ./wp-content/ |cut -f1 -d ':'|sort|cut -f3,4 -d '/'|uniq

By ignoring common bad directories, we can improve the performance of these checks.

See if the PHP can be run async, to prevent unneeded slowdowns (and indeed, make the save a faster thing!)

• https://stackoverflow.com/questions/26826780/asynchronous-php-code-in-wp-plugin
• https://techcrunch.com/2014/07/31/wp-async-task-our-new-open-source-library/

Any ideas why do we need strtotime here?

We have multiple varnish backend servers and using the plugin to send purge request to all of the varnish servers.
The issue is when i add 1 ip it works fawlessly but when added multiple IP's with the convention suggested it stops sending request to any backend.

I am getting Undefined variable: footer_text in varnish-http-purge/settings.php on line 507 with the latest version.

See
https://screenshots.firefox.com/AxnKwM4NG8LpV41q/fr.test

One request from Ellen earlier this week is that the grep output be formatted in a more digestible manner.

Specifically:

1. Remove the unnecessary part of the filepath:

2. Ensure the output for the matched line doesn't exceed some character count. Ellen had one example where our regex matched a minified JavaScript file, and the output overtook the whole screen.

This morning, I looked into a couple of options:

1. Update the $pattern to match up to 10 characters before and after the match ('.{0,10}(PHPSESSID|session_start|start_session|$cookie|setCookie).{0,10}')). However, doing so makes grep execution seemingly 5x slower, which I don't think is a reasonable compromise.
2. Use ack when available instead of grep. However, ack doesn't seem to be any faster with the before and after character matching.

Additional options we could explore include:

3. Limiting grep to PHP files, which mitigates the scenario where a bunch of minified JavaScript clogs the screen. The original match pattern seems specific to PHP.
4. Using proc_open to launch the process, and modifying output with PHP.

Dear list,

We are running [SSL/443] -> hitch -> [socket] + [port 80] -> varnish -> [HTTP/8080] -> apache.

Caching appears to be working, however, we always get this error message in the plugin:

Cache Service Varnish caching service is running but is unable to cache your site.

How can this be resolved.

Since Varnish doesn't (shouldn't unless you did something weird) cache query param URLs, it's reasonably safe to strip them and thus minimize the number of URLs having to be flushed per page.

Hi there!

The latest update of the plugin changed the "Purge Varnish" button to instead "Empty Cache" which I agree in some sense is more user friendly.

But in reality it's not really "true".

Sometimes when you're using a persistent cache this has to be flushed aswell to "flush the cache". Unless VHP started doing this is it not really an "Empty Cache". In reality it's really flushing Varnish only (or emptying the frontend cache).

Changing this to "Empty Varnish Cache" would be more precise.

From https://wordpress.org/support/topic/varnish-4?replies=3

Tested it now with Varnish 4.0.1 and got it working with this in the default.vcl:

acl invalidators {
"127.0.0.1"
"localhost";
"<servers_hostname>";
"<servers_ip>";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ invalidators) {
return (synth(405, "Not allowed"));
}
return (purge);
}
}

EDIT: I couldn't get this to work before i added my servers public IP in the authorized client list. Edited the post to fit the results in case it might help someone else.

Hi @Ipstenu,

I'm Thijs, I'm the technical evangelist at Varnish Software, and I'm looking to build a VCL that works for WordPress and that is kept up-to-date by us.

Your sample VCL files on https://github.com/Ipstenu/varnish-http-purge/wiki/Sample-VCLs look like a solid foundation, but I'd like to introduce some optimizations that would heavily reduce the line count.

But before I present potential changes, I'd like to hear your opinion on WordPress cookies.

I have found the following regex pattern in the main VCL file:

wordpress_(?!test_)[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+|woocommerce_cart_hash|woocommerce_items_in_cart|wp_woocommerce_session_[a-zA-Z0-9]+|wordpress_logged_in_|comment_author|PHPSESSID

I was wondering if this pattern captures all relevant cookies in WordPress. Instead of removing individual cookies, I'd like to suggest a more aggressive approach: removing the Cookie header entirely if we get passed the if-statement that checks this pattern

This is the if-statement that is currently used:

	if (
		req.http.Authorization ||
		req.http.Authenticate ||
		req.http.X-Logged-In == "True" ||
		req.http.Cookie ~ "wordpress_(?!test_)[a-zA-Z0-9_]+|wp-postpass|comment_author_[a-zA-Z0-9_]+|woocommerce_cart_hash|woocommerce_items_in_cart|wp_woocommerce_session_[a-zA-Z0-9]+|wordpress_logged_in_|comment_author|PHPSESSID"
	) {
		return(pass);
	}

If the request gets passed this conditional, I'm assuming cookies are no longer relevant to the page. As a consequence I would replace the regsuball() calls that replace individual cookies with unset req.http.Cookie;.

Please advise on the relevance of cookies when the request gets passed the cookie check in the VCL file.

Thanks
Thijs

Not purging the wp-api urls for retrieving posts

Since the last update the "Empty Cache" item has a green background color #46B450. But it actually means nothing as it's hardcoded in https://github.com/Ipstenu/varnish-http-purge/blob/b35efddb6924af5f03b00fcf6f8f4f5c91f65520/style.css. Note that it was previously #F56E28.

It's quite distracting. What's the reason for loading an additional stylesheet just to make the item green?

We should correctly check for Age header by making the request twice and comparing the second with the first.

Don't mind me. Network hairpin was screwing me over.

Hi Mika,

I am testing wordpress with varnish 4, purging works but not purge all.
I see an PURGE /.* in varnishncsa log (status 200) but pages are still old after this request has been send.

I´ve read that varnish 4 does not support regexp purging and BAN should be used instead. Might that be the problem?

In old issues I found a link to https://github.com/aaemnnosttv/varnish-vcl-collection/blob/5830257b956be56224f6ac676a7bd58b04ab7593/lib/purge.vcl but do I really need to include all these purge functions to make it work with varnish 4?

Thanks for any pointers!
Steve

Hi,
In sitediagnose I get the following error message about the Proxy Cache Purge Plugin.

'Cache Service: Site: https://example.com
We were unable find a caching service active for this domain. This may occur if you use a proxy service (such as CloudFlare or Sucuri) or if you're in the middle of a DNS move.'

Situation: I am not using Cloudflare, not in a DNS move and the IP address of the Varnish Cache is inserted.

Question: is this a bug or is this expected behavior?

Thanks for feedback.

Problem being they're SOMETIMES an array and sometimes not...

It'd be nice to spit out a list with the string/values split out into a nice table.

Have Varnish installed and working well. Recently upgraded infrastructure to have multiple machines/domains. Would like to request an option to put in more than one domain name.

Something like www1.xyz.com, www2.xyz.com www3.xyz.com…

When a post is saved using the Gutenberg editor, the editor makes two separate requests to save all the data: one request to save metaboxes and one request to save the post itself. Both of these requests dispatch a save_post event. As a result, all of the purges happen twice. This is causing performance issues: most notably, it worsens the slow saving of posts documented in #56.

There are a few workarounds in the comments at WordPress/gutenberg#12903, but I'm not sure which is best.

Using V4, purging with a regex does not work, don't know if this was previously available?

I notice BAN is also available in 3.0 so it may be better to switch to this method for backwards compatibility, the below works for v4 as it accepts regex.

https://github.com/joeldavuk/varnish-http-purge/commit/379c11cd56aa6d1e24f300e9e11a9597d3a7d920

Hey, this one I'm not sure how to interpret, but we're getting a regular stream of notices on our non-subdomain multisite install. It dates back to the dark ages of WPMU when path-based multisite wasn't considered insane.

Here's the notice:

Notice: Use of undefined constant BLOG_ID_CURRENT_SITE - assumed 'BLOG_ID_CURRENT_SITE' in /SITE/wp-content/plugins/varnish-http-purge/varnish-http-purge.php on line 92

Here's the code:

if (...
            ( is_multisite() && !current_user_can('manage_network') && ( SUBDOMAIN_INSTALL || ( !SUBDOMAIN_INSTALL && ( BLOG_ID_CURRENT_SITE != $blog_id ) ) ) )
...)

The code assumes that based on the other conditions (!SUBDOMAIN_INSTALL), BLOG_ID_CURRENT_SITE will already be set, but in our case it clearly isn't. I tried looking up BLOG_ID_CURRENT_SITE but couldn't find anything useful.

My inclination is to submit a PR where we validate BLOG_ID_CURRENT_SITE before using it, but I don't know if that would significantly change the behavior of your plugin. Here's what I'd think is right:

$blog_id_current_site = null;
if (defined('BLOG_ID_CURRENT_SITE')) :
    $blog_id_current_site = BLOG_ID_CURRENT_SITE;
endif;

Above the if statement, then using $blog_id_current_site in the test. Does that make sense to you?

https://wordpress.org/support/topic/is-there-any-way-to-hook-into-this-plugin/

Theory: Add a hook to the purge action so people can run their own commands following a successful page purge. This would need to be very carefully done to try and limit loops.

Looking at the $purgeUrls param of the 'vhp_purge_urls' filter I see these URLs to be purged for a WooCommerce variable product:

https://www.mysite.de/de/shop/my-variable-product/?attribute_pa_weight=500g
https://www.mysite.de/de/shop/my-variable-product/?attribute_pa_weight=500gamp/
...
https://www.mysite.de/de/shop/my-variable-product/?attribute_pa_weight=500g/feed/
...

Not sure about the amp version (we are not using it), but the normal URLs should be this instead as far as I understand:

https://www.mysite.de/de/shop/my-variable-product/
...
https://www.mysite.de/de/shop/my-variable-product/feed/
...

That is the query parameter should omitted here.

Sure I could add the correct URLs myself using the 'vhp_purge_urls' filter but I think it's still a bug that not the correct URLs.

Hi Mika,

I actively use Varnish and have a performance-oriented blog at www.getpagespeed.com with some articles like this touching on Varnish, NGINX, etc.

Your plugin is actually dependency for my small plugin I have that warms up the cache after purge. See Cacheability.

Fork created at https://github.com/dvershinin/varnish-http-purge

I'm having a weird issue with the new requests WP_Http API. I usually run this:

$response = wp_remote_request($purgeme, array('method' => 'PURGE', 'headers' => array( 'host' => $p['host'], 'X-Purge-Method' => $varnish_x_purgemethod ) ) );

But it no longer passes through a method of PURGE, so the whole thing fails. I can't see how to properly use a custom method with the new system. It defaults to a GET now which ... not what I wanted.

GET /
GET /feed/
GET /2016/07/dsafdsaf/
GET /feed/rdf/
GET /feed/rss/
GET /feed/
GET /feed/
GET /feed/atom/
GET /comments/feed/
GET /2016/07/dsafdsaf/feed/

Those should all be PURGE and not GET.

Similarly, when I do a full flush, I get this: GET /.*

I've reached out to @rmccue and @ocean90 via WP Slack.

We should show all header output when generating the report (close to curl -IL) because this level of verbosity is helpful to debuggers.

hi,
me and some other people built a client library to send purge and ban requests to varnish and also to nginx. this might be useful to reduce the code of this plugin to just the wordpress part, and use a well-tested library to talk with the varnish side.
https://github.com/FriendsOfSymfony/FOSHttpCache

if you want to discuss this, i am happy to explain further or clarify questions. we are short before a 1.0 release, it would still be time to adjust something if you miss flexibiltiy somewhere.

Make sure each URL is only purged ONCE per run.