Faster Index Updates about metacpan-api HOT 6 CLOSED

This would be the feed to use: http://cpan.cpantesters.org/modules/01modules.mtime.rss

from metacpan-api.

File::Rsync::Mirror::Recent and Linux::Inotify2 could be useful for this as well.

from metacpan-api.

We could write a simple IRC bot that sits in irc.freenode.net/perl and waits for GumbyPAN to announce new releases (or we use https://github.com/bingos/gumbypan directly). When there is a upload, we fire up File::Rsync::Mirror::Recent and get the latest and greatest releases.

I think this is the most lightweight and OS independent approach.

My idea is that we add a "queue" index to elasticsearch. Every time there is an upload to pause, the irc bot adds the name of the release to the queue. A worker checks the queue periodically and indexes the tarball, if the file exists in the local cpan mirror. Otherwise wait until the rsync process got the files.

This should give us a nice ~5 min delay from upload to index.

from metacpan-api.

Or we do this instead of the irc bot:

use AnyEvent::FriendFeed::Realtime;
my $done = AnyEvent->condvar;

my $client = AnyEvent::FriendFeed::Realtime->new(
    request    => "/feed/cpan",
    on_entry   => sub {
        my $entry = shift;
        $entry->{body} =~ /href="(.*?)"/;
        my $file = $1;
        # push to queue
    },
    on_error   => sub {
        warn "ERROR: $_[0]";
        $done->send;
    },
);

$done->recv;

from metacpan-api.

I implemented the above in my fork:
https://github.com/monken/cpan-api/blob/master/lib/MetaCPAN/Script/Watcher.pm

It forks a new perl process that indexes the release. This is pretty much as fast as you can update the index.

See http://api.netcubed.de:5000/release/_search?q=*&sort=date:desc&pretty=true&fields=name,author,date

from metacpan-api.

I think if we used the new faster mirroring setup and File::ChangeNotify, it would make local.metaCPAN trivial to implement. Your own mirror is indexed as the files appear and the same goes for your own DarkPAN. It would be a pretty cool setup to have your own custom packages show up in the index shortly after copying them over to the correct directory.

from metacpan-api.