blekhmanlab / rxivist Goto Github PK

Indicate last week, last month, last 3 months, etc

Just use the most recent monthly results

When a paper's traffic is recorded, it grabs all the months, including the current (unfinished) month. When we go back to add new months to that list, there's a section in spider.py to make sure we don't re-record any papers:

# make a list that excludes the records we already know about
      to_record = []
      for i, record in enumerate(stats):
        print(record)
        month = record[0]
        year = record[1]
        if year in done.keys() and month in done[year]:
          print("Found, not recording")
        else:
          to_record.append(record)

HOWEVER, this loop should also throw out the stats for the most recent month in the most recent year, because that would have been recorded during whatever that month was, and we have a chance to get a more up-to-date number now that we're revisiting it.

I put a crazy hacky rc.local thing in "api/startup.sh" and it's a travesty

Just whichever ones have the most hits out of all of them

Track the last time individual pages were crawled so we can pull the ones that need to be updated the most

Needed to run API and host frontend files

"Most appealing abstracts"

Uses data from #41 — "hot" papers with lots of traffic more recently.

Include rankings of the papers in each category!

We need a way to rank all papers by traffic, and it would be chaos to run that calculation every time someone asks.

Postgres looks like the winner

If a new version of a paper is released, we should pull in the download stats from all the old versions too.

NOTE: This may already happen in bioRxiv automatically, not sure.

OTHER NOTE: If bioRxiv doesn't combine traffic numbers between versions, should we keep crawling the old versions to get their updated traffic numbers?

Have column for "all-time"

https://www.postgresql.org/docs/10/static/indexes-ordering.html

When does a paper get most of its traffic? Is it right away, and then it tapers off? How quickly? (Could that be used for some kind of "enduring popularity" metric?) Do some papers start slower and pick up steam over time?

Which ones have the closest ratio of "abstract viewed" to "PDF" downloaded? If we set a minimum number of views, then we can have a list for "Most appealing abstracts"

Also link to all of its authors' pages

We may end up with a weird problem when this thing gets released—if Rxivist is popular enough, the patterns we're observing will end up being influenced by our site. My guess is that popular papers will end up getting more popular, but it will be hard to measure the impact.

If we can at least track how many visitors click on the "view paper" button for a particular article, we can try to account for our impact.

Check how easy it is to weight results. ie "title" hits are better than "abstract" hits

We don't need separate tables for each metric we're ranking on and what the rank is—we could have a table where the primary key is the article ID, and then each field is a metric we need to sort on. This way we could have queries be more precise (for example, only give the 'bounce rate leaders' for papers with more than 50 downloads or something)

Use the new "papers by bounce rate" endpoint

When crawling NEW papers from the list, check to see if we've already crawled an earlier version and update the record's URL, rather than creating a new record.

Use more includes for stuff like the "about" modal:
https://bottlepy.org/docs/dev/stpl.html#template-functions

Should we have a separate JS app to call our API? If so, do we add all the complexity of an advanced framework like Aurelia or Angular 6?

with basic info about each. (NOT all info.)

From about page:

Articles in bioRxiv are categorized as New Results, Confirmatory Results, or Contradictory Results. New Results describe an advance in a field. Confirmatory Results largely replicate and confirm previously published work, whereas Contradictory Results largely replicate experimental approaches used in previously published work but the results contradict and/or do not
support it.

It would be cool to have separate charts for these available: "Most popular contradictory results" sounds fun

Most papers? Most downloads?

Some of the stdout stuff that SHOULD be going to /var/log/rxivist.log is just being written to the console. Hiding things in /var/log/messages is silly, I should figure out how to get this to behave right.

To replicate, go to server:

cd /etc
./rc.local

then refresh the web page

Titles (and authors) can change between revisions, so use URL unless bioRxiv has some snazzy unique identifier we don't know about.

A-la the arxiv-sanity.com login system, where you can bookmark papers (and then it recommends new ones for you!)

Some papers at the top of the "all-time" list may have thousands of downloads, all of them years ago. If we assign a decreasing weight to traffic as it gets farther away from the present, we could have a list that incorporates papers with a lot of downloads but favors papers with RECENT downloads.

Add a flag to turn on a more polite mode of crawling that pauses between major operations to reduce load on bioRxiv

If the link we have for a paper isn't the right one (i.e. there's a newer one available), figure out the new one and update the old entry.

NOTE: Be careful with this one, if we record a new paper in this step, the spider will STOP when it runs into that paper in the "new" results because it thinks we've already seen everything after this one. That won't be the case.

We may be able to prevent this entire problem from happening if we only do the "re-crawl old papers" step AFTER we've pulled the latest papers, so all the URLs will already be updated.

https://www.postgresql.org/docs/9.1/static/textsearch-controls.html

http://www.user-agents.org/