extraneous newlines and whitespace in output about arxiv.py HOT 2 CLOSED

Thanks for the easy instructions for reproducing this!

The whitespace here is returned by the underlying arXiv API. It seems to be a ragged line wrap at 80 characters arXiv introduces themselves, but authors can validly include newlines of their own to break paragraphs (at least in abstracts):

So if you want a new paragraph or a table of contents, be sure to indent the lines after the carriage return.

I'd like to find an example paper with a multiparagraph abstract to confirm how this is represented by the API; "strip all newlines that aren't followed by whitespace" may be a valid rule here.

As far as I know, titles can never include user input newlines, so this might be a more straightforward place to strip them. I'm a little concerned about introducing a find-and-replace operation on every query result. I'll shoot an email to the arXiv API Google Group about this behavior.

In the mean time, a small function for stripping newlines that aren't followed by whitespace from strings:

import re

def rm_newlines(entry):
  rm_newlines = lambda s: re.sub(r'\n[^\s]', ' ', res)
  entry.title = rm_newlines(entry.title)
  entry.summary = rm_newlines(entry.summary)
  return entry

from arxiv.py.

Sounds like this was a bug introduced into the API itself (link to the Google Group message), but the arXiv API is likely frozen while they work on a new JSON API to replace the current Atom API.

from arxiv.py.