hbehrens / feedsanitizer Goto Github PK
View Code? Open in Web Editor NEWsimple web app for Google App Engine to normalize RSS/ATOM feeds
Home Page: http://feedsanitizer.appspot.com
License: MIT License
feedsanitizer's Introduction
The Feed Sanitizer is a web app to normalize RSS/ATOM feeds. It's open source and published under the MIT License. You can see the running version at http://feedsanitizer.appspot.com/ Heiko Behrens http://HeikoBehrens.net built The Feed Sanitizer with Python and djangoappengine (hence the large setup) to be execute by the fabulous Google App Engine using several open-source libraries such as feedparser by Mark Pilgrim http://www.feedparser.org SyntaxHighlighter by Alex Gorbatchev http://alexgorbatchev.com/SyntaxHighlighter The RSS and ATOM icons were provided by courtesy of feedicons.com and blogage.de.
feedsanitizer's People
Contributors
Stargazers
Watchers
feedsanitizer's Issues
Atom author metadata gets "flattened"
If an Atom feed contains an <author> element with <name> and <email> metadata, like so:
<author>
<name>Bob</name>
<email>[email protected]</email>
</author>...it gets turned into this in the sanitized output:
<author>
<name>Bob ([email protected])</name>
</author>Feedsanitizer should keep the metadata separate, as in the original feed.
Oops on http://www.miami.edu/it/index.php/it/it_alerts_rss/
url: http://www.miami.edu/it/index.php/it/it_alerts_rss/
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 83, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 32, in buildfeed
entry_link = entry.link if "link" in entry else entry.links[0]["href"]
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'links'
provide feedback for invalid urls
provide better feedback on errors
Error
Feed seems oke in Google Reader, b ut not for import in Yahoo pipes to merge with another feed.
url: http://www.civitas.eu/rss.php
format: atom
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 83, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 32, in buildfeed
entry_link = entry.link if "link" in entry else entry.links[0]["href"]
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'links'
Trunews rss xml not importing
url: http://www.trunews.com/listen_now_podcast_feed.xml
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.352729602152509413/views.py", line 71, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.352729602152509413/views.py", line 31, in buildfeed
idish = entry.id if "id" in entry else fixurl(entry.link)
File "/base/data/home/apps/feedsanitizer/1.352729602152509413/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'link'
Oops! An error occured...
url: http://artent.net/feed/
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 86, in userfriendly
prettyxml = xml.dom.minidom.parseString(feedxml.getvalue()).toprettyxml()
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/minidom.py", line 1923, in parseString
return expatbuilder.parseString(string)
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 940, in parseString
return builder.parseString(string)
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 223, in parseString
parser.Parse(string, True)
ExpatError: not well-formed (invalid token): line 2, column 5817
It looks to be a Unicode issue in the feed - the W3C feed validator says:
TheIntelHub Feed doesn't convert to RSS or Atom
url: http://theintelhub.com/feed/rss/
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353467691866276609/views.py", line 82, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353467691866276609/views.py", line 31, in buildfeed
updated = datetimefromparsed(entry.updated_parsed)
File "/base/data/home/apps/feedsanitizer/1.353467691866276609/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'updated_parsed'
IntelHub Feed Doesn't Convert to Atom RSS
url: http://theintelhub.com/feed/rss/
format: atom
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353467691866276609/views.py", line 82, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353467691866276609/views.py", line 31, in buildfeed
updated = datetimefromparsed(entry.updated_parsed)
File "/base/data/home/apps/feedsanitizer/1.353467691866276609/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'updated_parsed'
allow user to validate input and output feed
Oops on http://www.brainfeedersite.com/feed/
url: http://www.brainfeedersite.com/feed/
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 86, in userfriendly
prettyxml = xml.dom.minidom.parseString(feedxml.getvalue()).toprettyxml()
File "/base/python_runtime/python_dist/lib/python2.5/xml/dom/minidom.py", line 1923, in parseString
return expatbuilder.parseString(string)
File "/base/python_runtime/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 940, in parseString
return builder.parseString(string)
File "/base/python_runtime/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 223, in parseString
parser.Parse(string, True)
ExpatError: not well-formed (invalid token): line 57, column 45
Thanks!
handle feeds without authors
Oops! An error occured...
url: http://www.radiodixie.cz/xml/radio/podcasty_rss
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 86, in userfriendly
prettyxml = xml.dom.minidom.parseString(feedxml.getvalue()).toprettyxml()
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/minidom.py", line 1923, in parseString
return expatbuilder.parseString(string)
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 940, in parseString
return builder.parseString(string)
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 223, in parseString
parser.Parse(string, True)
ExpatError: not well-formed (invalid token): line 12, column 153
"Oops" on http://u3d.as/feed/latest.rss
When trying to sanitize http://u3d.as/feed/latest.rss
I'm seeing a trackback:
url: http://u3d.as/feed/latest.rss
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 83, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 32, in buildfeed
entry_link = entry.link if "link" in entry else entry.links[0]["href"]
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'links'
http://feedsanitizer.appspot.com/?url=http%3A%2F%2Fu3d.as%2Ffeed%2Flatest.rss&format=rss
Oops! An error occured..
url: http://www.lisa.gerda-henkel-stiftung.de/blog_feed.php?type=rss
format: atom
http://www.lisa.gerda-henkel-stiftung.de/blog_feed.php?type=rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 86, in userfriendly
prettyxml = xml.dom.minidom.parseString(feedxml.getvalue()).toprettyxml()
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/minidom.py", line 1923, in parseString
return expatbuilder.parseString(string)
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 940, in parseString
return builder.parseString(string)
File "/base/data/home/runtimes/python/python_dist/lib/python2.5/xml/dom/expatbuilder.py", line 223, in parseString
parser.Parse(string, True)
ExpatError: not well-formed (invalid token): line 357, column 73
Thanks a lot!
produce canonical URIs for ATOM feeds
http://validator.w3.org/feed/docs/warning/NonCanonicalURI.html
This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.
line 2, column 301: Identifier "http://HeikoBehrens.net" is not in canonical form (the canonical form would be "http://heikobehrens.net/") (4 occurrences) [help]
... self">http://HeikoBehrens.net2011-02-27T12:56:4 ...
http://feedsanitizer.appspot.com/ no longer working
Feed sanitizer error
url: http://www.samandfuzzy.com/rss.php
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.349123672774720160/views.py", line 70, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.349123672774720160/views.py", line 31, in buildfeed
id = "http://feedsanitizer.appspot.com/id/%s" % (urllib.quote_plus(entry.id))
File "/base/data/home/apps/feedsanitizer/1.349123672774720160/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'id'
Oops! An error occured...
url: http://rss.25ab2e52ba887be63c8849eab768729c.metarnew.db.bcinfo3.barrycarter.info/
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 83, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 32, in buildfeed
entry_link = entry.link if "link" in entry else entry.links[0]["href"]
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'links'
Globalresearch fails to clean properly
url: http://www.globalresearch.ca/rss.php
format: atom
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353722641856664597/views.py", line 82, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353722641856664597/views.py", line 36, in buildfeed
id = "http://feedsanitizer.appspot.com/id/%s" % (urllib.quote_plus(idish))
File "/base/python_runtime/python_dist/lib/python2.5/urllib.py", line 1222, in quote_plus
return quote(s, safe)
File "/base/python_runtime/python_dist/lib/python2.5/urllib.py", line 1214, in quote
res = map(safe_map.getitem, s)
KeyError: u'\u2019'
Oops! An error occured...
url: http://www.odin.dk/RSS/RSS.aspx?beredskabsID=bbbbbbbb-1111-2222-3333-bbbbbbbbbbbb
format: rss
Traceback (most recent call last):
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 83, in userfriendly
feed = buildfeed(url, feed_class, feed_link, ids)
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/views.py", line 32, in buildfeed
entry_link = entry.link if "link" in entry else entry.links[0]["href"]
File "/base/data/home/apps/feedsanitizer/1.353769649187310229/feedparser.py", line 356, in getattr
raise AttributeError, "object has no attribute '%s'" % key
AttributeError: object has no attribute 'links'
prevent submitting obviously wrong urls in UI
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.