comsaint / legco-watch Goto Github PK

Modify Scrapy codes so it only downloads formal Hansard to save time and bandwidth.

The current Hansard parser works only with Hansards from 2012 on. Modify current code/ make a new parser for the ones before.

Multiple issues with agenda questions:

• Questions of agendas on 2014.06.18 are successfully scraped and shows up in the parsed source documents, but some questions are missing in view (for English version only).
• When there are more than one responder for a question, the current view shows only one of them.
• There is at least one occasion that the content of a question is completely ripped off from the view, although this content exists in the parsed source document (for both languages), and the question entry for it also exists in view. See Q5 in agenda on 2014.10.15.
• It may be a good idea to include a field responders in the model RawCouncilQuestion.

council_agenda-20141120-e and council_agenda-20141120-c (in raw_agenda) failed to be parsed correctly - object was scraped but not successfully processed by processor.LibraryAgendaProcessor. Reason unknown.

Smarter recognition for

• Distinguishing Legislation and Other Papers sections; and
• Separating contents of papers, especially the Other Papers section of Chinese Hansards.

There are Hansards divided into several files. Append them together as one.