Support for citations via bibtex/csl about django-wiki HOT 13 OPEN

Thanks, I think the first step is to make a decision on how to both support quick and dirty citation and a structured one.

Hopefully they are not each others opposites. If a flexible structure can be built then it would be great to have citation data properly contained in the wiki. For instance, with typeahead to match the ID of a citation source...

For the actual style of the citations, this could be left for the template. I realized in uni that it's impossible to match everyone's preferences of citation styles, and that the best thing is to keep everything in a proper database and then format the output in the very end.

Is it safe to assume that citation sources are not unique to a particular article but actually span the whole wiki? Or should there perhaps be a differentiation between site-wide sources and article-specific sources.

Hi, here's a crazy statement [Ref:CrazyStatementSource]

...then the source should either be available throughout the wiki for reuse, statistics, special views etc... or it should just be attached to the article because it's irrelevant for other purposes.

from django-wiki.

I'd strongly recommend looking at Pandoc's approach to this – it's really very good, and draws on all the work that has been done with CSL (it's an xml format to describe citation styles which is now rather complete - see http://citationstyles.org)

I've actually made a start on this, based on the links plugin/preprocessor and the django-publications app mentioned above. It's only the quick and dirty part though – the harder bit will be integrating it with citeproc to get the formatting right. Would you be interested in taking a contribution?

From experience, I'd actually recommend against trying to implement citation formatting in django templates... you end up with far too much logic in the templates to handle the weird cases.

from django-wiki.

I would definitely handle the citation style in a template - if someone wants their wiki to do APA style, then the way forward should be to add an APA template etc.

The data should be in a Django model... that's the philosophy of the wiki... to store stuff using Django's ORM. If CSL is to be used, it would be through an export view that can generate the XML necessary. I don't see why a Django application should store any of its data in XML. But of course it would be awesome to copy their ideas for data structures.

Btw... beware that the python-markdown preprocessor stuff is currently a bug that I'm looking to fix :) It turns out that using markdown safe mode removes HTML generated by preprocessors, so any stuff like that must be handled in a BlockProcessor... or maybe something else :)

from django-wiki.

I would definitely handle the citation style in a template - if someone wants their wiki to do APA style, then the way forward should be to add an APA template etc.

Yes - but I really wouldn't underestimate the effort required to implement even 4 or 5 basic styles. APA, for example, is a pig to get really-right (e.g. handling the different format for the second in-text citation with more than 3 authors), and the CSL people have gone through the pain of getting the formats right already for hundreds of journals, each of which have slightly different requirements. I'd have a look at it before deciding it wasn't invented here etc.

The data should be in a Django model... that's the philosophy of the wiki... to store stuff using Django's ORM. If CSL is to be used, it would be through an export view that can generate the XML necessary. I don't see why a Django application should store any of its data in XML. But of course it would be awesome to copy their ideas for data

Of course - the data should be in a model. I just think CSL should be used to format the data.

Btw... beware that the python-markdown preprocessor stuff is currently a bug that I'm looking to fix :) It turns out that using markdown safe mode removes HTML generated by preprocessors, so any stuff like that must be handled in a BlockProcessor... or maybe something else :)

OK - will bear that in mind. However I'm actually reformatting text into markdown (e.g. to create links) rather than making html directly — would this still be a problem?

from django-wiki.

Of course - the data should be in a model. I just think CSL should be used to format the data.

In that regard I don't necessarily disagree.. but I just don't understand it :) Are you talking about the user typing in CSL? Because that really conflicts with the Markdown syntax / philosophy:

http://daringfireball.net/projects/markdown/syntax#philosophy

Wikipedia is already doing tons of things wrong in regards to user interfaces and syntax... over-complications introduced from academia. And I think that the citation circus shouldn't be imported to the project. However, I'm open to creating a design that allows people to output citation styles the way they like them.

Multi-author support might be a good idea from a POV that citation styles require its source data to maintain a data structure that supports many authors etc.

Have you seen the OpenOffice bibliography database? It's awful and it's a dead project development-wise. It never left the ground, probably because it got so cluttered that no one would bother working on it... I think there are some lessons here to be learned :)

from django-wiki.

There are differences between a wiki article and a scientific publication; One content is interactive, the other is meant to be printed... which is apparent from the way citation styles look like they do :) I think it's far more relevant to think about the possibilities inherent from having an interactive, searchable database of content-relational citations.

So the real question is not: How can we make a citation data model that will allow us to support printed media? -- Rather: How can we make a data structure that opens up new interactive possibilities for navigating knowledge, enhancing accountability and trustworthiness of collaborative contents? (and at the same time not mess everything up in the user interface)

from django-wiki.

OK - will bear that in mind. However I'm actually reformatting text into markdown (e.g. to create links) rather than making html directly — would this still be a problem?

No, that's perfect... the preprocessors purpose is afaik to preprocess stuff into markdown code.

from django-wiki.

I completely agree with you (both posts). On reflection realise I've got caught up in this from the pain of actually having to do it for papers.

What you implement needs to be simple and in line with the aims of the wiki. That said, I think there's an opportunity to do something that is simple but lets people extend it and reuse the content in useful ways.

In that regard I don't necessarily disagree.. but I just don't understand it :) Are you talking about the user typing in CSL? Because that really conflicts with the Markdown syntax / philosophy:

Completely agree. I think I need to check the standard again, but what pandoc does is very nice. Essentially the CSL is an xml file which the user just chooses to format references. They create in text citations and bibliographies with a very simple wiki-like syntax. So, for example, to cite a paper supporting this point I would say [@bibtexcitekeygere] . Depending on the CSL chosen, this would then format the in-text citation appropriately, e.g. (Bloggs, 2012) and create a bibliography in the right format at the end of the text.

In the first instance simply adopting this format to reference publications stored in a django model that is compatible with BibTex fields would be great. We can write a preprocessor to do simple in-text replacement of citations in a single format for the wiki itself, and it would mean that it would be easy to create nice offline pdf versions of texts from the wiki using pandoc, because the intext citation format would be the same.

Wikipedia is already doing tons of things wrong in regards to user interfaces and syntax... over-complications introduced from academia. And I think that the citation circus shouldn't be imported to the project. However, I'm open to creating a design that allows people to output citation styles the way they like them.

Yes - although maybe just using a format (markdown with csl style in text citation markers) would let people use pandoc to fiddle and get things 'just right' without overcomplicating things.

Multi-author support might be a good idea from a POV that citation styles require its source data to maintain a data structure that supports many authors etc. Have you seen the OpenOffice bibliography database? It's awful and it's a dead project development-wise. It never left the ground, probably because it got so cluttered that no one would bother working on it... I think there are some lessons here to be learned :)

Agreed... and actually it's not your problem to solve the mess that is academic writing and citation formatting.

For now I'm going to have a bash at integrating a publication model based on django-publications with a plugin that allows citations – in the first instance it can just be a pluggable app which depends on wiki.

from django-wiki.

I would add one thing then: Never store structured data in the content area ! The implications of this is that for instance you wouldn't put bibtex syntax here... I want the wiki to build upon this principle so it becomes evident what can be obtained with structured data that's not nested inside millions of awful syntaxes... but that everything is accessible in Django models.

But it's okay to reference data in the content area. So [image:id] is replaced by data from a Django model. And [ref:id] could be replaced as well... but the data inside this reference is saved in a Django model.

BibTex is a really good idea for a plugin that could handle import... same with PubMed IDs or ISBNs. But these should import a data structure into a Django model, right? Or would you rather just store the raw data and parse/output citation styles on the fly with Pandoc? Although I love Haskell, I think that Pandoc sounds like a heavy dependency.

from django-wiki.

Yes - agreed. I guess all I'm suggesting is to adopt the existing pandoc/csl syntax for marking where in the text to include a citation. It's very similar to a wiki syntax anyway, and would facilitate reuse.

I would add one thing then: Never store structured data in the content area ! The implications of this is that for instance you wouldn't put bibtex syntax here.

That's right. Actually django-publications parses bibtex into a model, then recreates it again if you need it.

BibTex is a really good idea for a plugin that could handle import... same with PubMed IDs or ISBNs. But these should import a data structure into a Django model, right? Or would you rather just store the raw data and parse/output citation styles on the fly with Pandoc?

Although I love Haskell, I think that Pandoc sounds like a heavy dependency.

No - pandoc is way too heavy for the core functionality... but a person could use it for some kinds of export if we just store markdown with [@citekey] style references.

from django-wiki.

Maybe I should have a better look at this django-publications :) It doesn't seem to be a very big project. If possible, parts of the code could just be imported.. I think it's possible (this is a GPLv3 project) to bring the MIT licensed code here. Afterall, it doesn't seem to be very actively developed anymore.

Do you think it would be a useful ally for implementing a citation plugin? Afterall, it does bibtex and already has a lot of the data structure.

I also think that importing the code is a better way to go because the wiki might generate a lot of feature requests...

from django-wiki.

I also thought I'd just post you what I've done with the citation preprocessor:

https://gist.github.com/3664375

And the plugin:

https://gist.github.com/3fbcfa28b4f53a74f59b

I've just created a copy of the links app to wrap this up in. One thing I did notice is that it would be nice to have more flexible autocomplete on links and citations. For example, when trying to autocomplete a citation it would be nice to be able to preview the titles, but then paste the citation syntax into the form field for insertion. This would take a bit of a rejig of what is there at present.

from django-wiki.

Is anyone actively working on this? I'm writing a Django app and this seems to be the best Wiki available unless I want to screw around with trying to set up one of the PHP wikis and my Django app on the same server (which I really don't want to do since I don't have much sysadmin experience and I can't ensure the security of such a setup).

I don't mind pitching in myself, but I wanted to make sure I'm not duplicating work!

from django-wiki.