jgm / cmark-hs Goto Github PK
View Code? Open in Web Editor NEWHaskell bindings to libcmark commonmark parser
License: Other
cmark-hs's People
Contributors
Stargazers
Watchers
cmark-hs's Issues
Problem with string decoding
Hello. I met a bug which drives me crazy.
The problem is that in very specific conditions code commonMarkToNode [] (T.pack "«") fails with this error:
Cannot decode byte '\xc2': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream.
I guess that the problem is with string marshaling. Probably byte length is messed up with codepoint length somewhere.
Here is a sniplet which allows to get the same error in pure haskell.
Prelude> :m +Data.Text
Prelude Data.Text> :m +Data.Text.Encoding
Prelude Data.Text Data.Text.Encoding> let s = pack "«"
Prelude Data.Text Data.Text.Encoding> decodeUtf8 $ Data.ByteString.take (Data.Text.length s) $ encodeUtf8 $ s
"*** Exception: Cannot decode byte '\xc2': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
Prelude Data.Text Data.Text.Encoding>
It happens only if the string contains non-ascii symbols.
To reproduce a whole problem a very specific conditions must be met. It never occurs in interactive evaluation (only when called from binary). Unimporting some modules from binary can solve the problem. And so on. I am trying to say that it's really hard to reproduce, but if it's reproduced it stably appears on every execution. In case if the error-line above wouldn't give you insight on a problem, I can capture it and ship for experiments.
Provide pattern synonyms for the AST?
I found it useful to have pattern synonyms like these: https://github.com/aelve/cmark-patterns/blob/master/lib/CMark/Patterns.hs. Perhaps they could be added to cmark itself? (If not, I'm happy to keep them in my library.)
Nondeterministic output with garbled strings
I suspect this is a memory handling bug related to PR #13; see the discussion there.
@andersk can you have a look? I'm almost positive it's an issue with your PR, rather than with cmark.
@kivikakk this affects cmark-gfm too.
If a quick fix is not forthcoming, we should roll back PR #13 and make a new release.
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "^") []]]
"^\n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "^") []]]
"^Z\n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "^") []]]
"^\\*%\n"
Also reproducible with other strings:
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT " ") []]]
" p@R\n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT " ") []]]
" \n"
Prelude CMarkGFM> nodeToCommonmark [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"hiC\SOHB\n"
Affects other writers as well:
Prelude CMarkGFM> nodeToMan [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
".PP\nhiz\v\SOH\n"
Prelude CMarkGFM> nodeToLaTeX [] Nothing $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"hiaR\n"
Prelude CMarkGFM> nodeToXml [] $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"*** Exception: Cannot decode byte '\xd2': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
Prelude CMarkGFM> nodeToHtml [] [] $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"<p>hi[\aB</p>\n"
Prelude CMarkGFM> nodeToHtml [] [] $ Node Nothing DOCUMENT [Node Nothing PARAGRAPH [Node Nothing (TEXT "hi") []]]
"<p>hio\EOTB</p>\n"
Position tracking is inconsistent
Position tracking of headers is inconsistent
tests failed on armv7
Not sure what is at fault but I note that some test-cases are failing when test building 0.5.1 in Fedora 24 development (Rawhide) for armv7hl:
:
+ ./Setup test
Running 1 test suites...
Test suite test-cmark: RUNNING...
Cases: 6 Tried: 0 Errors: 0 Failures: 0
Cases: 6 Tried: 1 Errors: 0 Failures: 0
Cases: 6 Tried: 2 Errors: 0 Failures: 0
Cases: 6 Tried: 3 Errors: 0 Failures: 0
Cases: 6 Tried: 4 Errors: 0 Failures: 0
### Failure in: 4
expected: Node (Just (PosInfo {startLine = 1, startColumn = 1, endLine = 1, endColumn = 13})) DOCUMENT [Node (Just (PosInfo {startLine = 1, startColumn = 1, endLine = 1, endColumn = 13})) PARAGRAPH [Node Nothing DOCUMENT [],Node Nothing EMPH [Node Nothing DOCUMENT []]]]
but got: Node (Just (PosInfo {startLine = 1, startColumn = 1, endLine = 1, endColumn = 13})) DOCUMENT [Node (Just (PosInfo {startLine = 1, startColumn = 1, endLine = 1, endColumn = 13})) PARAGRAPH [Node Nothing (TEXT "Hello ") [],Node Nothing EMPH [Node Nothing (TEXT "world") []]]]
Cases: 6 Tried: 5 Errors: 0 Failures: 1
### Failure in: 5
expected: "> Hello\n> *world*\n"
but got: "> **\n"
Cases: 6 Tried: 6 Errors: 0 Failures: 2
Test suite test-cmark: FAIL
:
This is with ghc-7.8.4.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.