dlang-community / libdparse Goto Github PK
View Code? Open in Web Editor NEWLibrary for lexing and parsing D source code
Home Page: https://libdparse.dlang.io
License: Boost Software License 1.0
Library for lexing and parsing D source code
Home Page: https://libdparse.dlang.io
License: Boost Software License 1.0
The ddoc page here http://dlang.org/spec/ddoc.html says "Multiple documentation comments applying to the same declaration are concatenated" and this example demos it with block and trailing:
/** documentation for g */
int g; /// more documentation for g
libdparse does not seem to recognize this:
node.comment = start.comment;
if (node.comment is null)
node.comment = start.trailingComment;
That's from https://github.com/Hackerpilot/libdparse/blob/master/src/dparse/parser.d#L3911 but the same pattern appears throughout the parser.
There also seem to be a few cases where the trailing comment just gets lost too, like in parsing an AliasDeclaration
: https://github.com/Hackerpilot/libdparse/blob/master/src/dparse/parser.d#L84
And I don't see it attached on the initializer list either, though I might be missing something there, I don't know your code very well yet.
With this test case:
/// test
alias cool = string; /// which comment is there?
I ran it through the visitor and checked AliasDeclaration.comment and it only gave me "/// test". If I take the top comment off, the comment is null.
My guess is what you'll want to do is remove those if(null) conditions and use ~= instead, but the whole allocation business probably makes that a bit more than it seems...
Hi.
Template value parameters with specializations break other parameters after them.
void func(string s : "foo", string t : "bar")()
{
}
void func2(string s : "foo", T : int)()
{
}
void func3(string s: "foo", T)() // no error, but produced wrong AST
{
}
$ dscanner -s test.d
test.d(1:36)[error]: Expected ) instead of t
test.d(5:32)[error]: Expected ) instead of :
AST for template parameters of func3:
<templateParameters>
<templateParameterList>
<templateParameter>
<templateValueParameter>
<type pretty="string">
<type2>
<symbol>
<identifierOrTemplateChain>
<identifierOrTemplateInstance>
<identifier>string</identifier>
</identifierOrTemplateInstance>
</identifierOrTemplateChain>
</symbol>
</type2>
</type>
<identifier>s</identifier>
<expression>
<assignExpression>
<ternaryExpression>
<cmpExpression>
<unaryExpression>
<primaryExpression>
<stringLiteral>"foo"</stringLiteral>
</primaryExpression>
</unaryExpression>
</cmpExpression>
</ternaryExpression>
</assignExpression>
<assignExpression>
<ternaryExpression>
<cmpExpression>
<unaryExpression>
<primaryExpression>
<identifierOrTemplateInstance>
<identifier>T</identifier> <!-- eaten! -->
</identifierOrTemplateInstance>
</primaryExpression>
</unaryExpression>
</cmpExpression>
</ternaryExpression>
</assignExpression>
</expression>
</templateValueParameter>
</templateParameter>
</templateParameterList>
</templateParameters>
{
int foo();
}
{}
libdparse will accept this code, while DMD will refuse it:
test/pass_files/issue0070.d(1): Error: declaration expected, not '{'
test/pass_files/issue0070.d(4): Error: unrecognized declaration
Tested with 2.069.
this is a minor feature request: line and column info in the ASTNode StaticConstructor
and StaticDestructor
.
Hi,
I get a lot of errors when trying to run dscanner on my code (snippet shown further down). First thing it triggers on below is the immutable in the foreach:
src/boss/core/boss.d(255:14)[error]: Expected ; instead of immutable
src/boss/core/boss.d(255:58)[error]: Expected ; instead of )
src/boss/core/boss.d(255:58)[error]: Primary expression expected
src/boss/core/boss.d(255:60)[error]: Expected ; instead of {
void showHelp() {
//First accumulate options
string procHelp[][];
foreach (immutable(ProcType) procType; ProcType.types) {
immutable string[] helpPrim = procType.helpPrim;
for (int i=0; i<helpPrim.length; i+=2) {
procHelp ~= [[helpPrim[i+0], helpPrim[i+1]]];
}
}
if (m_options.helpall) {
foreach (procType; ProcType.types) {
immutable string[] helpExtra = procType.helpExtra;
for (int i=0; i<helpExtra.length; i+=2) {
procHelp ~= [[helpExtra[i+0], helpExtra[i+1]]];
}
}
}
}
It also complains about the code further down, with the first error being on the line indexing into 'img'.. Image has an index operator:
src/cv/image/borderlinedet.d(432:19)[error]: Expected ] instead of ,
src/cv/image/borderlinedet.d(432:32)[error]: Expected identifier instead of ]
src/cv/image/borderlinedet.d(432:34)[error]: Primary expression expected
src/cv/image/borderlinedet.d(433:2)[error]: Expected ; instead of }
BorderLineDet!Mono8.Data imgFromStr(string str) {
string[] lines = str.split("|");
auto img = Image!Mono8(cast(int) lines[0].length, cast(int) lines.length);
foreach (y, l; lines) {
foreach (x, p; l) {
img[cast(int) x, cast(int) y] = p == '1' ? 0 : Mono8.Chan.max;
}
}
return BorderLineDet!Mono8.Data(img);
}
}
interface I
{
int foo(int i)
in { assert(i > 7); }
out (result) { assert(result & 1); }
void bar();
}
// took from http://dlang.org/interface.html#InterfaceContracts
foo.d(7:5)[error]: Expected body instead of void
foo.d(7:15)[warn]: Empty declaration
Without documentation your project may remain without attention.
And harbored is useless, it generates only half of docu because of some strange bugs.
D:\Code\D\xxxx\xxxx\libs\libdparse>harbored -m macros.ddoc src/
Writing documentation to ./doc
Generating documentation for src/std\allocator.d
Could not generate documentation for File(91FD08, "src/std\allocator.d"): outdent: Inconsistent ind
entation
Generating documentation for src/std\d\ast.d
Generating documentation for src/std\d\entities.d
Generating documentation for src/std\d\formatter.d
Generating documentation for src/std\d\lexer.d
Generating documentation for src/std\d\parser.d
Could not generate documentation for File(93D388, "src/std\d\parser.d"): Cannot open file ./doc\s td\d\parser.Parser.parseCompileCondition.html' in mode
w' (Too many open files)
Generating documentation for src/std\lexer.d
Could not generate documentation for File(93D388, "src/std\lexer.d"): Cannot open file ./doc\std\l exer.IdType.html' in mode
w' (Too many open files)
D:\Code\D\xxxx\xxxx\libs\libdparse>
Windows 7 / x86_64.
I'm counting 75 [error]
lines when running DScanner against my ae library:
Originally posted as dlang-community/D-Scanner#197
Line 612 in typecons.d:
private enum bool isPrintable(T) =
is(typeof({
import std.format : formattedWrite;
Appender!string w;
formattedWrite(w, "%s", T.init);
}));
Error: Expected body instead of =
module tango.core.memory;
version(D_Version2)
{
public import core.memory;
}
else:
test1.d(7:5)[error]: Declaration expected
Tested with master and 0.2.1
class Foo
{
class Bar {}
}
void main ()
{
auto foo = new Foo;
auto bar = foo.new Bar;
}
test2.d(9:20)[error]: Expected identifier instead of new
test2.d(10:1)[error]: Expected ; instead of }
Tested on 0.2.1 + master
Since dlang/dmd#4033 was merged, asm statements can have function attributes after asm
and before the opening {
Right now ddmd's lexer is much faster: http://i.imgur.com/k50dFbU.png
I've been messing with conditional declarations lately, as I want to modify the declaration finder in Dscanner to help me separate out the linux and glibc APIs in druntime. I wasn't getting the results I wanted so I tried narrowing down the problem, by simply adding this line to symbol_finder.d inside the FinderVisitor
class:
override void visit(const ConditionalDeclaration cd) { }
After recompiling Dscanner, I created this sample file foo.d to test against:
version(Windows){
version(X86_64){
int foo = 3;
char mar = "c";
}
long bar = 3;
}
If I run ./bin/dscanner -d bar foo.d
, I get zero results, just as expected. However, if I modify the file to this, which should be equivalent:
version(Windows):
version(X86_64){
int foo = 3;
char mar = "c";
}
long bar = 3;
I get the following result instead:
> ./bin/dscanner -d bar foo.d
foo.d(6:6)
In all cases, a search for foo
or mar
turns up nothing, so that still works: it's the colon-terminated conditional declarations that appear to be broken. I notice that there are no tests in libdparse for nested conditional declarations inside a colon-terminated conditional declaration, so perhaps that case hasn't been tested well, but it is ubiquitous in druntime.
The implicitly concatenated string is causing errors to be shown in dscanner.
Example:
deprecated("Please use transform1D with the alias template parameter instead"
" and remember to switch the DataT parameters!")
auto transform1D(string ChangingDim, CubeT, MapRange, DestDim, DataT = CubeT.Data,
MapVal = typeof(ElementType!(MapRange).value))
As discovered with D-Scanner's AST output, this code does not get the right parse tree:
b = 1, 2;
The grammar changed again, so the parser needs to be updated.
It would be better for library users if the 'new alias syntax warning' could be moved from the library to dscanner / dfix.
I'd really like to be able to fix the thousands of occurrences of this warning, but I have neither the time nor the possibility ;)
Example:
/** some doc comment */
version (linux) void doStuff(arg arg1) {}
Originally reported as dlang-community/D-Scanner#265
unittest
{
Label:
}
(I filed https://issues.dlang.org/show_bug.cgi?id=14905 to avoid repeating such warning messages in dmd in a reduced case, but regardless, this warning-as-error needs to be addressed)
=> sent out burner/inifiled#4 for this which removes 'return false' from the 2 offending lines.
Also:
I'd like to write a symbol list module for Coedit based on libdparse. I'm in front of a problem, how can I templatize this pattern ?
auto config = LexerConfig(filename, StringBehavior.source);
StringCache cache = StringCache(StringCache.defaultBucketCount);
const(Token)[] tokens = getTokensForParser(rawSrc, config, &cache);
auto ast = parseModule(tokens, filename, null, null);
Declaration[] decls = ast.declarations;
if(!decls.length) return;
foreach(q;queries)
{
// class declarations
if (q == "cl")
{
string classNames = q ~ "=\"";
foreach(i; 0 .. decls.length)
if (decls[i].classDeclaration !is null)
classNames ~= decls[i].classDeclaration.name.text ~';';
classNames ~= "\"";
writeln(classNames);
continue;
}
// structure declarations
if (q == "st")
{
string structNames = q ~ "=\"";
foreach(i; 0 .. decls.length)
if (decls[i].structDeclaration !is null)
structNames ~= decls[i].structDeclaration.name.text ~';';
structNames ~= "\"";
writeln(structNames);
continue;
}
// etc, following the pattern for interfaces, functions, templates, imports,...
}
each member list of a cat is get from the same fashion, but since a category is actually a member of a node, I cant see how to templatize the thing:
void listOf(C)(string category)
{
string list = category ~ "=\"";
foreach(i; 0 .. decls.length)
if (cast(C)decls[i] !is null)
list ~= (cast(C)decls[i]).name.text ~';';
list ~= "\"";
writeln(list);
}
does not work. I could list everything but the background problem is that the members to enumerate rely on a list of query, sometime just the module name, sometime just the classes. etc. Rhx, sorry to post this Q here, maybe not the right place.
I find the fact that Parser.messageFunction
is a function and not a delegate not very practical.
When a parser is dedicated to a particular class instance, there is no easy way to determine if an item in the errors stack is related to an instance or another (apart from the filename but testing a string for each message is not the best way to identify an instance).
At least a user parameter could be given to the parser and passed for each message so that it would be possible to get a particular class instance.
So either the type of the callback should be changed to
void delegate(string, size_t, size_t, string, bool)
or
void function(string, size_t, size_t, string, bool, void*)`
With the last pointer the user param passed back by the parser, and in this case parseModule
declaration would be something like that:
Module parseModule(const(Token)[] tokens, string fileName, CAllocator allocator = null,
void function(string, size_t, size_t, string, bool) messageFunction = null,
void* userParam = null, uint* errorCount = null, uint* warningCount = null)
// or maybe userParam at the end so that existing code des not break.
so that the callback can do that:
class Foo
{
struct AstError{}
AstError*[] errors; // no static !!
final static void parserError(string fname, size_t line, size_t col, string msg, bool isErr, void* userParam)
{
with( cast(Foo) userParam) errors ~= new AstError(line, col, msg, isErr);
}
}
the parser handles the occurence of __EOF__
not correctly.
it says Declaration expected
I found this by parsing the bundeld allocator.d
for a custom type that takes two ints in opSlice and also overload opIndex
CustomType t = CustomType[0..9, 1..6];
libdparse produces the following error about the comma between the two opSlices:
Expected ] instead of ,
while this is not error
Originally reported as dlang-community/D-Scanner#263.
unittest
{
ushort r = bswap(*(cast(ushort*) (bytes.ptr + index));
}
class Test
{
static @trusted this() {}
}
auto a = 1, 2;
$ dscanner --ast
@test void foo();
<?xml version="1.0"?>
<module>
<declaration>
<attribute>
<atAttribute>
<identifier>test</identifier>
</atAttribute>
</attribute>
<functionDeclaration line="1">
<name>foo</name><type pretty="void">
<type2>void</type2>
</type>
<parameters>
</parameters>
</functionDeclaration>
</declaration>
</module>
Yet:
$ dscanner --ast
@test: void foo();
<?xml version="1.0"?>
<module>
<declaration>
<attributeDeclaration>
<attribute>
<atAttribute>
<identifier>test</identifier>
</atAttribute>
</attribute>
</attributeDeclaration>
</declaration>
<declaration>
<functionDeclaration line="1">
<name>foo</name><type pretty="void">
<type2>void</type2>
</type>
<parameters>
</parameters>
</functionDeclaration>
</declaration>
</module>
Notice that the first one has the attribute inside the declaration, but the second one has it in a different declaration.
I think the way dmd does this is the colon starts a declaration list and the attribute is attached to everything in that list. Does the same with privacy keywords btw.
But, I think dmd also does that attachment as part of semantic analysis, so I'm not sure this is a bug strictly speaking... but I'd like to be sure.
The following ExpressionStatement containing a function literal gets parsed as a block statement.
import std.stdio;
void main()
{
{ writeln("test"); };
}
Note: DMD gets this wrong as well.
I'm experimenting with Markdown for documentation (Markdown depends on indentation).
I've noticed one more space than needed (?) is being stripped by unDecorateComment()
in /**
comment lines that didn't "start" with a non-space character.
I don't know if this is an off-by-one error or a peculiarity of the D spec.
It can be fixed by changing <=
into <
here in std.d.lexer.undecorateComment()
(line 1920
):
for (size_t s = 0; (i < j) && (s <= whitespaceToSkip)
&& (comment[i] == ' ' || comment[i] == '\t');)
Example of the behavoir:
/** Merge two entity prototypes; components from over override components from base. The
* returned prototype is not locked/trimmed.
*
...
*
* 1. This is a list item with two paragraphs. Lorem ipsum dolor
* sit amet, consectetuer adipiscing elit. Aliquam hendrerit
* mi posuere lectus.
*/
the second part would turn into:
1. This is a list item with two paragraphs. Lorem ipsum dolor
sit amet, consectetuer adipiscing elit. Aliquam hendrerit
Similarly,
* A
* B
* A
would turn into:
A
B
A
If I put libdparse as a DUB dependency, DUB will fetch 0.2.1 which is quite obsolete now.
the template
/// Ditto
template octal(alias s)
if (isIntegral!(typeof(s)))
{
enum auto octal = octal!(typeof(s), to!string(s));
}
produces four libdparse warnings:
Expected identifier instead of =
Declaration expected
Declaration expected
Empty declaration
Actually i wonder if it's not rather a dmd bug (to allow this syntax) since the type is defined two times (enum auto
) ?
[deleted]
I've modified Dscanner to help me find all version blocks in druntime that have a default else block which doesn't contain a static assert, and it was working well for that purpose. However, I noticed when going through some modules that it was missing some version blocks, those inside functions. When experimenting with the following test input, I noticed that it sometimes also won't find version blocks without braces.
Here's some D input that shows the problems:
version (linux)
{ printf("got here");}
else version (Android)
{ printf("got in here");}
void foo() {
version (linux)
{ printf("got here");}
else version (Android)
{ printf("got in here");}
version (linux) int x = 3;
else version (Android) int y = 5;
}
version (linux) printf("got here");
else version (Android) printf("got in here");
The first two version blocks are exactly the same, the fourth is the same as the first two but without braces.
I modified the symbol finder in Dscanner to notify on every version statement it finds:
--- a/src/symbol_finder.d
+++ b/src/symbol_finder.d
@@ -99,7 +99,14 @@ class FinderVisitor : ASTVisitor
}
}
- override void visit(const FunctionBody) {}
+ //override void visit(const FunctionBody fb) {output.writefln("found a function body");fb.accept(this);}
+
+ override void visit(const ConditionalDeclaration cd)
+ {
+ if(cd.compileCondition.versionCondition)
+ output.writefln("version checked is %s at line %d", cd.compileCondition.versionCondition.token.text, cd.compileCondition.versionCondition.token.line);
+ cd.accept(this);
+ }
mixin template generateVisit(T)
{
Running it on the input produces the following output:
version checked is linux at line 1
version checked is Android at line 3
version checked is linux at line 12
version checked is Android at line 13
The modified Dscanner only finds the first and third version blocks. It seems to miss the second version block because it's inside a function body and the fourth one because it doesn't have braces. However, those reasons don't seem to apply to the third version block, which is inside the function body and doesn't have braces.
I tried commenting out all the other node overrides in the Dscanner visitor, to see if they were interfering, but that didn't make a difference.
No new tag since this commit :
0edd67a
While testing my fork of harbored on various projects, I noticed huge memory usage with Tango in particular. I tracked it down to this file - the file is unusually big (~600kiB), but the parser allocates ~222 MiB of memory, which still seems like a lot (passed an allocator to measure it).
This is not really a pressing issue (everyone has >=4GiB nowadays), just thought it may be useful for optimization.
Consider:
const Foobar = "Foobar";
Results in:
1 dparse.parser.Parser.parseModule(1:1)
2 dparse.parser.Parser.parseDeclaration(1:1)
3 dparse.parser.Parser.parseAttribute(1:1)
3 dparse.parser.Parser.parseAttribute(1:7)
3 dparse.parser.Parser.parseVariableDeclaration(1:7)
4 dparse.parser.Parser.parseAutoDeclaration(1:7)
5 dparse.parser.Parser.parseNonVoidInitializer(1:16)
6 dparse.parser.Parser.parseAssignExpression(1:16)
7 dparse.parser.Parser.parseTernaryExpression(1:16)
8 dparse.parser.Parser.parseOrOrExpression(1:16)
9 dparse.parser.Parser.parseAndAndExpression(1:16)
10 dparse.parser.Parser.parseOrExpression(1:16)
11 dparse.parser.Parser.parseXorExpression(1:16)
12 dparse.parser.Parser.parseAndExpression(1:16)
13 dparse.parser.Parser.parseCmpExpression(1:16)
14 dparse.parser.Parser.parseShiftExpression(1:16)
15 dparse.parser.Parser.parseAddExpression(1:16)
16 dparse.parser.Parser.parseMulExpression(1:16)
17 dparse.parser.Parser.parsePowExpression(1:16)
18 dparse.parser.Parser.parseUnaryExpression(1:16)
19 dparse.parser.Parser.parsePrimaryExpression(1:16)
19 dparse.parser.Parser.parsePrimaryExpression(1:24)
18 dparse.parser.Parser.parseUnaryExpression(1:24)
17 dparse.parser.Parser.parsePowExpression(1:24)
16 dparse.parser.Parser.parseMulExpression(1:24)
15 dparse.parser.Parser.parseAddExpression(1:24)
14 dparse.parser.Parser.parseShiftExpression(1:24)
13 dparse.parser.Parser.parseCmpExpression(1:24)
12 dparse.parser.Parser.parseAndExpression(1:24)
11 dparse.parser.Parser.parseXorExpression(1:24)
10 dparse.parser.Parser.parseOrExpression(1:24)
9 dparse.parser.Parser.parseAndAndExpression(1:24)
8 dparse.parser.Parser.parseOrOrExpression(1:24)
7 dparse.parser.Parser.parseTernaryExpression(1:24)
6 dparse.parser.Parser.parseAssignExpression(1:24)
5 dparse.parser.Parser.parseNonVoidInitializer(1:24)
4 dparse.parser.Parser.parseAutoDeclaration(EOF:0)
3 dparse.parser.Parser.parseVariableDeclaration(EOF:0)
2 dparse.parser.Parser.parseDeclaration(EOF:0)
1 dparse.parser.Parser.parseModule(EOF:0)
The problem is, we end up with an AutoDeclaration
which has no StorageClass
, which is not possible according to the grammar.
(It actually ends up as attribute for the VariableDeclaration
instead of AutoDeclaration
storage class).
unittest
{
{
{
if (1)
{
}
else
{
x;
}
}
}
}
gives
testx.d(15:1)[error]: Declaration expected
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.